chore(evals): record the eight-model sweep behind #513 and #515 - #516
Merged
Conversation
Edge typing across eight models on the frozen 17-pair fixture, plus extraction recall for the two candidates a default swap would need. Parameter count is the dominant axis, not family: qwen3 8b->14b is +0.22 and gemma2 9b->27b is +0.18, while family only separates sharply among the small models. That corrects #513's implied reading that qwen3 is bad at this task -- qwen3:14b is second best. The finding that matters for #515: the best relation typer measured (gemma2:27b, 0.81) is the worst extractor measured -- subject recall 0.24 on the long English fixture and 0.00 on the Spanish one, with low adjudication debt, so not a scoring artifact. The gemma2 family under-produces on extraction at both sizes, the opposite of how it behaves on edge typing. So the +0.37 available on edge typing can only be collected per task, never by moving the global default. Also records the trap that caught me: qwen3:14b's higher produced count is decay and unjudged titles, not subjects, so the produced count is not a quality signal in that harness -- subject recall and known-facet decay have to be read separately.
This was referenced Aug 10, 2026
jasonssdev
added a commit
that referenced
this pull request
Aug 10, 2026
… is good at it (#515) (#517) `model:` in `openkos.yaml` is global: every verb that calls a model uses it. The eight-model sweep in #516 found that a single global value cannot express what the measurements say -- the best relation typer measured is the worst extractor measured. `gemma2:27b` scores 0.81 on `evals/edge_typing/`'s 17-edge fixture against the configured default `qwen3:8b`'s 0.44, and on the same corpus collapses extraction to 0.24 subject recall on the long English fixture and 0.00 on the Spanish one. A new optional `models:` map overrides `model:` for one task at a time. Every task it does not name keeps the global default, so a workspace that does not opt in behaves exactly as before. Keyed by task, never by verb: `suggest_edge_types` is used by both `curate`'s Structure stage and standalone `suggest-relations`, and a per-verb key would let the two drift onto different models -- the drift #385's design already prevents by routing both through one write core. Three decisions the issue's principles settled but did not spell out: - `models:` validates rather than degrading. Its two passthrough precedents (`volatility_windows`, `type_tiers`) silently degrade a malformed value; this one refuses an unknown key or a non-string/blank value at read time. #515's rejection of a silent fallback does not distinguish a model that is missing from a name that is malformed -- a typo would keep writing relation types from a model nobody chose. - The cost gate discloses the model on a SEPARATE line, and only when the stage resolves something other than the global default. A suffix on `cost_line` would have rewritten the literal the `curate-command` spec pins byte-identical, for every workspace including those that never opted in. - `query` and `curate`'s locality probe name no task. `query` has no harness, so #508's rule forbids picking a model for it; locality is a property of the host, not of any task. `curate` now tracks availability per model rather than per run. One failed connection no longer settles reachability for models it never contacted: Structure failing for want of `gemma2:27b` says nothing about Metadata's model. The deliberate cost is one failed connection per distinct model against a dead server; clients are cached by model so stages sharing a tag share one connection. The `curate-command` spec moves with the code rather than being quietly broken. A named model that is not installed fails only the stage that named it, with an `ollama pull` remediation naming THAT model -- never the global default, which would send the operator to pull something already installed while the missing one stayed missing. Only `edge_typing` has a harness today. The other four keys are accepted because restricting the schema would be arbitrary, and the docs say plainly which has evidence behind it. Suite 4052 -> 4079. Closes #515
This was referenced Aug 10, 2026
jasonssdev
added a commit
that referenced
this pull request
Aug 10, 2026
…518) `suggest_edge_types` answered about two thirds of rubric-decidable pairs against its own rubric, and did so stably: 0.35-0.44 type accuracy at 0.99 modal share across runs. Confidently wrong types are the worst shape this can take -- self-consistency, confidence thresholds, and majority voting all sample from the same confident mistake, and every accepted suggestion is written into a document's `relations:` where the graph projection and everything reading it believes the type string. #515 shipped the mechanism; this makes it the default. `edge_typing` now resolves `gemma2:27b`, measured at 0.81 on the same 17-edge fixture through `evals/edge_typing/` (#516's eight-model sweep). BREAKING: existing workspaces are affected without editing anything, and the model costs a 15.6 GB `ollama pull`. Until it is pulled, `curate`'s Structure stage reports unavailable with that exact pull command and every other stage runs normally -- the per-stage failure #515 built for. No other verb is touched. Two things the packaged default made mandatory: - An opt-out. `models: {edge_typing: null}` declines a packaged default and resolves the task back to the global `model:`. An explicit null is the ONLY opt-out: restating the global tag works today but silently goes stale the next time `model:` changes. Precedence is now explicit `models:` entry > packaged default > global `model:`, so an operator's stated choice always beats a shipped one. - A way to see it. `doctor` gains a twelfth, informational check reporting whether every model a task resolves is installed, naming the task and offering the pull command. Before it, `doctor` looked only at `cfg.model` and gave a clean bill of health to a workspace whose Structure stage would fail part-way through the next session. It is ONE check regardless of task count, so the total stays fixed; it examines only models differing from the global tag, so one root cause is never double-counted; it never changes the exit code, since a missing per-task model fails only its own stage; and it `[SKIP]`s when Ollama is unreachable. Nothing else is packaged. The same model is the worst extractor measured (0.24 subject recall on the long English fixture, 0.00 on the Spanish one, against `qwen3:8b`'s 0.81 and 0.76), so `extraction` deliberately stays on the model `evals/extraction_cap/` tuned. The three tasks with no harness get no packaged value at all -- #508's rule. Suite 4079 -> 4092. Closes #513
This was referenced Aug 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
evals/only — no production code.Records the measurement runs behind the last two comments on #513 and #515, so the numbers in those threads have their raw results in the repo.
Edge typing, eight models, frozen 17-pair fixture
gemma2:27bqwen3:14bgemma2:9bphi4:14bqwen2.5:7bllama3.1:8bqwen3:8b(default)mistral:7bParameter count dominates, not family —
qwen38b→14b is +0.22,gemma29b→27b is +0.18. That corrects #513's implied reading thatqwen3is bad at this:qwen3:14bis second best.And why the default still cannot move
Subject recall from
extraction_cap --runs 3:qwen3:8bqwen3:14bgemma2:27blarge-03(EN)medium-08medium-09small-04(ES)The best relation typer measured is the worst extractor measured, and the
gemma2family under-produces on extraction at both sizes (3.3 objects/run at 9b, 3.2 at 27b, against the default's 6.8) — the opposite of how it behaves on edge typing. The +0.37 on edge typing is only collectable per task (#515).A trap this PR documents
qwen3:14bproduces 10.2 objects per run against the default's 6.8, which reads as an upgrade and is not: the extra volume is decay and unjudged titles, not subjects —medium-09's known-facet count triples. In that harnessproducedis not a quality signal.Its adjudication debt (1.3–4.7 unjudged per run against roughly zero) also leaves its recall under-reported, so "worse at extraction" is not established either. That one stays open.
Disclosure: no review receipt
Same lens defect as the preceding PRs this session.