v0.2.0 — The judge has to show its work

A recommendation you cannot check is a guess with better formatting. This release makes the
judge show its work — and then checks the work.

Breaking: needs Ollama 0.5.0 or newer. See Upgrading at the end.

The judge has to cite the spec now

Before, the judge returned a level per dimension and a paragraph explaining itself. The
paragraph was doing no work: a fluent rationalisation is exactly what a language model
produces well, whether or not the score underneath it is sound. Nothing separated a good
assessment from a well-narrated bad one.

Now every rated dimension has to name the fragment of your spec that supports it, and
the tool checks that fragment actually exists in the text the judge was given:

Evidence: 3 of 3 dimensions grounded in cited evidence
   reasoning: medium — cites S:FR-004
   size: medium — cites T:T009
   domain_specialization: low — cites S:FR-001

A judge that invents a citation has its whole assessment thrown out, not just that field.

This validates grounding, not relevance — a judge can cite a fragment that exists but
does not really support the level, and no deterministic check catches that. It is the
cheapest real verification available, and it is a great deal more than a paragraph.

"I cannot tell" is now an answer

Dimensions the judge cannot ground come back as unsupported rather than estimated. Those
dimensions leave the fit calculation entirely — they are not quietly treated as easy,
which is what used to happen and made thin projects look simpler than they were.

If nothing can be grounded, you get no recommendation and a clear reason, the same as a
project with no tasks. If only part can, you get the recommendation plus a warning saying
which dimensions rest on nothing.

A regression suite for the judge itself

The prompt and the rating rules decide the recommendation, and until now changing either
was guesswork. There is now a corpus of twelve projects with expected profiles —
well-specified, thin, task-less, and adversarial specs that look thorough while hiding the
hard part.

Two tiers, because they answer different questions:

• In CI, a deterministic suite: classification, frozen rankings, prompt invariants. It
  proves a change did not break the contract.
• Locally, scripts/eval_judge.py against your own Ollama. It reports accuracy and
  abstention quality separately, because they fail in opposite directions — a judge that
  abstains on everything scores perfectly on accuracy alone.

A green CI run is not a statement about judgement quality. Only the local run is.

The suite immediately found a real bug

On its first run, 8B judges — the most common local setup — failed on every single
project. Not a judgement failure: they rated correctly and wrote sensible justifications,
then put [true] where a fragment id belonged. Ollama's format: "json" guarantees valid
JSON, not the JSON we asked for.

Sending a proper schema fixed it:

Judge	Usable cases before	After
llama3.1 (8B)	0 of 9	9 of 9
devstral-small-2 (24B)	5 of 9	9 of 9, all 19 ratings in their expected band

That is the suite paying for itself within an hour of existing.

Same project, same answer

Judge sampling is now pinned. Two runs over one project used to be able to recommend
different models, which is hard to defend in a tool about spending money. It pins
sampling, not the world — a different Ollama build or quantisation still moves answers.

Upgrading

Upgrade Ollama to 0.5.0 or newer. This is the change that turns a working setup into a
failing one. Older versions reject the schema and you will get exit 3 with an error naming
the requirement.

If your judge cannot cite evidence, or you cannot upgrade Ollama, add to
data/rating-rules.yaml:

evidence:
  require_spans: false

That restores the previous behaviour, and loses the grounding check with it.

If you script against --json: the schema gains a demand key; nothing was removed.
But data_state can now be scarce where 0.1.4 reported sufficient, when the judge
grounds only part of its assessment.

Full detail, including the exit-code table: CHANGELOG