v0.5.5 — "Pinned sampling" was never determinism

Mostly a release about what SpecJudge claims. The behaviour barely moves; a promise it
could not keep is gone.

"Pinned sampling" was never determinism

SpecJudge sets temperature: 0 and a fixed seed, and FR-021 said that meant two runs on
the same project produce the same assessment. Checked, because a change to the retry logic
depended on it — two identical calls to qwen3:8b, same seed, same temperature:

attempt 1: {'n': 456789}
attempt 2: {'n': 123456}

The variation belongs to the local runtime — batching, cache state, quantisation, the
Ollama build — and no option we can send removes it. Pinning the seed stops SpecJudge
adding randomness; it does not make a local model deterministic.

So the guarantee is now the true one: the tool contributes no randomness of its own, and
any retry sequence is deterministic. And
docs/judges.md says
how much the figures actually move — devstral-small-2 went 32/32 → 32/33 and llama3.1
24/32 → 25/31 between two runs with no change to any input. If you compare two runs of
that table, a point or two is noise; the twelve separating the two 8B judges is not.

Small judges now get a second attempt

A judge on the full prompt could retry with the compact one. A judge already on the compact
prompt had nothing to fall back to, so it got a single try — leaving the models most likely
to fumble the response shape as the only ones without a retry.

Both get two now, the second with a different seed by design rather than by relying on that
runtime variation.

It recovered nothing. Both 8B judges needed a retry on this corpus and both refused
twice. The asymmetry was a real defect and fixing it is right, but it did not buy what it
was expected to, and the guide does not pretend otherwise — the retry count is its own
column there, because passing on the second attempt is not passing on the first.

Also

The spec-kit extension's README explains spec-kit's "discovery only — not installable",
which applies to every third-party extension until a project approves the community
catalog, and gives the command to change it — including the part that command does not say
out loud: it approves the whole catalog, not one extension.

Upgrading

pip install --upgrade specjudge

Nothing to do. No API, schema or exit code changed, and recommendations are unchanged
within the run-to-run variation now documented.

Full detail in the changelog.

v0.5.4 — Extension versioned with the tool

A packaging release. No change to the code the package installs, and none to what the
spec-kit extension does.

The extension used to carry its own version (0.1.x), on the reasoning that a manifest and
a prompt change far less often than the tool they wrap. The community catalog pins a
single download URL and validates that the declared version matches the release tag it
points at, so an independent version needs a release tag of its own for every change.
One number for the whole repository is the cheaper answer.

The cost, stated here so nobody has to work it out later: the extension's version now
moves whenever SpecJudge moves, even when nothing in the extension changed. A jump from
0.5.4 to 0.6.0 says nothing about the extension by itself — the changelog does.

specify extension add specjudge --from https://github.com/JoaquinRuiz/SpecJudge/releases/download/v0.5.4/spec-kit-specjudge.zip

The archive attached to v0.5.3 is left exactly as it was published. It already has
downloads, and rewriting a released artifact to correct its metadata is not something
this project does.

Upgrading

Nothing to do. pip install --upgrade specjudge installs a package identical to 0.5.3
apart from its metadata.

Full detail in the changelog.

v0.5.3 — The judge was rating projects it had not read

This release changes what SpecJudge recommends. It is a bug fix, and the version
number says so — but the effect is not cosmetic, and it is worth two minutes before you
upgrade.

The judge was rating projects it had not read

The compact prompt sends a digest instead of your artifacts. That digest was built from
text that had already been truncated, so it summarised the first 1,500 characters of
each file and said nothing false about them. Which is why nothing caught it.

On a real feature — the first live run of the spec-kit extension — the judge received
this, in full:

spec: 1340 chars
  sections (4): Feature Specification… | Context | User Scenarios…
tasks: 1429 chars
  sections (3): Tasks: Wiki multiproyecto | Format: `[ID] [P?]…`

No requirements. No tasks. FR-001 sat at line 82 of the spec and T001 at line 41, both
past the cut. Of 28 citable fragments, 22 were section headings.

So a rewrite of access-control resolution — 20 requirements, 29 tasks, the kind where a
silent regression leaks one client's content to another — came back reasoning: medium, size: low, recommended for an 8B model, citing the title of the tasks file as evidence.

What changed

A digest is bounded by its own structure, not by the length of the document it describes.
Truncating the summariser's input was the wrong way round.

• Built over the whole source, with the budget applied to the finished digest.
• Every named unit carries its own text, so it stays citable and verifiable — a catalogue
  of bare ids would turn a citation into a label picked from an inventory.
• Samples span the file. A task list opens with setup and closes with the hard part; a
  sample taken from the head describes the easy end and then decides the difficulty from
  it.
• Counts come from the whole file even when the sample does not. The tool was already
  computing "29 tasks" and throwing it away while the judge answered size: low.

Measured on the 18-project regression corpus:

Judge	Before	After
devstral-small-2 (24B)	30/30 in band	32/32, two more dimensions graded
qwen3:8b	25/28 (89%)	26/30 (87%)
llama3.1:8b q4	22/27 (81%)	24/32 (75%)

Read the denominators with the percentages: more of each project now reaches the judge, so
there is more to be right or wrong about. llama3.1 answers five more dimensions and gets
more of them wrong.

Also fixed

• Unfilled template placeholders ([PRINCIPLE_1_NAME]) are no longer offered as
  evidence. An untouched spec-kit constitution was contributing eight of them.

• The constraint table resolved citations against the wrong fragment set for judges
  above the compact-prompt threshold, so it came out with no text and every row marked
  customary — silently, and only for people running a large judge.

• The spec-kit extension's install command. --from supplies the source, it does not
  replace the name. Extension 0.1.2:

  specify extension add specjudge --from https://github.com/JoaquinRuiz/SpecJudge/releases/latest/download/spec-kit-specjudge.zip

A recommendation that changed

The quick start now suggests qwen3:8b instead of llama3.1:8b, and
docs/judges.md
states the trade instead of a headline: it is wrong half as often and half as far off, and
where it cannot cope it refuses rather than answering — costing about one project in
eighteen its recommendation. If you would rather always get an answer and judge it
yourself, the other one is the other side of that trade.

Which 8B matters more than the fact that it is an 8B. Two models of identical size and
price sit twelve points apart in that table.

Upgrading

pip install --upgrade specjudge

No API, schema or exit code changed. What changed is the input the judge receives, and
therefore its answers. If you keep recommendations under review, expect movement on
projects with long artifacts.

Full detail in the changelog.

v0.5.2 — The extension installs on current spec-kit

The spec-kit extension shipped yesterday with an upper bound on the spec-kit version it
accepts — >=0.13.0,<0.14.0. If you are on 0.14 or newer, it refused to install:

Compatibility Error: Extension requires spec-kit >=0.13.0,<0.14.0, but 0.15.2 is installed.

Nothing about the extension was actually incompatible. The bound was a guess about the
future, and it was reasoned from the wrong end: the range is checked when you install, so
an untested upper bound protects nobody and breaks everybody, on a date set by somebody
else's release calendar.

Fixed. The manifest now declares a floor and no ceiling, and the versions that were
actually tested — 0.13.0 and 0.15.2 — are named in the extension's README, which is where
a fact about the past belongs.

specify extension add specjudge --from https://github.com/JoaquinRuiz/SpecJudge/releases/latest/download/spec-kit-specjudge.zip

Extension version 0.1.1. If you installed it yesterday and it worked, nothing changes for
you.

Upgrading

Nothing to do. The installed Python package is identical to 0.5.1; this release exists to
carry the corrected extension archive.

Full detail in the changelog.

v0.5.1 — Integration with spec-kit

If you drive your work with spec-kit, SpecJudge now
runs itself at the one moment it is worth running.

You finish /speckit.tasks. The spec, the plan and the task list exist, and not a token
has been spent. That is when picking a model is both informed and free — and it is
precisely the moment that slips by, because the next thing spec-kit offers you is
/speckit.implement.

Recommend a model for these tasks? [y/N]

Say yes and you get the podium, plus the fragment of your own spec behind every level:

Budget envelope
   default: reasoning medium, size low
   reasoning: top — S:FR-001 (requirement)
   size: low — T:T002 (customary)

S:FR-001 is a line you wrote. If the recommendation looks more expensive than you
expected, you can go and read the sentence that made it expensive.

Install

specify extension add specjudge --from https://github.com/JoaquinRuiz/SpecJudge/releases/latest/download/spec-kit-specjudge.zip

Needs spec-kit 0.13.x, SpecJudge 0.5.0+, and Ollama with a local model. An 8B judge is
enough — see choosing a judge.

What it will not do to you

• It asks first. The judge is a model running on your machine, and starting one costs
  you a minute or two. Say no and nothing happens.
• It writes nothing. effect: read-only. The recommendation is reported, not filed
  next to your tasks.
• It never blocks. No judge configured, Ollama not running, a spec too thin to judge —
  it says so in one line and gets out of the way. You came here to implement.

Why this took no new code

The extension is a manifest and a prompt. It works because the --json payload has been a
versioned contract and the exit codes have been distinguishable since 0.3.0 — so wiring
SpecJudge into another tool's workflow cost a YAML file and a markdown one.

Upgrading

pip install --upgrade specjudge

Nothing to do, and nothing changes if you do not use spec-kit: the installed package is
identical to 0.5.0, and this release exists to carry the extension archive.

Full detail in the changelog.

v0.5.0 — One project, two complexities

Your task list has twenty mechanical edits and one architecture decision. Which model
should implement it?

That question has no single right answer, and until now SpecJudge gave one anyway. Rank on
the hardest task and you pay frontier prices for twenty edits that did not need them. Rank
on the average and you under-serve the one part that decides whether the thing works.

The demand is now an envelope

Budget envelope (escalating: ranked on the bulk of the work)
   default: reasoning medium, size low
   reasoning: top — S:FR-001 (requirement)
   size: low — T:T002 (customary)
   escalate for:
     • S:FR-001 — needs reasoning top

Every row names the fragment of your project that demands that level, and whether that
fragment states a requirement (MUST, a numbered FR-NNN) or merely describes a habit.
That call is derived from the text, not asked of the judge — so if you disagree, you can
open the fragment and see the same words the rule saw. An opinion returned by a model
gives you nothing to argue with.

You decide which reading ranks the podium

specjudge .                              # one model implements everything (default)
specjudge . --execution-model escalating # you can switch model per task

Whether one demanding task is decisive is not a property of your spec. It is a property of
how you work: if you can switch model halfway through, that task is a trigger, not a floor.
The same constraint table supports both readings; what changes is which one orders the
podium.

Without the flag, nothing moves. The default ranks on the hardest part, exactly as
0.4.0 did, and there is a test that fails if that ever stops being true.

Choosing a judge, with numbers

New: docs/judges.md
— what actually matters in a local judge, and measured results rather than impressions.

Judge	Params	Dimensions in band	Answers refused
devstral-small-2	24B	30/30 (100%)	0
qwen3:8b	8B	25/28 (89%)	1
llama3.1:8b-instruct-q4_K_M	8B	22/27 (81%)	1

An 8B judge gives very good results; a 24B-class judge gives perfect ones on our corpus.
Both run on a laptop, neither sends your specs anywhere.

That measurement also changed the design. Asking a judge to separate the bulk from the
peak costs an 8B model five points of accuracy and doubles its refused answers, while a
24B model answers it correctly and loses nothing — so only judges above 20B are asked. A
smaller judge still ranks and still cites; it reports one level instead of a range, and
says so rather than leaving you to infer it.

Run it on your own hardware and send the row:

uv run python scripts/eval_judge.py --judge <your-model> --markdown-row

For anything built on --json

Schema 1.2, additive: envelope carries the demand the ranking used, the demand the
hardest part needs, the constraint table behind both, and execution_model. Branch on
that last one if you automate spend — under escalating, best_choice answers "what
should implement most of this" rather than "what can implement all of it".

Envelope, Constraint and ExecutionModel join the public Python API, and
api.analyze takes an execution_model argument.

Upgrading

pip install --upgrade specjudge

No action needed. Everything here is additive, and the default ranking is unchanged.

Thanks to @rmarable for
#23, which is why the judge guide
exists.

Full detail in the changelog.

v0.4.0 — A spec is no longer the price of entry

SpecJudge used to answer one question: "I finished writing my spec — which model should implement
it?" If you had no .specify/ directory, it refused with exit code 2.

Most repositories have never run Spec-Driven Development. Nearly all of them have written down
something — an AGENTS.md, a CLAUDE.md, a .cursorrules, a folder of decision records. This
release reads all of it.

$ specjudge ~/code/some-repo-with-no-specs

⚠  No tasks or specification found; judged from environment context only (AGENTS.md). This is a
   floor — how demanding this repository is to work in — not a recommendation for a specific
   piece of work.

🥇 Gold: Claude Opus 5
   ...

Read: AGENTS.md

Evidence: 3 of 3 dimensions grounded in cited evidence
   reasoning: high — cites AG:1
   size: medium — cites AG:4
   domain_specialization: top — cites AG:6

This ranking rests on a thin definition. Before acting on it:
   • nothing describes the work in progress; only the repository's own context files were read
     → describe what you are about to build — a spec, a task list, or even a paragraph

An answer, and an honest label on it.

What it reads now

Source	Where it looks
spec-kit artifacts	constitution.md, spec.md, plan.md, tasks.md
agent-context files	AGENTS.md, CLAUDE.md, including nested ones in a monorepo
editor rules	.cursorrules, .github/copilot-instructions.md
decision records	docs/adr/, docs/decisions/, adr/

Read together, never one instead of another. Your spec describes the work you are about to do;
your AGENTS.md describes how demanding the codebase is to work in at all. Neither contains the
other, so picking one would throw away signal. plan.md had been sitting beside the spec all
along, unread.

Why this is safe now, and would not have been a release ago

Mixing formats used to mean deciding in advance how much to believe a .cursorrules versus a
spec — trust rules per format, five of them, each evolving separately.

0.2.0 removed the need. Every rated dimension must cite a fragment that exists in the input, and
a dimension that cannot be grounded returns unsupported and drops out of the calculation. The
source stopped mattering; only what is citable does. A thin source produces a thin profile on its
own, with no format-specific logic anywhere.

Reading a lot of files without drowning the judge

A monorepo can carry dozens of AGENTS.md — the Codex repository has 88. Twelve caps of 8000
characters is a 96k-character prompt, which is not a cap.

So context sources share one budget instead of each getting their own: a 200-character
.cursorrules costs 200 characters and leaves the rest to a long root AGENTS.md. Files nearest
the root are kept first, at most 12 are read (sources.max_context_files), and anything left out
is reported rather than silently dropped. docs/ is not swallowed whole — only decision records,
which are decisions with consequences rather than user guides.

Context files that announce a tool generated them are skipped: generated context mostly restates
what the code already shows, and it would take budget from the hand-written file next to it.

For anything built on --json

Schema 1.1, both fields additive:

• sources_read — which kinds of source fed the assessment
• environment_only — true when nothing described the work, so the result is a floor for the
  repository rather than a recommendation for a piece of work

A 1.0 consumer is unaffected. specjudge --print-schema still emits the contract.

Measured

The regression corpus grew from 12 cases to 16. On devstral-small-2: 28/28 dimensions in
band, 0 steps of ordinal distance, over-abstention down to 1. No existing case regressed — one
almost did, and the harness caught it before merge.

Over-confidence rose from 1 to 4: with nothing in the input describing an amount of work, the
judge answers size instead of abstaining. A real calibration gap, left visible in the corpus
rather than labelled away.

Upgrading

pip install --upgrade specjudge

No action needed. A project with .specify/ artifacts is read exactly as before. If you run
SpecJudge inside a large monorepo it will now pick up per-package AGENTS.md files it previously
ignored — sources.max_context_files in data/rating-rules.yaml bounds that.

Full detail in the changelog.

v0.3.1 — Cited evidence was half a sentence

Two things: a correctness fix worth upgrading for, and a warning that finally tells you what to
do about it.

Nothing covered by semantic versioning changed — no specjudge.api symbol, no schema field, no
exit code. Upgrading needs no action.

Cited evidence was half a sentence

Since 0.2.0 the judge has had to cite the fragment of your spec behind each rating. It turns out
it was being shown mutilated fragments.

Bullets were matched line by line, so any requirement that wrapped lost everything after the
wrap:

your spec:   - **FR-002**: Report generation MUST run entirely on the user's device, since no
                 personal data may leave it.

the judge:   "**FR-002**: Report generation MUST run entirely on the user's device, since no"

In the evaluation corpus, 140 of 160 fragments were affected. Most real specs wrap their
requirements, so most real projects have been judged on half-requirements and shown
half-sentences as evidence for two releases.

Fixed. Measured on a real 24B judge, accuracy went from 19/19 to 20/20 with one more dimension
answered — the tool was not only quoting less, it was judging worse.

A thin spec now tells you what is missing

Reported by @pete-lewis83, whose diagnosis was the useful part:

the problem isn't that the warning is missing, it's that the podium prints right underneath
it and wins.

Exactly right. A one-line caveat above a ranked table with medals and prices is not a fair
fight. So the caveat moved to where reading actually ends, and it stopped being vague:

This ranking rests on a thin definition. Before acting on it:
   • the spec declares no numbered requirements (looked for FR-NNN)
     → state what the system must do, one numbered requirement per behaviour
   • no acceptance criteria found (looked for SC-NNN or NFR-NNN)
     → say how you will know it works — the measurable bar each requirement meets

Each line names what it searched for. These are heuristics over Spec Kit conventions and they
will sometimes be wrong; a project that numbers its criteria differently deserves to be able to
dismiss the warning in a second rather than be accused of not having written them.

The HTML report had the same problem and gets the same fix.

Also

The judge is now asked for the fragment most relevant to a dimension rather than one that
proves the rating. Demanding proof made any whole-project dimension unanswerable — every
over-abstention measured across two judges was domain_specialization, which no single sentence
supports. That halved.

One thing that did not get fixed, stated plainly: the judge still does not notice when a
spec contradicts itself. Two prompt formulations were tried and produced identical results. It
is documented in #19 with what was measured and the two untried levers, and the evaluation
corpus keeps expecting an abstention there, so every run keeps reporting the gap until somebody
closes it.

Full changelog: https://github.com/JoaquinRuiz/SpecJudge/blob/main/CHANGELOG.md#031---2026-08-07

v0.3.0 — Something you can build on

Until now SpecJudge was something you ran. This release makes it something you can build
on — from any language, not just Python.

Nothing here is breaking. Upgrading from 0.2.0 needs no action.

A contract you can pin, in any language

--json has always worked, but it carried no version and was described in prose in a spec
document. That is fine to read and useless to depend on: you could not pin anything, and
you would find out it moved when your parser broke.

Now every payload carries a schema_version, and there is a real JSON Schema you can fetch
without cloning the repo or reading a line of Python:

specjudge --print-schema > specjudge.schema.json

Generate types from it, validate against it, pin to it. The rules are written down: adding
a field is a MINOR bump, removing one or changing what a value means is MAJOR. The version
is deliberately independent of the package version — otherwise the contract would appear to
change every time somebody updated a price in the catalog.

Two things the schema pins tightly and two it deliberately leaves loose. rating and
evidence.status are closed vocabularies — the first is fixed by the project constitution,
so widening it needs an amendment, not just a release. But demand levels and dimension names
come from rating-rules.yaml, which you are encouraged to edit, so they are typed as plain
strings. Enumerating them would have made the published schema wrong for anyone who
customised their rules.

And a Python entry point that is actually a promise

from specjudge import api

comparison = api.analyze("path/to/project", judge_model="llama3.1:8b")
print(comparison.best_choice)

Sixteen names, listed in docs/api.md,
and everything else under specjudge.* is explicitly internal and free to change in a patch
release. That second half matters as much as the first: a surface with no stated edges is not
a stable API, it is an accident waiting to be relied on.

The errors are exported too, each carrying the exit code the CLI uses, so you can tell "this
project cannot be judged" from "your environment is the problem" without matching on strings.

Also in this release

The README now states the Ollama 0.5.0+ requirement that 0.2.0 introduced. It had only
ever appeared in the changelog and in the error message you hit — which is exactly the wrong
order. It also documents the cited evidence the judge has to provide, which 0.2.0 shipped
without ever explaining on the front page.

And a small embarrassment fixed: the HTML report was advertising Del vibe coding al
Spec-Driven Development as forthcoming, without a link, when it had been on sale since July
— and omitting Explora la Inteligencia Artificial entirely. It shipped that way twice. The
cause was the same list living in two files with nothing tying them together, so there is now
a test that compares them and rejects a book that is both linked and marked as forthcoming.

Upgrading

Nothing to do. schema_version is a new field, and adding one is additive by the contract's
own rules: a consumer written against the 0.2.0 payload keeps working and simply gains a
version to pin against.

One thing worth knowing if you were already importing from the package: the public surface is
specjudge.api and nothing else. If you were reaching into specjudge.cli for the
serialiser, it now lives in specjudge.serialize and is re-exported as api.to_dict.

Full changelog: https://github.com/JoaquinRuiz/SpecJudge/blob/main/CHANGELOG.md#030---2026-08-06

v0.2.0 — The judge has to show its work

A recommendation you cannot check is a guess with better formatting. This release makes the
judge show its work — and then checks the work.

Breaking: needs Ollama 0.5.0 or newer. See Upgrading at the end.

The judge has to cite the spec now

Before, the judge returned a level per dimension and a paragraph explaining itself. The
paragraph was doing no work: a fluent rationalisation is exactly what a language model
produces well, whether or not the score underneath it is sound. Nothing separated a good
assessment from a well-narrated bad one.

Now every rated dimension has to name the fragment of your spec that supports it, and
the tool checks that fragment actually exists in the text the judge was given:

Evidence: 3 of 3 dimensions grounded in cited evidence
   reasoning: medium — cites S:FR-004
   size: medium — cites T:T009
   domain_specialization: low — cites S:FR-001

A judge that invents a citation has its whole assessment thrown out, not just that field.

This validates grounding, not relevance — a judge can cite a fragment that exists but
does not really support the level, and no deterministic check catches that. It is the
cheapest real verification available, and it is a great deal more than a paragraph.

"I cannot tell" is now an answer

Dimensions the judge cannot ground come back as unsupported rather than estimated. Those
dimensions leave the fit calculation entirely — they are not quietly treated as easy,
which is what used to happen and made thin projects look simpler than they were.

If nothing can be grounded, you get no recommendation and a clear reason, the same as a
project with no tasks. If only part can, you get the recommendation plus a warning saying
which dimensions rest on nothing.

A regression suite for the judge itself

The prompt and the rating rules decide the recommendation, and until now changing either
was guesswork. There is now a corpus of twelve projects with expected profiles —
well-specified, thin, task-less, and adversarial specs that look thorough while hiding the
hard part.

Two tiers, because they answer different questions:

• In CI, a deterministic suite: classification, frozen rankings, prompt invariants. It
  proves a change did not break the contract.
• Locally, scripts/eval_judge.py against your own Ollama. It reports accuracy and
  abstention quality separately, because they fail in opposite directions — a judge that
  abstains on everything scores perfectly on accuracy alone.

A green CI run is not a statement about judgement quality. Only the local run is.

The suite immediately found a real bug

On its first run, 8B judges — the most common local setup — failed on every single
project. Not a judgement failure: they rated correctly and wrote sensible justifications,
then put [true] where a fragment id belonged. Ollama's format: "json" guarantees valid
JSON, not the JSON we asked for.

Sending a proper schema fixed it:

Judge	Usable cases before	After
llama3.1 (8B)	0 of 9	9 of 9
devstral-small-2 (24B)	5 of 9	9 of 9, all 19 ratings in their expected band

That is the suite paying for itself within an hour of existing.

Same project, same answer

Judge sampling is now pinned. Two runs over one project used to be able to recommend
different models, which is hard to defend in a tool about spending money. It pins
sampling, not the world — a different Ollama build or quantisation still moves answers.

Upgrading

Upgrade Ollama to 0.5.0 or newer. This is the change that turns a working setup into a
failing one. Older versions reject the schema and you will get exit 3 with an error naming
the requirement.

If your judge cannot cite evidence, or you cannot upgrade Ollama, add to
data/rating-rules.yaml:

evidence:
  require_spans: false

That restores the previous behaviour, and loses the grounding check with it.

If you script against --json: the schema gains a demand key; nothing was removed.
But data_state can now be scarce where 0.1.4 reported sufficient, when the judge
grounds only part of its assessment.

Full detail, including the exit-code table: CHANGELOG