Skip to content

v0.5.5 — "Pinned sampling" was never determinism

Latest

Choose a tag to compare

@JoaquinRuiz JoaquinRuiz released this 04 Sep 10:04

Mostly a release about what SpecJudge claims. The behaviour barely moves; a promise it
could not keep is gone.

"Pinned sampling" was never determinism

SpecJudge sets temperature: 0 and a fixed seed, and FR-021 said that meant two runs on
the same project produce the same assessment. Checked, because a change to the retry logic
depended on it — two identical calls to qwen3:8b, same seed, same temperature:

attempt 1: {'n': 456789}
attempt 2: {'n': 123456}

The variation belongs to the local runtime — batching, cache state, quantisation, the
Ollama build — and no option we can send removes it. Pinning the seed stops SpecJudge
adding randomness; it does not make a local model deterministic.

So the guarantee is now the true one: the tool contributes no randomness of its own, and
any retry sequence is deterministic. And
docs/judges.md says
how much the figures actually move — devstral-small-2 went 32/32 → 32/33 and llama3.1
24/32 → 25/31 between two runs with no change to any input. If you compare two runs of
that table, a point or two is noise; the twelve separating the two 8B judges is not.

Small judges now get a second attempt

A judge on the full prompt could retry with the compact one. A judge already on the compact
prompt had nothing to fall back to, so it got a single try — leaving the models most likely
to fumble the response shape as the only ones without a retry.

Both get two now, the second with a different seed by design rather than by relying on that
runtime variation.

It recovered nothing. Both 8B judges needed a retry on this corpus and both refused
twice. The asymmetry was a real defect and fixing it is right, but it did not buy what it
was expected to, and the guide does not pretend otherwise — the retry count is its own
column there, because passing on the second attempt is not passing on the first.

Also

The spec-kit extension's README explains spec-kit's "discovery only — not installable",
which applies to every third-party extension until a project approves the community
catalog, and gives the command to change it — including the part that command does not say
out loud: it approves the whole catalog, not one extension.

Upgrading

pip install --upgrade specjudge

Nothing to do. No API, schema or exit code changed, and recommendations are unchanged
within the run-to-run variation now documented.

Full detail in the changelog.