Skip to content

v0.5.0 — One project, two complexities

Choose a tag to compare

@JoaquinRuiz JoaquinRuiz released this 10 Aug 12:23
· 11 commits to main since this release

Your task list has twenty mechanical edits and one architecture decision. Which model
should implement it?

That question has no single right answer, and until now SpecJudge gave one anyway. Rank on
the hardest task and you pay frontier prices for twenty edits that did not need them. Rank
on the average and you under-serve the one part that decides whether the thing works.

The demand is now an envelope

Budget envelope (escalating: ranked on the bulk of the work)
   default: reasoning medium, size low
   reasoning: top — S:FR-001 (requirement)
   size: low — T:T002 (customary)
   escalate for:
     • S:FR-001 — needs reasoning top

Every row names the fragment of your project that demands that level, and whether that
fragment states a requirement (MUST, a numbered FR-NNN) or merely describes a habit.
That call is derived from the text, not asked of the judge — so if you disagree, you can
open the fragment and see the same words the rule saw. An opinion returned by a model
gives you nothing to argue with.

You decide which reading ranks the podium

specjudge .                              # one model implements everything (default)
specjudge . --execution-model escalating # you can switch model per task

Whether one demanding task is decisive is not a property of your spec. It is a property of
how you work: if you can switch model halfway through, that task is a trigger, not a floor.
The same constraint table supports both readings; what changes is which one orders the
podium.

Without the flag, nothing moves. The default ranks on the hardest part, exactly as
0.4.0 did, and there is a test that fails if that ever stops being true.

Choosing a judge, with numbers

New: docs/judges.md
— what actually matters in a local judge, and measured results rather than impressions.

Judge Params Dimensions in band Answers refused
devstral-small-2 24B 30/30 (100%) 0
qwen3:8b 8B 25/28 (89%) 1
llama3.1:8b-instruct-q4_K_M 8B 22/27 (81%) 1

An 8B judge gives very good results; a 24B-class judge gives perfect ones on our corpus.
Both run on a laptop, neither sends your specs anywhere.

That measurement also changed the design. Asking a judge to separate the bulk from the
peak costs an 8B model five points of accuracy and doubles its refused answers, while a
24B model answers it correctly and loses nothing — so only judges above 20B are asked. A
smaller judge still ranks and still cites; it reports one level instead of a range, and
says so rather than leaving you to infer it.

Run it on your own hardware and send the row:

uv run python scripts/eval_judge.py --judge <your-model> --markdown-row

For anything built on --json

Schema 1.2, additive: envelope carries the demand the ranking used, the demand the
hardest part needs, the constraint table behind both, and execution_model. Branch on
that last one if you automate spend — under escalating, best_choice answers "what
should implement most of this" rather than "what can implement all of it".

Envelope, Constraint and ExecutionModel join the public Python API, and
api.analyze takes an execution_model argument.

Upgrading

pip install --upgrade specjudge

No action needed. Everything here is additive, and the default ranking is unchanged.

Thanks to @rmarable for
#23, which is why the judge guide
exists.

Full detail in the changelog.