Skip to content

v0.4.0 — A spec is no longer the price of entry

Choose a tag to compare

View all tags
@JoaquinRuiz JoaquinRuiz released this 07 Aug 14:21
· 15 commits to main since this release
v0.4.0
225215f

SpecJudge used to answer one question: "I finished writing my spec — which model should implement
it?" If you had no .specify/ directory, it refused with exit code 2.

Most repositories have never run Spec-Driven Development. Nearly all of them have written down
something — an AGENTS.md, a CLAUDE.md, a .cursorrules, a folder of decision records. This
release reads all of it.

$ specjudge ~/code/some-repo-with-no-specs

⚠  No tasks or specification found; judged from environment context only (AGENTS.md). This is a
   floor — how demanding this repository is to work in — not a recommendation for a specific
   piece of work.

🥇 Gold: Claude Opus 5
   ...

Read: AGENTS.md

Evidence: 3 of 3 dimensions grounded in cited evidence
   reasoning: high — cites AG:1
   size: medium — cites AG:4
   domain_specialization: top — cites AG:6

This ranking rests on a thin definition. Before acting on it:
   • nothing describes the work in progress; only the repository's own context files were read
     → describe what you are about to build — a spec, a task list, or even a paragraph

An answer, and an honest label on it.

What it reads now

Source Where it looks
spec-kit artifacts constitution.md, spec.md, plan.md, tasks.md
agent-context files AGENTS.md, CLAUDE.md, including nested ones in a monorepo
editor rules .cursorrules, .github/copilot-instructions.md
decision records docs/adr/, docs/decisions/, adr/

Read together, never one instead of another. Your spec describes the work you are about to do;
your AGENTS.md describes how demanding the codebase is to work in at all. Neither contains the
other, so picking one would throw away signal. plan.md had been sitting beside the spec all
along, unread.

Why this is safe now, and would not have been a release ago

Mixing formats used to mean deciding in advance how much to believe a .cursorrules versus a
spec — trust rules per format, five of them, each evolving separately.

0.2.0 removed the need. Every rated dimension must cite a fragment that exists in the input, and
a dimension that cannot be grounded returns unsupported and drops out of the calculation. The
source stopped mattering; only what is citable does. A thin source produces a thin profile on its
own, with no format-specific logic anywhere.

Reading a lot of files without drowning the judge

A monorepo can carry dozens of AGENTS.md — the Codex repository has 88. Twelve caps of 8000
characters is a 96k-character prompt, which is not a cap.

So context sources share one budget instead of each getting their own: a 200-character
.cursorrules costs 200 characters and leaves the rest to a long root AGENTS.md. Files nearest
the root are kept first, at most 12 are read (sources.max_context_files), and anything left out
is reported rather than silently dropped. docs/ is not swallowed whole — only decision records,
which are decisions with consequences rather than user guides.

Context files that announce a tool generated them are skipped: generated context mostly restates
what the code already shows, and it would take budget from the hand-written file next to it.

For anything built on --json

Schema 1.1, both fields additive:

  • sources_read — which kinds of source fed the assessment
  • environment_only — true when nothing described the work, so the result is a floor for the
    repository rather than a recommendation for a piece of work

A 1.0 consumer is unaffected. specjudge --print-schema still emits the contract.

Measured

The regression corpus grew from 12 cases to 16. On devstral-small-2: 28/28 dimensions in
band, 0 steps of ordinal distance, over-abstention down to 1. No existing case regressed — one
almost did, and the harness caught it before merge.

Over-confidence rose from 1 to 4: with nothing in the input describing an amount of work, the
judge answers size instead of abstaining. A real calibration gap, left visible in the corpus
rather than labelled away.

Upgrading

pip install --upgrade specjudge

No action needed. A project with .specify/ artifacts is read exactly as before. If you run
SpecJudge inside a large monorepo it will now pick up per-package AGENTS.md files it previously
ignored — sources.max_context_files in data/rating-rules.yaml bounds that.

Full detail in the changelog.

Assets 2
Loading