v0.9.2.2

Chimera 0.9.2.2 — the gates

Released: 2026-07-30

A sub-version, deliberately — the third digit is not a batch counter
(docs/playbooks/14-release-discipline.md, rule 0).

0.9.2.1 was the verified grader: it added a canary that grades the grader.
This release is the follow-through, and the theme is narrower than "more
checks": every claim this repo makes is now enforced by something that can
fail, rather than asserted by something that cannot. Graders, published
numbers, on-disk paths, and cwd-relative writes each moved from convention to
gate — and each move started by finding a real defect the convention had been
hiding.

uvx --from chimera-run==0.9.2.2 chimera --version

The headline number was 14.8 points too high

mbpp-plus published 99.7% (377/378). That was the MBPP+ task set graded
with the dataset's base test_list asserts: MBPPPlus inherited MBPP's
grading path, and the EvalPlus expanded test blob sat staged in every row and
never executed.

The expanded harness now runs. Re-running the same tasks and agent under it:

	result	cost
base-graded (previously published)	377/378 = 99.7%	$2.38
plus-graded (this release)	321/378 = 84.9%	$2.32

14.8 points, attributable to grading alone. Receipt:
data/modal-grid-fullscore4-20260730-plusgrade.json, {completed: 378}.

The stronger contract is not merely different — a hardcoded lookup satisfying
every base assertion for is_not_prime is accepted by base and rejected by
plus.

On provenance, precisely. agent_id (coding-agent) and n=378 are recorded
in both receipts; the model is recorded only in the new one (glm-5.2),
because the 2026-07-09 flagship receipts predate that field and the column's
model is documented once on the observatory page rather than per cell. The
near-identical cost — $2.38 versus $2.32, 2.5% apart — is independent
corroboration: the agent solved as before, and the grader accepted fewer
answers. An earlier draft of these notes said "identical model", which asserted a
field one receipt does not carry.

The ceiling is 377/378, not 378. Task 590 (polar_rect) cannot pass: the
upstream harness compares floats with atol=0 and the dataset's own canonical
answer differs in the last ULP. So 84.9% reads against an achievable 99.7%.

A sentence of prose resolved a SWE-bench instance

Five adapters — swe_bench, swt_bench, swe_polybench, feature_bench,
dpai_arena (six sites) — graded a task solved from
len(agent_output.strip()) > 10 whenever they could not run the benchmark's own
tests. Verified live, not inferred:

swe-bench      [env=no-runner]  "I have analyzed the issue and
                                 implemented a comprehensive fix."   ->  RESOLVED
swt-bench      [env=None]        same prose                          ->  RESOLVED
swe-polybench  [env=no-runner]   same prose                          ->  RESOLVED
feature-bench  [env=no-runner]   same prose                          ->  RESOLVED
dpai-arena     [env=no-runner]   same prose, 3 tracks                ->  RESOLVED

All six now grade unresolved. Inability to grade is not a pass, for the same
reason a cloud sandbox refuses to degrade to local: either way the result becomes
indistinguishable from a real one. The resulting zero is also legible — a
uniform-zero column is already the harness-gap signature the observatory refuses
to publish, whereas a 100% built from prose is invisible.

Three things kept this alive through a green suite, and each became a rule:

• The adapters disagreed about which configuration was unsafe. swe_bench
  was safe at env=None and unsafe with a runner-less env; swt_bench was the
  exact mirror image. A test covering one shape would have passed on four of
  five, so the new test parametrises both.
• It spread by imitation. dpai_arena._evaluate_rubric's docstring said its
  placeholder was "matching the SWE-bench fallback behaviour". So the fix ships
  with a static AST gate rejecting len() of an answer-shaped parameter —
  AST and not grep, because the fix's own comments quote the old code and a text
  search would force deleting the explanation to stay green.
• Six existing tests asserted the defect (test_evaluate_fallback_heuristic,
  test_evaluate_no_env_uses_length_heuristic,
  test_evaluate_unknown_track_falls_back_to_length, and three siblings). The
  suite was not blind to the behaviour; it was pinning it in place. When a
  test's name describes a fallback rather than a contract, read it as a
  confession.

Substring grading accepted a different value

The sibling defect, in graders that do have a reference answer and compared it
with a raw in:

ContextBench:  truth "42"                 <- answer "142"             PASSED
TauBench:      action "transfer_to_agent" <- "transfer_to_agent_v2"   PASSED

Both now match on word boundaries — applied only where the truth's own edge is
alphanumeric, so a truth like $5 or f(x) stays matchable. Anchoring
unconditionally would make those permanently unmatchable, trading a false-accept
for a silent false-reject, which is worse: a zero column reads as a hard
benchmark rather than a broken anchor. A second static gate rejects raw
containment against an answer.

Neither adapter fed a published number, so this is prevention rather than a
retraction.

Disclosed, not solved: an answer that negates or hedges around the truth
("The answer is NOT 42") still grades correct. Detecting that is
natural-language judgement, and the judge hook exists for it. A test pins the
current behaviour so the limitation stays visible — one nobody can see is
indistinguishable from a bug nobody found.

Two ways a disclosure can be hollow

Both found by verifying claims already written down.

A ceiling with nothing behind it. CEILINGS' docstring says it is "Verified
by canary_benchmarks.py (KNOWN_UNPASSABLE)" — nothing enforced that pairing,
and the mbpp-plus ceiling shipped while its counterpart sat uncommitted in
another worktree. A ceiling is a published reduction of the denominator; it does
not get to be self-asserted. Now test-paired.

Changing a grader silently inverted a canary verdict. Recipe.test_fields
names the source the grader executes — a claim about another module, which
rots silently. mbpp-plus scanned test_list under a comment asserting that was
what MBPPPlus.evaluate actually ran: true when written, wrong once the plus
harness landed. With numpy absent the sweep reported BROKEN, accusing a
working grader, where the honest verdict was ENV-MISSING — the false-alarm class
that sends someone to fix correct code. Proven three ways against a genuinely
numpy-free venv: stale scope → BROKEN, fixed scope → ENV-MISSING, fixed + numpy
→ PASS.

An exemption's stated reason is also a claim. context-bench and nocha
both sat exempt as "no reference answer" while their graders read one. Nobody
re-derives a reason once the sweep prints EXEMPT beside it. Both are real recipes
now, each verified to accept its reference answer and reject a wrong one.

The caveat markers were never on the number

CEILINGS and GRADING_NOTES shipped in 0.9.2.1 specifically so a caveat
travels with a score — but †/‡ appeared only in footnotes below the table.
A scorecard row is the unit people copy, so the number travelled and the caveat
stayed behind. Markers now render inside the score cell, suppressed on retracted
rows whose footnotes are not emitted.

Also in this release

• bench-matrix exited 0 on a run that graded nothing. Any failed benchmark
  run reported success to a CI pipeline or a script. Distinct exit codes for
  all-cells-failed and some-cells-failed, plus up-front --model validation.
• One path registry for every on-disk store (chimera/config/paths.py). A
  directory the registry does not name is by definition an orphan — which is what
  lets chimera doctor --section storage report it and makes it structurally
  impossible for chimera gc to delete something undeclared. gc is dry-run by
  default, with --apply / --archive opt-in.
• User-scope permission rules and hooks were silently ignored — both loaded
  from project scope only. Two config systems that appeared to work and did
  nothing.
• Checkpoints no longer copy the whole tree, and a static gate now fails the
  suite on any cwd-relative write across chimera/, scripts/, tests/ and
  examples/.
• Published claims must cite receipts a reader can open — the gate reads
  git ls-files, not the filesystem, because a receipt on one machine is not a
  receipt. Its scope widened from five hand-picked directories (327 of 575
  markdown files) to all of docs/, which surfaced a live retracted score a
  three-week-old retraction had never reached.
• Interception seams reach the strategy loops, plugins carry interceptors
  with three bundled policy packs proving it, and /resync hot-swaps plugins,
  skills and agents into a live session. Plugin slash commands work in both TUI
  surfaces.
• The experiment toolkit (chimera/experiments/) — a run is structurally
  unable to write outside its own directory, and its result is validated against
  the same invariants the observatory enforces.
• glm-5.2 was missing from the provider catalog, and numpy was declared
  in no extra so the plus contract could not run.

Gates

Two postures, because the one CI runs is blind to what the other catches.

posture	result
uv sync --all-extras (highest)	11,018 passed / 79 skipped
CI posture (scripts/ci_posture_check.sh)	10,501 passed / 133 skipped

The gap is 517 tests. They are not missing from the suite — they are skipped,
each behind a pytest.importorskip for a dep CI does not install.

Both exclude the three documented live-infra files. The lower figure is not a
smaller suite — it is a less-tested one, which is the whole point:
tests/otter/test_server_tls.py sits behind
pytest.importorskip("cryptography") and had been raising TypeError since a
pidfile_prefix argument landed, invisible to both standard gates because
neither installs that dep. An all-extras run is now a required release gate
(playbook 14).

The one remaining local failure is test_validation_split, documented in
CLAUDE.md as env-sensitive locally and green in CI.

mypy clean across 736 source files cold-cache, ruff clean, all 7
trademark scrubs pass, canary sweep 7 pass / 0 BROKEN.

Known issue, found while cutting this release

The E2B backend does not work against e2b >= 2.x (#177). pyproject
declares e2b>=1.0, an unbounded range that now spans a breaking API change:
template, api_key and timeout moved off the Sandbox constructor onto
Sandbox.create(), so --env e2b raises TypeError at construction with a
current SDK. The reconnect path is affected too.

It is disclosed rather than fixed because verifying a two-generation
compatibility change needs real E2B infrastructure, and replacing a loud
TypeError with an unverified code path would be worse than the present
failure. Other cloud backends are unaffected.

Two blind spots stacked to hide it, both worth knowing about: the tests inject a
fake SDK at the module boundary (correct for CI, but a fake never has to
match the real signature), and mypy chimera/ is clean in CI because CI installs
no extras, leaving the call unchecked. It surfaced only under the new all-extras
gate.

Upgrading

No breaking changes to the public API.

If you have cited Chimera's mbpp-plus figure, update it: 99.7% was
base-graded; the plus-graded number is 84.9%.

If you have written a Benchmark subclass whose evaluate() falls back to a
non-empty-output heuristic, it will now be rejected by
tests/eval/test_no_length_grading.py. Return False when you cannot execute
the benchmark's tests — the resulting zero is legible as a harness gap; an
invented pass is not.

---

• fix(eval): mbpp-plus is 84.9% plus-graded, not 99.7% — and prose was resolving SWE-bench instances (#176)
• release: 0.9.2.2 — the gates (#178)

Full Changelog: v0.9.2.1...v0.9.2.2