v0.0.16 — 8-model matrix, Vera 0.1.x compatibility, reasoning-mode benchmarking

Highlights

Benchmark-matrix release. Expands to an 8-model, 3-tier matrix across three providers, adds AILANG as a fifth comparison language throughout the charting path, and restores compatibility with Vera 0.1.x (which had left vera-bench validate at 22/60 failures on CI).

Tier	Anthropic	OpenAI	Moonshot
ceiling	claude-fable-5	openai-pro/gpt-5.6-sol	—
flagship	claude-opus-4-8	gpt-5.6-sol	moonshot/kimi-k3
workhorse	claude-sonnet-5	gpt-5.6-terra	moonshot/kimi-k2.6

The new openai-pro/ routing gives a controlled reasoning-budget comparison: the same model at reasoning.mode standard vs pro, so deliberation is the only variable between the two entries.

Also new: scripts/preflight.sh, a pre-sweep gate that verifies every configured model id, provider auth, the request parameters each model accepts, and all four toolchains — at one problem per check, before a sweep spends hours and real money.

A note on what this release is

This is a methodology and harness release, not a results release. It contains no benchmark results — the v0.0.16 sweep has not been run yet. The assets/results-graph.png in the repository still shows v0.0.7 data, and the README results narrative is still the v0.0.7 one. Both are updated in a follow-up once the sweep completes.

Several fixes here were found by running the harness against live APIs rather than by CI, and a few of them were bugs that produced plausible-looking wrong numbers rather than failures — an un-run comparison language rendering as a Vera win, a partially-failed sweep reporting an excellent score, a compiler crash scored against the model. Those are detailed in Fixed below.

---

Added

• Per-test subprocess-failure diagnostics for the Aver evaluator (#72). The _evaluate_aver_code per-test loop previously continued silently on timeout and non-zero exit — a model whose type-correct Aver crashed at runtime was indistinguishable in JSONL from one with wrong logic (error_message=None, check_pass=True, run_correct=False). The loop now captures the first failing test's diagnostic (stderr-or-stdout coalesce, 400-char truncation, explicit exit-code marker when silent) into error_message, matching the AILANG evaluator's existing pattern. Both evaluators now share a _first_run_error formatting helper.

• openai-pro/ model-routing prefix — reasoning-mode benchmark entries. openai-pro/gpt-5.6-sol runs Sol at reasoning.mode: pro as a distinct benchmark entry from default-mode gpt-5.6-sol: same model, different reasoning budget — a controlled comparison of whether Vera's contracts do the work the reasoning budget otherwise would. Mechanics:

  • create_client routes the prefix to OpenAIClient(..., reasoning_mode="pro"); the runner needs no changes (the complete() Protocol carries no per-call config by design).
  • Both Sol arms run through the Responses API, which is the only endpoint carrying reasoning.mode, and both get the same 16k output floor — an unequal budget would be a second variable in a comparison that claims to have one. gpt-5.6-terra is not half of the pair and stays on Chat Completions.
  • JSONL rows are self-describing: model: "<api-model>#pro" and "<api-model>#standard" respectively. Result filenames are already distinct (built from the CLI model string).
  • The response echoes the effective mode, and a mismatch — or an absent echo — raises rather than being recorded as a pro result.
  • OpenAIClient now sends max_completion_tokens (the GPT-5.x reasoning families reject the legacy max_tokens kwarg).
  • scripts/run_full_benchmark.py _detect_provider recognises the prefix (→ OPENAI_API_KEY).

• Prompt-cache instrumentation for OpenAI and Moonshot clients (#61). Both providers now cache automatically server-side (OpenAI ≥1024-token prompts; Moonshot longest-prefix matching — a request cache-hits only when it shares a prefix with a prior request) — the work is routing and observability, not cache management:

  • New cached_tokens field on LLMResponse and ProblemResult (JSONL rows): the cache-hit portion of input_tokens. Anthropic reports cache_read_input_tokens; OpenAI-compatible providers report usage.prompt_tokens_details.cached_tokens (read via a shared _openai_cached_tokens guard that tolerates absent / None / non-int fields). Sweep cost analysis can now compute cache-hit rates per provider directly from result files.
  • OpenAI requests carry a stable prompt_cache_key (SHA-256 of the system prompt, via extra_body) so same-prefix requests route to the same cache shard — recommended by OpenAI for the GPT-5.6 family. During sweeps the ~28k-token SKILL.md prefix gets its own shard per language-mode.
  • Moonshot sends no cache parameter — their Context Caching is fully automatic with no routing key (their earlier explicit X-Msh-* header scheme is obsolete).

• OpenAI and Moonshot clients hardened to the OpenRouter error standard: AuthenticationError → EnvironmentError (abort the run — retrying 60 problems on a bad key is waste), RateLimitError / BadRequestError / APIStatusError → clean RuntimeError messages, and explicit errors on empty choices / empty content (previously returned text="" and the harness blamed the model for "did not define entry point").

Changed

• Minimum openai SDK raised from >=1.50 to >=2.45. The openai-pro/ path calls responses.create with prompt_cache_key and reasoning={"mode": "pro"}; the declared floor supported none of the three. Verified against the wheels rather than inferred: responses.create arrived in 1.66.0, prompt_cache_key in 2.0.0, and reasoning.mode in 2.45.0. 2.0 would in fact work at runtime — maybe_transform forwards keys a TypedDict does not declare, and the response model is pydantic extra="allow" — but relying on that makes a silent failure possible: if either behaviour changed, the "pro" entry would run in standard mode, the effective-mode guard would no-op, and the reasoning chart would compare a model against itself while looking entirely normal.

Fixed

• A sweep with most calls failing could report an excellent score (#95). run_correct is measured over run_eligible — problems whose attempt compiled — so rows that never reached the compiler (API errors, auth rejections, timeouts) left the denominator entirely. Verified: 40 API failures plus 20 successes reported run_correct = 100%, and nothing in the output said so; check@1 merely dropped, which is a normal result for a model struggling with Vera. The rate itself is unchanged, since altering it would break comparability with published results. Instead BenchmarkMetrics gained run_eligible and errored, the CLI summary now prints the graded denominator whenever it differs from the problem count plus an errored: N/60 row in red, and extract_data treats a file with nothing gradeable as missing rather than plotting it as a genuine 0% — the same conflation the missing set already prevents for absent files, one layer down.

• vera run failures were indistinguishable from wrong answers. The per-test loop in _evaluate_vera_code discarded the exit code and stderr on failure, and swallowed every exception through a bare except Exception with no message — so a compiler crash was recorded as error_message=None, check_pass=True, run_correct=False, byte-identical to a model writing a wrong program. The #72 diagnostics work reached the Aver and AILANG evaluators but never the Vera path, which is the headline number. Now captures the first failure via a _vera_run_error sibling of _first_run_error, including signal detection: subprocess reports a signal death as a negative return code, so a compiler SIGBUS arrives as exit_code == -10 and is now named as such. Not hypothetical — vera SIGBUS'd repeatedly on 2026-07-23 (aallan/vera#1145), and every one of those would have been scored against the model.

• preflight.sh now exits non-zero when the gate fails. It could print *** pro may be SILENTLY IGNORED *** — the one finding that invalidates the reasoning slide — and still exit 0, because the S2 and S3 analyses printed their verdicts from inside Python heredocs without touching the pass/fail counters. A chained preflight.sh && run_full_benchmark.py would have sailed straight past it. S2's verdict now feeds the counters and the script ends on its own tally.

• Both Sol arms now send store=False. The Responses API defaults to store=True where Chat Completions does not persist at all, so the pro routing silently introduced 30-day server-side retention of every prompt and completion. Beyond the data-handling question, retained content could feed cross-run caching or personalisation — and a benchmark whose second run is informed by its first is not measuring what it claims to.

• Claude Fable 5 returned no code at all. AnthropicClient.complete read response.content[0].text, but models with extended thinking return ThinkingBlock entries ahead of the TextBlock — so every Fable 5 call died with 'ThinkingBlock' object has no attribute 'text', recorded as an API-error row with no generated code. The whole fable tier of the v0.0.16 matrix would have come back empty. A new _anthropic_text helper selects blocks by .type == "text" and joins them, ignoring thinking and redacted-thinking blocks. Caught by smoke S1 (2026-07-23); the pre-existing unit test missed it because a bare MagicMock auto-supplies any attribute, including the .text the real ThinkingBlock lacks.

• openai-pro/ went to the wrong endpoint — now uses the Responses API. The reasoning tier was passed as extra_body={"reasoning": {"mode": "pro"}} on Chat Completions, which rejects it with 400 Unknown parameter: 'reasoning', so every Sol@pro call failed. Per OpenAI's reasoning guide, mode and effort are independent axes — mode selects standard vs pro execution, effort controls how much reasoning happens — and reasoning.mode exists only on the Responses API (Literal["standard", "pro"] in openai-python 2.47). Pro is therefore not expressible on Chat Completions at all: reasoning is rejected as unknown, and reasoning_effort="max" is rejected too (gpt-5.6-sol accepts only none/low/medium/high/xhigh there — max is Responses-only). A client with a reasoning mode set now routes to responses.create with instructions/input/ max_output_tokens and reads usage from input_tokens / output_tokens / input_tokens_details.cached_tokens. Two new guards: an unknown mode raises at construction, and a response whose echoed effective reasoning.mode differs from the requested one raises rather than being recorded as a pro result — a silent downgrade would turn the headline pro-vs-default comparison into a model compared against itself. Confirms the CodeRabbit finding on #92 that was declined pending evidence.

  Both Sol entries are pinned to the Responses API via RESPONSES_API_MODELS, with the default entry sending an explicit reasoning.mode: "standard". Pro mode exists only on Responses, so leaving the default arm on Chat Completions would vary endpoint and mode together and the reasoning-budget comparison could not attribute its delta to deliberation. The 16000-token output floor applies to both arms for the same reason. gpt-5.6-terra is a separate tier row rather than half of a controlled pair, and stays on Chat Completions. Sol rows now report model as gpt-5.6-sol#standard / gpt-5.6-sol#pro; charts key on filenames, which are unchanged.

• Long result paths wrapped mid-token in console output. Output: <path> was printed through rich, which wraps at the console width (80 when not a tty — CI, and any sweep log piped to a file), breaking paths across lines mid-word and making them un-greppable and un-copy-pasteable. Now printed with soft_wrap=True. This was also failing test_run_ailang_full_path_success on main.

• aver/ailang version-probe timeouts reported the wrong cause. A TimeoutExpired from --version was either unhandled (ailang) or folded into the not-found branch (aver), which advised reinstalling a compiler that is already installed. Both now report the timeout distinctly.

• AILANG version parsing corrupted result filenames. run --language ailang built its output filename from the compiler's raw --version stdout. ailang --version prints a seven-line banner (version, commit, full SHA, build stamp, blank, tagline, copyright), and the old .strip().replace("ailang ", "") matched nothing in it — the binary prints AILANG v0.30.0, capitalised. The entire banner therefore landed in the filename, producing a 216-character name containing embedded newlines and colons. macOS creates such a file happily, so this failed silently: every shell glob in the sweep runbook, the file_prefix matching in plot_results.py, and the release tarball would all have missed the AILANG results. Never caught because baselines --language ailang uses a separate code path that does not embed versions. Both the Aver and AILANG probes now share a _parse_version_banner helper that takes the first line and extracts the numeric version token, falling back to "unknown" (which callers already treat as "omit the version from the filename"). The AILANG probe also now catches TimeoutExpired, matching the Aver probe.

• Vera 0.1.x compatibility for the problem set and canonical solutions. Vera moved from v0.0.177 to v0.1.6+ (the v0.1.0 bug burndown plus the 0.1.x line) and three compiler changes broke vera-bench validate (22/60 failures). All three are now resolved; the benchmark validates 60/60 against Vera HEAD (v0.1.6, f6f586b0):

  • VB-T1-002 renamed clamp → clamp_to_range across the problem JSON (signature, entry_point, prose) and all five canonical solutions (TypeScript as clampToRange). Vera 0.1.x makes clamp a built-in (spec §9.6) and rejects redefinition with E151 — the problem was unwritable as specified. Problem id is unchanged (VB-T1-002).
  • State-handler solutions updated for intrinsic-hybrid clause semantics (vera#1003, shipped in Vera 0.1.x; see also vera#976/#973/#988). Handler clause bodies used to be type-checked but never executed — the runtime always used builtin state-cell semantics. Now clause bodies execute, and the old put(@Int) -> { resume(()) } with @Int = @Int.0 idiom resolves @Int.0 to the handler state, so the "update" is state = old-state — a silent no-op that made VB-T5-001 return 0 instead of 3 and broke 3 of 4 VB-T5-006 test cases. All four State solutions (T5-001, T5-004, T5-006, T5-009) drop the with clause; the intrinsic store threads state natively (the canonical form in SKILL.md's run_counter example). T5-004/T5-009 passed tests only incidentally and carried the same latent no-op.
  • vera_verify_tier1 flipped to false on 19 problems (T1-008, T1-010, T2-001, T2-010, T2-014, T3-001..005, T3-008, T3-012, T4-001, T4-004, T4-005, T4-007, T4-008, T4-010, T5-004). Vera 0.1.x auto-synthesises int_overflow / nat_to_int_coerce proof obligations on all integer arithmetic; on unbounded Int/Nat inputs these are legitimately unprovable at Tier 1 (proving them would require adding range preconditions — changing the published contracts of 19 problems). The obligations are checked at Tier 3 (runtime) instead, which is the correct classification. This flag gates only canonical validation; LLM verify@1 scoring is computed from verify_pass and is unaffected.

Compatibility note

Requires Vera ≥ 0.1.x (the intrinsic-hybrid handler semantics and clamp built-in). Canonical Vera solutions no longer compile/run correctly on Vera ≤ 0.0.x: the renamed clamp_to_range does not collide there (fine), but the State solutions rely on the new clause-execution semantics. Aver baselines verified unchanged on Aver 0.27.1 (the v0.0.14 Int.div migration was forward-compatible throughout); AILANG baselines verified on AILANG v0.30.0.