v0.9 eval/report: run full gate suite and publish claim audit

[AbdelStark]

Parent

#385

What to build

Evaluate the verified v0.9 checkpoints across the full gate suite, publish tracked eval artifacts, and write the final report/cards/artifact index. This is the issue that decides whether the v0.9 claim opens or remains diagnostic.

Acceptance criteria

• retrieval, surprise, probe, held-out p_pass, and WS-D rerank evals run for both seeds with required parent manifests.
• HumanEval and MBPP-Plus downstream tables include CodeLeWM, no-action, shuffled-action, lexical/static, LLM-order, random, confidence intervals, and claim decisions.
• tracked docs/benchmark/v0_9/ artifacts include manifests, reports, and score rows required to reproduce the report.
• dataset/model cards and public artifact index list exact artifact ids, checkpoint SHA-256 values, run ids, and claim posture.
• final report explicitly states whether v0.9 opens or keeps closed the public downstream claim gate.
• validation includes full local tests, CI, parent-aware manifest verification, checkpoint inspection, and secret scans.
• tracker [TRACKER] v0.9 data/eval repair for cross-benchmark correctness claims #385 is updated and closed only after this issue merges to main.

Blocked by

#391.

[norika1207-lab]

For the final claim audit, I would make the “claim posture” table brutally explicit.

Useful columns:

• benchmark / split
• seed
• checkpoint hash
• decoding settings
• pass/fail metric with CI
• baseline deltas
• artifact id needed to reproduce
• decision: open claim, diagnostic only, or closed

The key is to separate “interesting internal signal” from “public downstream claim.” A model can show a real improvement on retrieval/probe gates and still fail the downstream claim gate if MBPP-Plus or HumanEval does not reproduce cleanly.

I would also preserve the no-action, shuffled-action, lexical/static, and random baselines as first-class rows, not appendix material. Those controls are what prevent the report from over-crediting the learned ordering when a simpler heuristic explains the gain.