Repository navigation
Context Fidelity: provisional evaluation results
Pre-releaseContext Fidelity evaluates whether an agent reports its completed work accurately after context compression. This prerelease contains synthetic coding-task evaluations generated with Qwen3-4B-Instruct-2507, plus frozen inputs and native Inspect logs.
The held-out run completed 24 histories and 192 reports without a technical failure. The compact evidence context did not demonstrate the predicted benefit over ordinary summaries. Human review is pending: 49 reports have definite errors and 143 primary verdicts remain unresolved. The evidence-restoration diagnostic had no eligible cases and did not run.
Assets:
context-fidelity-evidence-2026-09-12.zip: completed development and held-out runs, exact frozen held-out inputs, audits, analysis, plots, and scored logs. Extract at the repository root for offline reanalysis. SHA256:f981e1d4db6d2a5a3fe4b0e45b3367295222151204e20b5c0d5d44fd51d0c4dc.context-fidelity-inspect-heldout-2026-09-12.zip: Inspect's portable viewer with 192 held-out scored logs. Hosting requires HTTP Range support. SHA256:bb23c14572a023b81e94d0c48f8d4c3a62fb10a409488495b5dcf4d3d24f6f96.
These archives are unblinded: they include scores, examples, and review keys. Give a fresh reviewer only the blinded items and rubric until their judgments are saved.
Generation used source 8f8fc08; final analysis source f8fa80c reproduced all 192 cells, arm counts, planned pairs, effects, and intervals in a fresh directory without model calls. The same locked Python environment was used. This does not constitute human review or a fresh dependency installation.
See docs/reproduction.md, docs/review-rubric.md, and docs/experiments/results.md at this tag. The implementation remains subject to PR review; this is a provisional research release.
For human review, use the review handoff and tracking fix in PR #3. Commit 56f63bb retains later prose-review labels for malformed responses; the original tagged scorer discarded those labels. The fix leaves all 192 unreviewed verdicts and effect estimates unchanged. The release tag and both assets remain unchanged.