Separate infrastructure and Agent evaluation lifecycles

[Chloride233]

Summary

• bound infrastructure readiness and Agent/MCP startup to one 60-second pre-model window, keep the 90-second evaluation window separate, and use a 170-second Modal sandbox envelope with finalization margin
• make host startup condition-aware for Claude Code and Codex, including one bounded all-tools-missing startup retry
• harden the evaluation-only validated-SQL ledger with atomic concurrent writes and a sticky failure marker that remains fail-closed across transport errors and timeouts
• require validation-ledger and marker failures to block infrastructure, scoring, batch health, calibration authorization, and Pilot dispatch
• move the current calibration evidence contract to schema v7 / 170 seconds, reject non-finite budgets and ambiguous JSON, and publish success/failure artifacts atomically
• stop public workflow uploads of raw Inspect logs or sealed BIRD data; upload only sanitized evidence after its scan succeeds
• add strict public-artifact profiles for formal, report, readiness, canary, calibration, and Pilot evidence: exact terminal or ordered-prefix inventories, typed schemas, duplicate-free finite canonical JSON, private-content rejection, cross-file invariants, and deterministic Markdown reconstruction
• pin the public-artifact verifier to immutable commit 0389c4bd34597f06676d4c560f5e7683fd2b20c9 in a separate checkout, so frozen Pilot commits do not need to contain or select the verifier implementation
• pin all 32 remote GitHub Actions dependencies across CI and formal workflows to full commits corresponding to their official release versions, with a repository-wide regression test that rejects mutable tag, branch, and Docker uses: references
• add a tested schema-v2 append-only reserved-upper campaign ledger primitive with exact micro-CNY accounting, run-level replay rejection, canonical input-bound receipts, unique Git commit ownership, bounded non-force ref CAS retries, repository-ID binding, a caller-pinned empty genesis, per-reservation append-only history verification before each write, exact readback recovery for ambiguous initialization/ref-update results, and pre-API nonce validation
• add a fail-closed admission layer that derives a fixed repository/workflow/job/run identity, admits only readiness/CNY 2.10, calibration/CNY 4, and Pilot/CNY 20, verifies a clean actual checkout HEAD including special Git index flags, binds calibration/Pilot reservations to the same verified frozen-input lock bytes and every consumed database, rejects invalid mode/lock combinations, removes the caller-defined reservation CLI, and returns only after a fresh reservation is verified against the live ledger head
• correct Canary input-token weighting and account for one complete response after the Inspect stop line; the conservative CNY 3.05314 upper now rejects the obsolete CNY 2.25 approval before Inspect starts
• fail closed on reruns, failed evidence scans, missing artifacts, cleanup failures, and any paid path without an authoritative atomic campaign reservation
• serialize all five paid jobs through one non-cancelling repository-scoped single-flight group; this reduces concurrent execution risk but does not replace atomic budget reservation

Current operating state

• no paid Modal or model run was launched by this change
• the full 80-run Pilot was not launched
• no v11 frozen archive was uploaded
• formal-evaluation-report, formal-evaluation, formal-pilot-canary, formal-pilot, and prepare-bird-subset are manually disabled on GitHub
• their paid/report jobs also begin with an unconditional block in the current workflow definitions
• the campaign ledger and admission layer are tested primitives, not a production authority; no paid workflow imports or consumes them
• the admission layer now verifies the actual clean checkout HEAD plus the frozen-input lock and consumed databases; its Mapping inputs and fixed-directory ownership are still trusted only inside a clean workflow-owned job before evaluated code and are not a signed identity proof
• standalone receipt verification is a repeatable snapshot check, not a consume-once execution ticket; the one-shot reservation entry point must be used before any future paid command
• re-enabling paid evaluation still requires a protected authoritative ledger genesis and a clean workflow handoff that populates the fixed checkout/frozen-input paths and invokes admission before evaluated code; genesis ancestry alone cannot detect a force-reset to a formerly valid prefix, so the ledger ref must forbid force-push/delete or use an independent monotonic checkpoint
• the pinned artifact-verifier checkout controls validator code provenance but shares the job runner; it is not adversarial process isolation from an untrusted evaluated commit
• existing GitHub Environment secrets were not deleted or modified
• released runtime code under src/joinlint/ is unchanged; these changes remain inside the evaluation harness, tests, documentation, and workflows

Verification

• both clean GitHub CI runs for commit 42d3785 completed with 550 passed, 12 skipped, one upstream warning, Ruff green, and the million-row benchmark gate green while executing the pinned Action commits
• 12 focused Action-pinning and affected workflow tests passed locally
• 131 focused campaign-ledger and admission tests passed locally; the campaign-ledger file alone completed 86 tests
• 206 focused campaign-ledger/admission/frozen-input/calibration tests passed locally after the final index-flag and single-read input-lock changes
• the macOS full rerun stopped at the known third-party import hang before collection; both clean GitHub CI runs completed the full collected suite
• actionlint passed for all six repository workflows
• git diff --check passed
• repeated ledger concurrency, real STDIO marker, timeout, and cross-process stress checks passed during review
• two zero-cost temporary GitHub ref exercises passed and were cleaned up: different concurrent reservations produced one authorization plus one budget rejection, and identical concurrent reservations produced one authorization plus one replay rejection

Remaining blockers before re-enable

• approve public ledger disclosure and freeze the production campaign ID, budget, opening reserved balance, and pinned genesis; protect main and the dedicated ledger ref from force-push/delete
• wire a clean workflow-owned one-shot entry point that reads GitHub default variables and event inputs directly, populates the fixed checkout/frozen-input paths, reserves the fixed mode upper, live-verifies the fresh receipt, and only then launches the one mapped paid command
• refreeze the one-task Canary budget or add a provider-enforced pre-response context limit; the current conservative upper intentionally exceeds its obsolete approval
• approve each evaluated commit as a runner trust input or move artifact validation to a clean verifier runner; the separate pinned checkout alone is not a hostile-code sandbox
• implement the report source-run metadata verifier and complete the trusted frozen-input/blind-review producer boundary before private evidence can be consumed
• freeze reliable upper bounds and access-controlled output handling for legacy formal evaluation and BIRD preparation
• land the hardened workflow definitions on the default branch before enabling any workflow; enabling first would expose the older default-branch definitions