Skip to content

Authenticated pool loader binds the build worktree's absolute interpreter path; out-of-tree scoring of a sealed pool is impossible (c-27 lane F1) #863

Description

@MaxGhenis

What is wrong

The authenticated US stacked-pool scoring loader refuses a sealed manifest whenever the scorer runs from a different worktree than the build, even when the interpreter is byte-identical. The late primary-QRF worker binding stores argv_template[0] and interpreter.executable as str(Path(sys.executable)) — the build worktree's absolute .venv/bin/python — and the loader requires exact equality of the whole binding against the live _late_primary_qrf_worker_execution_binding().

Reproduced 2026-09-03 by the sealed c-27 root-cause lane (run 20260903-073827-c27-root-cause, sol) against the sealed candidate-26 pool from a standalone clone at campaign b8819b3f with origin/main pinned to 6022e4f5:

ValueError: US stacked pool manifest .../candidate-26/pool/pool.manifest.json late producer 'primary_puf_qrf': late primary-QRF worker binding changed.

Only two fields differed:

  • sealed: argv_template[0] and interpreter.executable = /Users/maxghenis/PolicyEngine/_worktrees/microcosm-c26-build/.venv/bin/python
  • replay: both = /private/tmp/microcosm-c27-rootcause/.venv/bin/python

Every other compared field matched: resolved interpreter ~/.local/share/uv/python/cpython-3.14.4-macos-aarch64-none/bin/python3.14, CPython 3.14.4, module, remaining arguments, fit jobs -1, predict workers 18.

Code: packages/microcosm-build/src/microcosm/build/us_runtime/stacked_spine.py — campaign b8819b3f lines 5724–5727 and 6049–6121; origin/main@6022e4f5 lines 5592–5595 and 5907–5979. Loader boundary: us_runtime/h5_io.py 823–886 (authenticated-manifest boundary) and 1053–1090 (downstream late-DAG validator).

Why it matters

Any out-of-tree scoring of a sealed pool — an independent battery replay, a root-cause diagnosis, a referee's re-score — is impossible without either rebuilding in the original worktree path or bypassing authentication. The c-27 diagnosis lane therefore returned every H5-table result as not-evaluated; it correctly used no retry and no alternate loader.

Fix (from the lane's FIX-PLAN.md, section F1, unconditional)

  • Bump the primary execution-config schema and every enclosing identity version that hashes it.
  • Separate audit-only launcher aliases from canonical semantic identity. Semantic identity binds: interpreter-byte SHA-256, implementation, version/ABI, cache tag, canonicalized semantic pyvenv.cfg fields (absolute sys.prefix kept as an audit-only alias), worker-module source and transitive-import digests, the exact approved uv.lock, the installed-distribution/RECORD digest, arguments after argv[0], and the semantic environment/fit controls. Canonicalize argv[0] to an interpreter placeholder.
  • Legacy gate-failed pools (scoring only): no implicit exception from alias equality. Accept a relocation only with an externally plan-gated compatibility attestation binding the sealed manifest/H5 digests, exact campaign tree, lock, installed transitive environment/code digest, recorded worker fields, the permitted two-field mismatch, and the scoring-only purpose; the mismatch set must be exactly those two alias fields and every semantic field equal. Never extend this to a simulation-ready or release loader.
  • Regression: a scorer in a second worktree path with the same interpreter bytes loads; a scorer with a different interpreter byte hash, lock, or RECORD digest is refused.

Evidence

Sealed deliverables: _buildo-runtime/out/candidate-27/root-cause/deliverables/ (STOP.md, FIX-PLAN.md, FINAL-REPORT.md sha256 2f71c43cb68e2a942250b5158195eca7c17e0e1780fd264fec2490811dfa1c60), hashes in ~/chief-of-staff/state/subfleet/dispatch/c27-root-cause/OUTPUTS.json. Related: #857 (deny-list), #861.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions