Skip to content

Portable authenticated worker identity: semantic identity plus audit-only aliases for the primary-QRF worker binding (fixes #863) - #871

Open
MaxGhenis wants to merge 20 commits into
mainfrom
f1-portable-worker-identity
Open

Portable authenticated worker identity: semantic identity plus audit-only aliases for the primary-QRF worker binding (fixes #863)#871
MaxGhenis wants to merge 20 commits into
mainfrom
f1-portable-worker-identity

Conversation

@MaxGhenis

@MaxGhenis MaxGhenis commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Summary

Fixes #863. The authenticated US multispine pool loader bound the late primary-QRF worker to the build worktree's absolute interpreter path (argv_template[0] and interpreter.executable), so a byte-identical interpreter reached through another virtualenv path was refused and no sealed pool could be scored out of tree (the c-27 root-cause lane's STOP.md).

This PR versions the worker binding into a semantic identity and audit-only aliases:

  • Semantic identity binds what determines the computation: interpreter-byte digest, implementation and version/ABI, cache tag, canonicalized semantic pyvenv.cfg fields, worker-module source and transitive-import digests, the exact uv.lock digest, the installed-distribution RECORD digest, arguments after argv[0] (with argv[0] canonicalized to an interpreter placeholder), and the semantic fit controls (fit jobs, predict workers). Authentication compares this identity.
  • Audit aliases (sys.executable, sys.prefix, raw argv_template[0]) are recorded and never compared.
  • Legacy gate-failed pools can be scored from another worktree only with an explicit, plan-authorized compatibility attestation that binds the sealed manifest and H5 digests, the exact campaign tree, the lock, the installed environment/code digest, the recorded worker fields, the permitted two-alias mismatch, and the scoring-only purpose. The release loader ignores the attestation and refuses.
  • The primary execution-config schema and every enclosing identity version are bumped; US spec mirrors, checked-in YAML, and the coverage evidence are regenerated; exact-k release receipts carry the worker schema, digest, and audit receipt so the identity survives into build and release manifests.

Verification

Implemented by a sol build lane from the c-27 lane's FIX-PLAN.md §F1 with fail-before evidence recorded in docs/f1-portable-worker-identity/; the lane's Codex lane ran out before its final verification, which was completed by hand: the resolver-binding case in the spec-bundle identity allowlist, the builder fake's manifest-payload binding, and the resource-semantics and full-checkpoint identity pins were fixed and the coverage evidence regenerated (42154/42154 configuration fields, 41/41 inventory checks). CI on this branch is the verification of record.

Not in this PR

The immigration composition gate's anchor (#864) and the c-27 replay itself, which this PR unblocks.

Rebase onto post-#847 main and CI fixes (9/4)

Rebased onto main at 5e9ef2e (graph amendments 11–18). Pins that both sides moved were regenerated from the rebased tree: US spec_sha256 9db29b4d…, late_resource_semantics c7d26e86…, full_checkpoint 5deaf532…, and docs/evidence/spec-engine/us-f0-coverage.json (42154/42154 fields, 41/41 inventory checks). The materializer version 13 and pool manifest schema 10 bumps from this branch are unchanged.

Two defects CI found on the rebased head that the macOS lane run could not:

  • uv 0.12 writes version_info = 3.13 (major.minor) to pyvenv.cfg for uv-managed interpreters; the parser demanded a triplet, so every uv sync job failed at _canonical_pyvenv_config. It now accepts major.minor or major.minor.micro, requires the declared prefix to match the running interpreter, and canonicalizes to sys.version_info[:3], so the semantic identity does not depend on which uv wrote the venv. Verified locally: uvx uv@0.12.9 venv writes 3.13, uv 0.11.7 writes 3.13.9.
  • Wheels job: Duplicate source modules for worker identity: 'microcosm.build'. _module_source_index now walks the namespace search locations in import order and lets a later root shadow an earlier one only with byte-identical source (an installed wheel next to its checkout, lib64 beside lib); differing shadowed source still refuses, naming both paths.

Tests cover both accept paths and every refusal (test_worker_source_index_accepts_a_byte_identical_shadow_root, ..._rejects_a_shadow_root_with_different_source, test_canonical_pyvenv_config_accepts_uv_major_minor_and_triplet_versions, ..._rejects_versions_that_are_not_the_interpreter).

Two further omissions CI surfaced on the rebased head, both fixed: us_runtime/worker_identity.py is now classified as a reviewed non-registry module in the spine-blindness registry (the pool-build tool's pinned runtime import graph is 70 modules), and the release-gate preflight test's fake AuthenticatedPoolH5 binds manifest_payload_sha256, which the release receipt re-derives from the manifest it is handed.

Digest portability. Linux CI computed different late_resource_semantics and full_checkpoint inventory digests from the macOS pins. This branch had narrowed inventory_coverage._without_operational_bindings from stripping the whole worker_execution subtree (main's rule) to stripping only its audit aliases, on the premise that the new semantic identity is spec evidence. It is not: it hashes the interpreter binary, ABI, pyvenv.cfg, the installed distributions' RECORD files and the resolved CPU count, all of which legitimately differ between platforms and authenticate an environment for replay. Main's rule is restored, the two digests re-pinned from the stripped receipts, and test_operational_free_digest_ignores_the_worker_execution_binding pins the invariant (a macOS binding, a Linux binding and no binding digest identically).

Design note for review: the semantic identity is portable across interpreter paths (the #863 ask) but remains bound to one machine's interpreter bytes and installed distributions. A c-27 replay on a different box therefore authenticates through the legacy attestation path or needs a looser projection; this PR does not decide that.

MaxGhenis and others added 17 commits September 3, 2026 22:50
…he builder fake binds its manifest payload, re-pin the resource-semantics and full-checkpoint identities, regenerate coverage evidence
…v 0.12's major.minor pyvenv version

CI failed on the rebased head in two ways the local macOS run could not show:

- uv 0.12 writes `version_info = 3.13` (major.minor) to pyvenv.cfg for
  uv-managed interpreters; the parser demanded a triplet. It now accepts
  major.minor or major.minor.micro, requires the declared prefix to match the
  running interpreter, and canonicalizes to sys.version_info[:3], so the
  semantic identity does not depend on which uv wrote the venv.
- The wheels job reported `Duplicate source modules for worker identity:
  'microcosm.build'`. The namespace index now walks the search locations in
  import order and lets a later root shadow an earlier one only with
  byte-identical source (an installed wheel next to its checkout, lib64 beside
  lib); differing shadowed source still refuses, naming both paths.

Tests cover both accept paths and every refusal.
@MaxGhenis
MaxGhenis force-pushed the f1-portable-worker-identity branch from 97b44d3 to 5e6c895 Compare September 4, 2026 03:25
Both this branch and #847 moved the US spec digest, the late resource
semantics and full-checkpoint identities, and the coverage evidence; the
rebase kept main's values, which this tree no longer produces. Regenerated
from the rebased tree: spec_sha256 9db29b4d…, late_resource_semantics
c7d26e86…, full_checkpoint 5deaf532…, us-f0-coverage.json (42154/42154
fields, 41/41 inventory checks). Restores this branch's materializer 13 and
pool manifest schema 10 assertions that the conflict resolution had dropped.
…; bind the preflight fake's manifest payload

CI on the rebased head found two omissions the lane's local run did not
cover: the new us_runtime/worker_identity.py module was unclassified in the
spine-blindness registry (and pushed the pool-build tool's pinned runtime
import graph from 69 to 70 modules), and the release-gate preflight test's
fake AuthenticatedPoolH5 did not carry manifest_payload_sha256, which the
release receipt now re-derives from the manifest it is handed.
…gests

CI on Linux computed different late-resource-semantics and full-checkpoint
digests from the macOS pins. The branch had narrowed the pre-digest
stripping from the whole `worker_execution` subtree to its audit aliases,
on the premise that the new semantic identity is spec evidence; it is not:
it hashes the interpreter binary, ABI, pyvenv.cfg, the installed
distributions' RECORD files and the resolved CPU count, all of which
legitimately differ between platforms and authenticate an environment for
replay rather than describe the spec. Restore main's rule (strip the whole
subtree, plus audit aliases and receipt self-hashes), re-pin the two digests
from the stripped receipts, regenerate the coverage evidence, and pin the
invariant with a test that a macOS binding, a Linux binding and no binding
digest identically.
@MaxGhenis
MaxGhenis marked this pull request as ready for review September 4, 2026 09:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Authenticated pool loader binds the build worktree's absolute interpreter path; out-of-tree scoring of a sealed pool is impossible (c-27 lane F1)

1 participant