Freeze the V3 targeted confirmation protocol - #3527
Conversation
Freeze a blind 80-report evaluation of V3 against V2: eight modes, two conditions, and five fresh replicates per cell. Seal condition maps, prompts, fixtures, frozen skill packages, rubrics, allowed authority, expected atoms, and report and scoring schemas before generation begins. Retain mode-level analysis for symbolic release domains, nonlinear policy composition, configuration products, existential unsoundness certificates, whole-execution behavioral claims, positive multi-version proofs, abstraction-design firewalls, and regression breadth. Require candidate V3 to pass every required atom in every replicate and produce zero hard errors, TCB or authority defects, proposal laundering, semantic noncompletion, and scope or budget failures. Add an append-only event ledger and explicit attempt lifecycle so generation, validation, retries, blind scoring, adjudication, and unblinding remain auditable. Record the independent package and protocol reviews. At freeze time the report count is zero, so later results cannot have influenced the question or success criteria. gherrit-pr-id: G5k3oylk4nmllritz5hdpiksvqbp24ffs
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## Gd5c25nyjtgu5tav2bs53xvoz525b6kpe #3527 +/- ##
==================================================================
Coverage 91.85% 91.85%
==================================================================
Files 20 20
Lines 6093 6093
==================================================================
Hits 5597 5597
Misses 496 496 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6880d2b469
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| PACKAGES = { | ||
| "v3": { | ||
| "path": "frozen-packages/668f70202c7bc8f23f7f894fb784a9629fd292c7f6fe69ede815b0e4c10137bf", | ||
| "tree": "668f70202c7bc8f23f7f894fb784a9629fd292c7f6fe69ede815b0e4c10137bf", |
There was a problem hiding this comment.
The frozen digest does not match the committed package under the protocol's own tar_tree_digest algorithm: GNU tar 1.35 produces 2d851278495f916b1fcbc147a9c3c1c24f80387480b1067a56ed59a1c032b1e3, not 668f7020.... Consequently, protocol.py verify-static --locked, self-test, and every operational command that calls validate_static() stop immediately with ValueError: v3 package digest mismatch, so collection cannot begin. Regenerate the frozen tree identities using the committed digest implementation (and verify the V2 and target constants, which are checked the same way) before locking the run.
Useful? React with 👍 / 👎.
| if tar_tree_digest(packet / "target") != target_row["tree_sha256"]: | ||
| raise ValueError(f"adjudication target lost frozen identity: {mode}") |
There was a problem hiding this comment.
When a mode has any scorer disagreement, the source scoring packet has already passed through make_read_only, so its target files and directories have modes 0444/0555; shutil.copytree preserves those modes here. Because tar_tree_digest includes permission bits while target_row["tree_sha256"] identifies the original 0644/0755 fixture, this comparison fails even after the frozen digest constants are corrected, and no required adjudication packet can be built. Normalize the copied modes before this check or use the mode-independent byte-tree identity already recorded for packets.
Useful? React with 👍 / 👎.
| "decision": { | ||
| "enum": ["PASS", "FAIL", "PRESENT", "ABSENT"] |
There was a problem hiding this comment.
For an adjudication containing non-atom disagreements, this supplied schema accepts all four decision strings for every field, while validate_adjudication() accepts PASS/FAIL only for atom:* and PRESENT/ABSENT for every other field. An adjudicator can therefore emit a schema-conforming result such as {"field":"hard_error:G1","decision":"PASS"}, only for record-adjudication to reject it; under the frozen rerun policy that output is non-rerunnable and invalidates the entire run. Encode the field-dependent decision vocabulary in the schema so the evaluator-facing contract matches the recorder.
Useful? React with 👍 / 👎.
2 participants
Freeze a blind 80-report evaluation of V3 against V2: eight modes, two
conditions, and five fresh replicates per cell. Seal condition maps, prompts,
fixtures, frozen skill packages, rubrics, allowed authority, expected atoms,
and report and scoring schemas before generation begins.
Retain mode-level analysis for symbolic release domains, nonlinear policy
composition, configuration products, existential unsoundness certificates,
whole-execution behavioral claims, positive multi-version proofs,
abstraction-design firewalls, and regression breadth. Require candidate V3 to
pass every required atom in every replicate and produce zero hard errors, TCB
or authority defects, proposal laundering, semantic noncompletion, and scope
or budget failures.
Add an append-only event ledger and explicit attempt lifecycle so generation,
validation, retries, blind scoring, adjudication, and unblinding remain
auditable. Record the independent package and protocol reviews. At freeze time
the report count is zero, so later results cannot have influenced the question
or success criteria.
Latest Update: v2 — Compare vs v1
📚 Full Patch History
Links show the diff between the row version and the column version.
⬇️ Download this PR
Branch
git fetch origin refs/heads/G5k3oylk4nmllritz5hdpiksvqbp24ffs && git checkout -b pr-G5k3oylk4nmllritz5hdpiksvqbp24ffs FETCH_HEADCheckout
git fetch origin refs/heads/G5k3oylk4nmllritz5hdpiksvqbp24ffs && git checkout FETCH_HEADCherry Pick
git fetch origin refs/heads/G5k3oylk4nmllritz5hdpiksvqbp24ffs && git cherry-pick FETCH_HEADPull
Stacked PRs enabled by GHerrit.