Evaluate V1 legacy behavior and abstraction-design workflow - #3521
Conversation
Replay the predecessor pilot targets against the frozen V1 package and run a 54-report treatment/core-ablation study over nine abstraction-design modes. Preserve the fixtures, frozen packages, manifests, raw reports, blind scores, adjudications, and limitations needed to interpret both exploratory studies. The legacy replay recovered every known synthetic and historical defect and accepted every fixed control without a hard error. It improved historical version applicability, literal HasField contract discovery, and exposure of reconstructed proofs, while still missing an admissible indirect Copy and UnsafeCell derivation on the current-source challenge. The abstraction-design workflow produced the clearest measured improvement. No treatment report certified an unimplemented proposal, while 16 of 27 core-ablation reports did; treatment matched or exceeded every adjudicated mode and elicited parsimonious designs such as checked construction, real sealing, safe slice splitting, and receiver-bound lifetimes. The preregistered gates nevertheless failed. Four treatment reports used executions containing UB as defined behavioral counterexamples. Two of three treatment reports incorrectly proved a Rust-1.70 empty-slice pointer loop after promoting constructor or slice admission facts into a universal invariant; the third treatment report correctly falsified the evaluator's original oracle. These failures motivate V2's whole-execution verdict, exact-domain, boundary-case, and producer-quantifier rules. The results remain exploratory: isolation was procedural, model identity and sampling seed were unavailable, and the legacy replay was neither byte-identical to the pilot nor a statistical release evaluation. gherrit-pr-id: G3y45zv35fuuyeejc26bftqdd33lqz2oh
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## G6f4nmapfoz5ur4camwh2yhyazexc2lct #3521 +/- ##
==================================================================
Coverage 91.85% 91.85%
==================================================================
Files 20 20
Lines 6093 6093
==================================================================
Hits 5597 5597
Misses 496 496 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9da92525b7
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| Digests use GNU tar streams rooted at `.`, sorted names, timestamp zero, | ||
| numeric owner/group zero, and preserved contents/modes. Every opaque runtime | ||
| copy was verified against its mode digest before the first run. |
There was a problem hiding this comment.
On a normal checkout, the documented GNU-tar recipe cannot reproduce any of these digests because Git does not preserve the group/other permission distinction used when they were calculated. For example, tar --sort=name --mtime=@0 --owner=0 --group=0 --numeric-owner -cf - . over the committed treatment package produces 004f3f0c..., not d97b9ace...; the published value appears only after adding the undocumented normalization --mode='u=rwX,g=rX,o='. The fixture, report, and blind-score hashes have the same issue, so consumers cannot verify the claimed frozen identities from this repository. Document and apply a fixed mode normalization, or recalculate the hashes from modes representable by Git, rather than saying modes are preserved.
Useful? React with 👍 / 👎.
| Raw reports will be copied byte-for-byte into `reports/rNNN.md` only after all | ||
| evaluated runs finish. Agent identity, report digest, deviations, blind score, | ||
| and adjudication will then be appended without changing the frozen material |
There was a problem hiding this comment.
After collection, this section is never populated with the promised agent identities or per-report digests; the manifest proceeds directly to the aggregate collection result and only supplies a report-tree digest and report-label mappings. Without agent identities, readers cannot verify the protocol's central claim that all 54 cells used distinct fresh agents and that no agent saw multiple cells, while the missing expected digest for each report prevents independent report-level provenance checks. Append the completed ledger or link to a committed ledger containing these fields.
Useful? React with 👍 / 👎.
2 participants
Replay the predecessor pilot targets against the frozen V1 package and run a
54-report treatment/core-ablation study over nine abstraction-design modes.
Preserve the fixtures, frozen packages, manifests, raw reports, blind scores,
adjudications, and limitations needed to interpret both exploratory studies.
The legacy replay recovered every known synthetic and historical defect and
accepted every fixed control without a hard error. It improved historical
version applicability, literal HasField contract discovery, and exposure of
reconstructed proofs, while still missing an admissible indirect Copy and
UnsafeCell derivation on the current-source challenge.
The abstraction-design workflow produced the clearest measured improvement.
No treatment report certified an unimplemented proposal, while 16 of 27
core-ablation reports did; treatment matched or exceeded every adjudicated mode
and elicited parsimonious designs such as checked construction, real sealing,
safe slice splitting, and receiver-bound lifetimes.
The preregistered gates nevertheless failed. Four treatment reports used
executions containing UB as defined behavioral counterexamples. Two of three
treatment reports incorrectly proved a Rust-1.70 empty-slice pointer loop after
promoting constructor or slice admission facts into a universal invariant; the
third treatment report correctly falsified the evaluator's original oracle.
These failures motivate V2's whole-execution verdict, exact-domain,
boundary-case, and producer-quantifier rules. The results remain exploratory:
isolation was procedural, model identity and sampling seed were unavailable,
and the legacy replay was neither byte-identical to the pilot nor a statistical
release evaluation.
Latest Update: v2 — Compare vs v1
📚 Full Patch History
Links show the diff between the row version and the column version.
⬇️ Download this PR
Branch
git fetch origin refs/heads/G3y45zv35fuuyeejc26bftqdd33lqz2oh && git checkout -b pr-G3y45zv35fuuyeejc26bftqdd33lqz2oh FETCH_HEADCheckout
git fetch origin refs/heads/G3y45zv35fuuyeejc26bftqdd33lqz2oh && git checkout FETCH_HEADCherry Pick
git fetch origin refs/heads/G3y45zv35fuuyeejc26bftqdd33lqz2oh && git cherry-pick FETCH_HEADPull
Stacked PRs enabled by GHerrit.