-
Notifications
You must be signed in to change notification settings - Fork 170
Blind-score the V3 targeted confirmation #3529
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
joshlf
wants to merge
1
commit into
Gxgsta5jm3iasu652ow7bfbefe3c2r6nb
Choose a base branch
from
G3rihw6xuj2lcqvojuxabx73mkjzsgdd5
base: Gxgsta5jm3iasu652ow7bfbefe3c2r6nb
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+19,042
−0
Open
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
102 changes: 102 additions & 0 deletions
102
evals/unsafe-rust/runs/2026-07-31-v3-targeted/events.jsonl
Large diffs are not rendered by default.
Oops, something went wrong.
1,107 changes: 1,107 additions & 0 deletions
1,107
evals/unsafe-rust/runs/2026-07-31-v3-targeted/results/aggregate.json
Large diffs are not rendered by default.
Oops, something went wrong.
108 changes: 108 additions & 0 deletions
108
evals/unsafe-rust/runs/2026-07-31-v3-targeted/results/summary.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,108 @@ | ||
| # V3 Targeted Confirmation Results | ||
|
|
||
| **Primary V3 gate: FAIL.** | ||
|
|
||
| Each atom cell is a pass count out of five. V3 is the confirmatory candidate; V2 is diagnostic. | ||
|
|
||
| | Mode | Condition | Atom pass counts | Defective reports | | ||
| |---|---|---|---:| | ||
| | S | V3 | S1 5/5; S2 5/5; S3 5/5; S4 5/5; S5 5/5; S6 5/5; S7 5/5 | 0 | | ||
| | S | V2 | S1 5/5; S2 5/5; S3 5/5; S4 5/5; S5 5/5; S6 5/5; S7 5/5 | 0 | | ||
| | C | V3 | C1 1/5; C2 5/5; C3 3/5; C4 3/5; C5 3/5; C6 5/5 | 2 | | ||
| | C | V2 | C1 0/5; C2 5/5; C3 5/5; C4 5/5; C5 5/5; C6 5/5 | 0 | | ||
| | X | V3 | X1 5/5; X2 5/5; X3 5/5; X4 0/5; X5 5/5; X6 0/5; X7 0/5; X8 5/5; X9 5/5; X10 5/5; X11 2/5; X12 5/5; X13 5/5 | 0 | | ||
| | X | V2 | X1 5/5; X2 5/5; X3 5/5; X4 0/5; X5 5/5; X6 0/5; X7 1/5; X8 5/5; X9 5/5; X10 5/5; X11 4/5; X12 5/5; X13 5/5 | 0 | | ||
| | Q | V3 | Q1 5/5; Q2 5/5; Q3 5/5; Q4 5/5; Q5 5/5 | 0 | | ||
| | Q | V2 | Q1 5/5; Q2 5/5; Q3 5/5; Q4 5/5; Q5 5/5 | 0 | | ||
| | W | V3 | W1 5/5; W2 5/5; W3 5/5 | 0 | | ||
| | W | V2 | W1 5/5; W2 5/5; W3 5/5 | 0 | | ||
| | M | V3 | M1 5/5; M2 5/5; M3 5/5; M4 5/5; M5 5/5; M6 5/5; M7 5/5; M8 5/5; M9 5/5; M10 5/5; M11 5/5 | 0 | | ||
| | M | V2 | M1 5/5; M2 5/5; M3 5/5; M4 5/5; M5 5/5; M6 5/5; M7 5/5; M8 5/5; M9 5/5; M10 5/5; M11 5/5 | 0 | | ||
| | R | V3 | R1 5/5; R2 5/5; R3 5/5; R4 5/5; R5 5/5; R6 5/5; R7 5/5 | 0 | | ||
| | R | V2 | R1 5/5; R2 5/5; R3 5/5; R4 5/5; R5 5/5; R6 5/5; R7 5/5 | 1 | | ||
| | K | V3 | K1 5/5; K2 5/5; K3 5/5; K4 5/5; K5 5/5; K6 5/5; K7 5/5; K8 5/5 | 1 | | ||
| | K | V2 | K1 5/5; K2 5/5; K3 5/5; K4 5/5; K5 5/5; K6 5/5; K7 5/5; K8 5/5 | 0 | | ||
|
|
||
| ## Diagnostic comparison | ||
|
|
||
| Any V3 atom below matched V2: YES. | ||
|
|
||
| `TARGETED_LIFT_EVIDENCE` means V3 passed 5/5 while matched V2 was lower. `CEILING_REPLICATION` means both passed 5/5. These coherent packages differ in more than one isolated instruction, so no classification is causal proof. | ||
|
|
||
| | Mode | Atom | V3 | V2 | Classification | | ||
| |---|---|---:|---:|---| | ||
| | S | S1 | 5/5 | 5/5 | CEILING_REPLICATION | | ||
| | S | S2 | 5/5 | 5/5 | CEILING_REPLICATION | | ||
| | S | S3 | 5/5 | 5/5 | CEILING_REPLICATION | | ||
| | S | S4 | 5/5 | 5/5 | CEILING_REPLICATION | | ||
| | S | S5 | 5/5 | 5/5 | CEILING_REPLICATION | | ||
| | S | S6 | 5/5 | 5/5 | CEILING_REPLICATION | | ||
| | S | S7 | 5/5 | 5/5 | CEILING_REPLICATION | | ||
| | C | C1 | 1/5 | 0/5 | V3_HIGHER_BUT_CONFIRMATION_FAILED | | ||
| | C | C2 | 5/5 | 5/5 | CEILING_REPLICATION | | ||
| | C | C3 | 3/5 | 5/5 | V3_BELOW_V2 | | ||
| | C | C4 | 3/5 | 5/5 | V3_BELOW_V2 | | ||
| | C | C5 | 3/5 | 5/5 | V3_BELOW_V2 | | ||
| | C | C6 | 5/5 | 5/5 | CEILING_REPLICATION | | ||
| | X | X1 | 5/5 | 5/5 | CEILING_REPLICATION | | ||
| | X | X2 | 5/5 | 5/5 | CEILING_REPLICATION | | ||
| | X | X3 | 5/5 | 5/5 | CEILING_REPLICATION | | ||
| | X | X4 | 0/5 | 0/5 | MATCHED_BELOW_CEILING | | ||
| | X | X5 | 5/5 | 5/5 | CEILING_REPLICATION | | ||
| | X | X6 | 0/5 | 0/5 | MATCHED_BELOW_CEILING | | ||
| | X | X7 | 0/5 | 1/5 | V3_BELOW_V2 | | ||
| | X | X8 | 5/5 | 5/5 | CEILING_REPLICATION | | ||
| | X | X9 | 5/5 | 5/5 | CEILING_REPLICATION | | ||
| | X | X10 | 5/5 | 5/5 | CEILING_REPLICATION | | ||
| | X | X11 | 2/5 | 4/5 | V3_BELOW_V2 | | ||
| | X | X12 | 5/5 | 5/5 | CEILING_REPLICATION | | ||
| | X | X13 | 5/5 | 5/5 | CEILING_REPLICATION | | ||
| | Q | Q1 | 5/5 | 5/5 | CEILING_REPLICATION | | ||
| | Q | Q2 | 5/5 | 5/5 | CEILING_REPLICATION | | ||
| | Q | Q3 | 5/5 | 5/5 | CEILING_REPLICATION | | ||
| | Q | Q4 | 5/5 | 5/5 | CEILING_REPLICATION | | ||
| | Q | Q5 | 5/5 | 5/5 | CEILING_REPLICATION | | ||
| | W | W1 | 5/5 | 5/5 | CEILING_REPLICATION | | ||
| | W | W2 | 5/5 | 5/5 | CEILING_REPLICATION | | ||
| | W | W3 | 5/5 | 5/5 | CEILING_REPLICATION | | ||
| | M | M1 | 5/5 | 5/5 | CEILING_REPLICATION | | ||
| | M | M2 | 5/5 | 5/5 | CEILING_REPLICATION | | ||
| | M | M3 | 5/5 | 5/5 | CEILING_REPLICATION | | ||
| | M | M4 | 5/5 | 5/5 | CEILING_REPLICATION | | ||
| | M | M5 | 5/5 | 5/5 | CEILING_REPLICATION | | ||
| | M | M6 | 5/5 | 5/5 | CEILING_REPLICATION | | ||
| | M | M7 | 5/5 | 5/5 | CEILING_REPLICATION | | ||
| | M | M8 | 5/5 | 5/5 | CEILING_REPLICATION | | ||
| | M | M9 | 5/5 | 5/5 | CEILING_REPLICATION | | ||
| | M | M10 | 5/5 | 5/5 | CEILING_REPLICATION | | ||
| | M | M11 | 5/5 | 5/5 | CEILING_REPLICATION | | ||
| | R | R1 | 5/5 | 5/5 | CEILING_REPLICATION | | ||
| | R | R2 | 5/5 | 5/5 | CEILING_REPLICATION | | ||
| | R | R3 | 5/5 | 5/5 | CEILING_REPLICATION | | ||
| | R | R4 | 5/5 | 5/5 | CEILING_REPLICATION | | ||
| | R | R5 | 5/5 | 5/5 | CEILING_REPLICATION | | ||
| | R | R6 | 5/5 | 5/5 | CEILING_REPLICATION | | ||
| | R | R7 | 5/5 | 5/5 | CEILING_REPLICATION | | ||
| | K | K1 | 5/5 | 5/5 | CEILING_REPLICATION | | ||
| | K | K2 | 5/5 | 5/5 | CEILING_REPLICATION | | ||
| | K | K3 | 5/5 | 5/5 | CEILING_REPLICATION | | ||
| | K | K4 | 5/5 | 5/5 | CEILING_REPLICATION | | ||
| | K | K5 | 5/5 | 5/5 | CEILING_REPLICATION | | ||
| | K | K6 | 5/5 | 5/5 | CEILING_REPLICATION | | ||
| | K | K7 | 5/5 | 5/5 | CEILING_REPLICATION | | ||
| | K | K8 | 5/5 | 5/5 | CEILING_REPLICATION | | ||
|
|
||
| ## Primary gates | ||
|
|
||
| | Gate | Result | | ||
| |---|---| | ||
| | all v3 atoms 5 of 5 | FAIL | | ||
| | zero v3 hard errors | FAIL | | ||
| | zero v3 proposal laundering | PASS | | ||
| | zero v3 tcb authority defects | FAIL | | ||
| | zero v3 semantic noncompletion | PASS | | ||
| | zero v3 scope budget defects | PASS | | ||
|
|
||
| ## Integrity limitations | ||
|
|
||
| Filesystem and URL isolation were procedural on a shared host. Exact hosted model-build and sampling-seed metadata were unavailable. Results are source-review capability observations under those constraints. |
186 changes: 186 additions & 0 deletions
186
...unsafe-rust/runs/2026-07-31-v3-targeted/scoring/adjudication-packets/C/DISAGREEMENTS.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,186 @@ | ||
| { | ||
| "cells": [ | ||
| { | ||
| "field": "atom:C3", | ||
| "label": "C", | ||
| "s1": { | ||
| "decision": "PASS", | ||
| "evidence": "BUILD-MAP-C and version-matched cfg and compile_error! contracts are combined to prove every turbo-W case fails before an artifact can ship." | ||
| }, | ||
| "s2": { | ||
| "decision": "FAIL", | ||
| "evidence": "It verifies cfg-attribute inclusion and compile_error!, but omits version-matched Rust all semantics needed to show the source predicate selects the macro." | ||
| } | ||
| }, | ||
| { | ||
| "field": "atom:C4", | ||
| "label": "C", | ||
| "s1": { | ||
| "decision": "PASS", | ||
| "evidence": "OBL-2 proves unwrap_or, OBL-3/4 prove the guarded unchecked call, and Covered=Required supplies closure across every profile and debug state with a relative PROVED verdict." | ||
| }, | ||
| "s2": { | ||
| "decision": "FAIL", | ||
| "evidence": "OBL-2 through OBL-4 are locally correct, but Covered=Required relies on OBL-1 body selection without verified version-matched not/all predicate semantics." | ||
| } | ||
| }, | ||
| { | ||
| "field": "atom:C5", | ||
| "label": "C", | ||
| "s1": { | ||
| "decision": "PASS", | ||
| "evidence": "OBL-2 through OBL-4 prove zero for None and the payload for Some in both feature cases, then close over Required." | ||
| }, | ||
| "s2": { | ||
| "decision": "FAIL", | ||
| "evidence": "The return-value cases are locally right, but their aggregate configuration closure depends on the same incomplete cfg-selection authority." | ||
| } | ||
| }, | ||
| { | ||
| "field": "hard_error:G11", | ||
| "label": "C", | ||
| "s1": { | ||
| "decision": "ABSENT", | ||
| "evidence": "No applicable hard error recorded." | ||
| }, | ||
| "s2": { | ||
| "decision": "PRESENT", | ||
| "evidence": "The positive selection and rejection claims necessarily rely on unchecked Rust all/not semantics outside BUILD-MAP-C's admitted proposition." | ||
| } | ||
| }, | ||
| { | ||
| "field": "hard_error:CH2", | ||
| "label": "C", | ||
| "s1": { | ||
| "decision": "ABSENT", | ||
| "evidence": "No applicable hard error recorded." | ||
| }, | ||
| "s2": { | ||
| "decision": "PRESENT", | ||
| "evidence": "It claims Covered=Required although its stated cfg-selection derivation omits material version-matched all/not semantics." | ||
| } | ||
| }, | ||
| { | ||
| "field": "hard_error:CH6", | ||
| "label": "C", | ||
| "s1": { | ||
| "decision": "ABSENT", | ||
| "evidence": "No applicable hard error recorded." | ||
| }, | ||
| "s2": { | ||
| "decision": "PRESENT", | ||
| "evidence": "It claims full PROVED verdicts over Required while OBL-1 remains incomplete throughout that domain." | ||
| } | ||
| }, | ||
| { | ||
| "field": "tcb_authority_defect", | ||
| "label": "C", | ||
| "s1": { | ||
| "decision": "ABSENT", | ||
| "evidence": "The report consumes BUILD-MAP-C only for OBL-1/reachability and uses correctly matched Rust documentation for semantics." | ||
| }, | ||
| "s2": { | ||
| "decision": "PRESENT", | ||
| "evidence": "The report uses all/not cfg-predicate behavior for rejection and body selection without verifying those propositions in version-matched Rust authority." | ||
| } | ||
| }, | ||
| { | ||
| "field": "novel:s2:N1", | ||
| "label": "E", | ||
| "s1": { | ||
| "decision": "ABSENT", | ||
| "evidence": "Not independently proposed; adjudicate the candidate on its merits." | ||
| }, | ||
| "s2": { | ||
| "decision": "PRESENT", | ||
| "evidence": "The report locates unwrap_unchecked at line 22, but the supplied source places that unsafe call at line 20." | ||
| } | ||
| }, | ||
| { | ||
| "field": "atom:C3", | ||
| "label": "G", | ||
| "s1": { | ||
| "decision": "PASS", | ||
| "evidence": "It combines BUILD-MAP-C with versioned cfg and compile_error! semantics to prove every turbo-W build fails." | ||
| }, | ||
| "s2": { | ||
| "decision": "FAIL", | ||
| "evidence": "The rejection argument verifies cfg-attribute inclusion and compile_error!, but never verifies the version-matched Rust semantics of all needed to select the macro." | ||
| } | ||
| }, | ||
| { | ||
| "field": "atom:C4", | ||
| "label": "G", | ||
| "s1": { | ||
| "decision": "PASS", | ||
| "evidence": "O-NON/O-GUARD/O-UNWRAP prove both branches, define parametric Covered, and explicitly derive Required subset-of Covered before the relative PROVED result." | ||
| }, | ||
| "s2": { | ||
| "decision": "FAIL", | ||
| "evidence": "The body proofs are correct locally, but the claimed configuration partition and Covered closure rely on unverified version-matched not/all cfg-predicate semantics." | ||
| } | ||
| }, | ||
| { | ||
| "field": "atom:C5", | ||
| "label": "G", | ||
| "s1": { | ||
| "decision": "PASS", | ||
| "evidence": "The same obligations prove zero for None and the payload for Some, and state their configuration union is Covered." | ||
| }, | ||
| "s2": { | ||
| "decision": "FAIL", | ||
| "evidence": "The local return-value cases are correct, but the aggregate behavioral closure uses the same incomplete cfg-selection premise." | ||
| } | ||
| }, | ||
| { | ||
| "field": "hard_error:G11", | ||
| "label": "G", | ||
| "s1": { | ||
| "decision": "ABSENT", | ||
| "evidence": "No applicable hard error recorded." | ||
| }, | ||
| "s2": { | ||
| "decision": "PRESENT", | ||
| "evidence": "The positive selection and rejection claims necessarily rely on unchecked Rust all/not semantics that BUILD-MAP-C does not authorize." | ||
| } | ||
| }, | ||
| { | ||
| "field": "hard_error:CH2", | ||
| "label": "G", | ||
| "s1": { | ||
| "decision": "ABSENT", | ||
| "evidence": "No applicable hard error recorded." | ||
| }, | ||
| "s2": { | ||
| "decision": "PRESENT", | ||
| "evidence": "It claims Covered closure although the stated cfg-selection derivation omits the material version-matched all/not predicate semantics." | ||
| } | ||
| }, | ||
| { | ||
| "field": "hard_error:CH6", | ||
| "label": "G", | ||
| "s1": { | ||
| "decision": "ABSENT", | ||
| "evidence": "No applicable hard error recorded." | ||
| }, | ||
| "s2": { | ||
| "decision": "PRESENT", | ||
| "evidence": "It claims full PROVED verdicts over the audit domain while the cfg-selection obligation used across that domain is incomplete." | ||
| } | ||
| }, | ||
| { | ||
| "field": "tcb_authority_defect", | ||
| "label": "G", | ||
| "s1": { | ||
| "decision": "ABSENT", | ||
| "evidence": "It keeps BUILD-MAP-C conspicuous and within configuration reachability/selection/rejection, with semantic claims from matched Rust docs." | ||
| }, | ||
| "s2": { | ||
| "decision": "PRESENT", | ||
| "evidence": "The report cites cfg-attribute inclusion but omits authoritative version-matched all/not predicate propositions while relying on them for selection and rejection." | ||
| } | ||
| } | ||
| ], | ||
| "mode": "C", | ||
| "schema_version": 1 | ||
| } |
21 changes: 21 additions & 0 deletions
21
evals/unsafe-rust/runs/2026-07-31-v3-targeted/scoring/adjudication-packets/C/PACKET.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,21 @@ | ||
| { | ||
| "allowlist_sha256": "ad025876ef66f6e6578eea565088eaa99c24c0a603391d1e9f924858b3f63c48", | ||
| "common_rules_sha256": "b7ad63a8a816034852d1542e904eccce33c3e9ff60e6b4b2bb7f37f503ce8b82", | ||
| "disagreements_sha256": "780f7355b0edd93c48f6c97b335a60f184ec0af65aa4e7ab019b38dbceb611b1", | ||
| "disputed_labels": [ | ||
| "C", | ||
| "E", | ||
| "G" | ||
| ], | ||
| "mode": "C", | ||
| "report_sha256": { | ||
| "C": "1409aa1649cd9e0950368deb23ade05351c03b9ab89a32e637514b9d43293e22", | ||
| "E": "5c927aebfbdd7b0c2213525642f94b9b2d634ddbe371c2de93761a8799809093", | ||
| "G": "78953c2a86ff27a79669c044278b47b0cb996b7bf6a47ca72e92fd5ec643ee43" | ||
| }, | ||
| "rubric_sha256": "3953c17ad001ee677061118952a5ac4fae44e21cc1ed1d64a0a0b2ae41c0049c", | ||
| "schema_sha256": "58d36e6a54002a29fe7f45c4f65d966a307212e6d082be75d34772fc5b128209", | ||
| "schema_version": 1, | ||
| "source_score_packet_sha256": "d62b94c908cf13f894b80f4308c7c2b836bd750cb59fd82916880b92ad92aec1", | ||
| "target_byte_tree_sha256": "dc91c627439d64fdf16069e999d98b815021cda01721f3fb5dd63fd4ccc33c86" | ||
| } | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In a fresh checkout, this digest cannot be verified because the packet's
target/directory was not committed: the repository.gitignoreignores directories namedtarget, and none of the C, K, R, or X adjudication packets contains target files.build_adjudication_packetincludes that tree in the preserved packet digest, whileverify_adjudication_packetrequires it, so every committed adjudication packet differs from its event hash and the evaluation fails integrity verification. Force-add these trees or store them under a nonignored path.Useful? React with 👍 / 👎.