spike: pre-register the external-validity study for the PDF extraction seam (design + confirmatory freeze) - #560
Draft
willhea wants to merge 21 commits into
Draft
Conversation
…xists The two items blocking an ADR since phase 1 are a trustworthy heading oracle and a fresh structure-rich holdout. Neither is a word-spacing question, and phase 3 closed word spacing, so this is a new study rather than a phase 4: phases 1-3 interrogated development evidence, this one tests external validity with a new oracle and a new population. Three design measurements, because a protocol may not be built on inherited reasons: x00, on development documents only: once both seams are fed by PDFium and run in the same soft-hyphen mode, they differ on 2 of 3,381 aligned printed lines and 0 of 85 heading occurrences. Both differences are letter-spaced display type on front matter, and both architectures are wrong on one of them. So RQ1 is an equivalence-and-direction study, not a superiority study, and its power lives in a bound computed on the whole holdout without an oracle. The first pass of x00 read 98 differences on one document. Every one was a line truncated at a U+FFFD carrier, because reconstruct_hybrid renders the soft hyphen unconditionally and reconstruct_extended only under repaired=True. That is a measurement of the mode, not of the seam; mode parity is now a protocol clause and S1 (advances x1.25 -> 458 differences) is what establishes the comparator is live. x02: every phase disqualifies the XML heading oracle because the parser drops <quoted-block>. DeltaTrack#11 is CLOSED, and of 27,275 appropriations elements across all 58 corpus XMLs, zero sit inside a quoted block; 6,617 ordinary headers do. The mechanism is real for general legislation and has no instances for appropriations, so the pre-registration states the reasons it can defend instead. x01: freshness is enumerated rather than asserted, from five exposure classes including PDFs ever added on any ref and bill ids merely named in prose, giving 93 bills and 33 report packages to subtract. x04 audits the gate and ships a self-test that seeds a known-contaminated bill and requires detection, because a freshness check that has never fired cannot tell clean from broken. It currently reports the gate CLOSED, and F2/F3 report VACUOUS rather than PASS while there are no members. Refs #553
The probe resolved its repository root one directory too high, so it globbed
an empty tree, wrote totals of {}, and still printed the "no appropriations
heading sits inside a quoted block" verdict -- because that verdict tests
count == 0, which an empty counter satisfies. The absence claim was true (a
scratch probe measured it correctly from the repo root) but the committed
artifact did not produce it, and PRE-REGISTRATION.md section 5.1 cites the
committed artifact.
This is the failure class the study exists to prevent, in the study's own
harness, and it is the second one this branch has caught: x00 first measured
a soft-hyphen MODE rather than the seam, and x02 measured nothing at all.
Fixed the root in x02, x03 and x04 (x00 and x01 were already correct, which
is why only these three were wrong). Added a completeness FLOOR to x02: an
assertion of absence now fails hard below 40 documents or 10,000
appropriations elements, rather than passing vacuously. Verified against the
known-bad case by re-running the original wrong root -- it exits 2.
Re-run reproduces the cited numbers exactly: 58 documents, 27,275
appropriations elements, 0 inside a quoted block, 6,617 ordinary headers
inside one, 30 of 58 documents carrying one.
Also bounded stratum 8's search (a page gate can only be evaluated by
downloading, so an unbounded walk fetches arbitrarily many large PDFs) and
made it take a bill's LAST version rather than its first, since the
introduced print is the smallest and would fail a page gate its own bill
passes. A stratum that stops on its examine budget is now recorded distinctly
from one that runs out of candidates.
Refs #553
…cored Selection only. No extractor was imported; the sole PDF operation is a page count, which reads no text. Membership, per-file SHA-256, stratum, population and the full strata accounting are in results/holdout_membership.json, and the protocol was committed first (f4dfaf1), so git order is the freeze proof. 7 of 8 strata filled -> adequacy "generalization", pending the >=800 heading-occurrence check at extraction time. P-head is 14 numbered-layout appropriations acts, 106-1,547 pp, both chambers, Congresses 113-119. P-robust is 2 enrolled bills and 3 appropriations committee reports, which carry M0/M9 only. Stratum 5 filled 1 of 2 on a pool of 8, all 8 examined, so that is scarcity rather than budget and it is named rather than rounded up. Four defects found while building the frame, each of which would have produced a plausible-looking but wrong population: - Committee referral is not "carries an account tree". The first run selected "Pay Our Troops Act" (3 pp) and "Chips and Science Act" -- referred to Appropriations, no account headings -- which is precisely what voided the prior holdout's heading metric. Replaced by GPO's title convention plus a 25-page floor, both BILLSTATUS/container facts that read no PDF text. - The CRPT sitemap pattern required a trailing slash the URLs do not have and returned a frame of 0 packages. - MODS lives under /metadata/pkg, not /content/pkg, so the report classifier 404'd on all 60 candidates and reported "0 appropriations reports". Both failures read exactly like a rare class, so the run now records mods_liveness (now 93/93) and stopped_on_budget alongside every count. - Package ids are case-sensitive on govinfo, and the sitemap ids are upper-cased to match the contamination inventory, so the first fix still 404'd 250 times. mods_liveness is what caught it. Also excluded enrolled prints from every P-head stratum: "prefer last" had chosen 116-hjres-31's enrolled print, and an unnumbered enrolled layout is declined by production and yields no account anchors, so it would have been a guaranteed zero denominator inside the population that exists to avoid them. Protocol amendments, all made with no score in existence: split P-head from P-robust after finding that enrolled bills and report PDFs cannot carry a heading metric at all (committee_report.py reads GPO's HTML <pre> dump, not the PDF); added M9 structural viability, because extract_anchors emits nothing below 0.85 line-number coverage and the margin-number space is re-derived by the rule under extended glyph, so that architecture can silently lose a whole document's heading tree; and gave M9 precedence over every other rule. Refs #553
…draw it External review made five findings. Four are valid, one partially. Each was tested against the code rather than accepted, and three are now demonstrated by measurement rather than argued. 1. THE GATE CONTRADICTED THE PROTOCOL. x04 checked six freeze invariants and printed EXECUTION GATE OPEN while the protocol's own gate listed two prerequisites it never tested. Split into FREEZE INTEGRITY and EXECUTION READINESS, both machine-checked, both required. G1-G4 now assert the adapter, the X2-a/X2-b evidence, the adjudicator prompt and the exposure list. --self-test drives five known-bad cases, including one that feeds G2 evidence recorded on the HOLDOUT and requires rejection. 2. THE PROTOCOL WAS AMENDED AFTER SELECTION. Confirmed: 4.4.1, M9, Rule 0 and the revised selection rules were all written because selection runs surfaced problems, and 1350710 amended this document in the SAME commit that froze the population. F4 tested first_commit(PREREG), which proves only that SOME version predated it. F4 now tests the LAST-modifying commit and requires a strict ancestor; run against the old state it fails, which is how it should have read all along. So the 19-document population is withdrawn as confirmatory and kept as design history. The five design runs are preserved, the 37 documents they selected are derived into design_exposure.json by x05 and excluded, and the confirmatory seed moves to 20260808. Nothing was ever scored. 3. THE MARGIN LIVED ON THE WRONG UNIT. Measured: zero events on 600 headings gives a 95% bound of 0.00498; zero events on the 14 documents the protocol itself calls independent gives 0.1926 -- a 39x overstatement, and 598 documents would be needed for 0.005 on the document unit. The zero-event cluster bootstrap is worse: simulated on all-zero clusters it returns [0.0, 0.0] with a single distinct statistic, so it carries no information in exactly the case the design expects. The per-heading margin is withdrawn, the primary outcome is descriptive, and the one inferential statement is an exact document-unit bound whose weakness is stated rather than dressed up. 4. A SHARED FAILURE COULD VANISH FROM RQ2. Rule 0 excluded documents where BOTH architectures lose the heading tree. Neutral for RQ1, but for RQ2 that conditions absolute correctness on successful extraction and inflates it. Shared failures are now neutral for RQ1 and RETAINED as failures in RQ2 and M9; only 4.6's source-level exclusions, all properties of the file, may remove a frozen document. 5. M3 WAS A TOKEN MULTISET. It discards order and charges character errors to the seam. Redefined on aligned character boundaries with WELD, SPLIT, OK, TEXT_ERROR and UNALIGNABLE as explicit outcomes; UNALIGNABLE is reported split by discordance status, and the decision rule consumes WELD/SPLIT only. Also narrowed two claims. M2 normalises whitespace runs, so it licenses "exact up to whitespace-run normalisation", not "character for character". And the oracle is AI image-adjudication: a 25-item human audit estimates agreement, it does not make the key ground truth. Because the D-frame is the only evidence that can flip the ADR and is expected to be tiny, every D-frame item now requires human adjudication. Refs #553
F4 requires this document's last-modifying commit to strictly precede the membership commit, so the status line cannot restate what selection produced without either closing the gate or being written after seeing the result. It now points at the two artifacts that carry the state instead. Refs #553
…hing scored Selected under the protocol frozen at c399e9d, which is a strict ancestor of this commit -- the condition F4 now enforces and the one the withdrawn population failed. 17 documents. The 37 design-exposed documents are excluded, so no member was surfaced by a run whose selection rules its own appearance helped write; the membership and the exposure list are disjoint, asserted by F3 rather than claimed. mods_liveness 78/78, so the report classifier could see what it was filtering. ADEQUACY DOWNGRADES TO "sampled-classes-only", 6 of 8 strata filled, and that is reported rather than repaired: - stratum 5 (full-year CR / joint resolution) fills 0 of 2. After exclusions its entire candidate pool is 5 bills and all 5 were examined -- scarcity, not budget. - stratum 8 (omnibus >= 400 pp) fills 1 of 2, stopped on its pre-committed examine budget of 100 with 274 candidates. Raising that budget now would change a sampling rule after seeing what the sample produced, which section 4.7 forbids and which is the exact defect this whole pass exists to correct. The unfilled strata are named in the headline, per the pre-committed adequacy rule. P-head is 12 documents / 2,864 pages, so the tightest achievable zero-event document-unit bound is about 22 percent. Section 8.3 already states that this holdout can fail to falsify equivalence and bound it loosely, and cannot establish a small per-document rate; the smaller P-head makes that limit larger, not different in kind. Refs #553
F4 read first_commit(membership), but the design-era population was withdrawn and the confirmatory one written to the same path, so git log --reverse still returned the DESIGN commit 1350710. F4 therefore judged the current protocol against a population that no longer exists and reported FAIL on a freeze that is sound. Compares the last-modifying commit of both files now. Extracted F4's predicate into f4_ok so the self-test can drive it with real commits instead of asserting around it. A first attempt at that test was `not (head != head)`, which is true regardless of the predicate -- a vacuous check inside the very self-test that exists to prove checks are not vacuous. It now drives four cases: strict ancestor accepted, same-commit rejected, reversed order rejected, uncommitted rejected. The same-commit case is the one that matters, because a commit is its own ancestor and without the inequality a protocol amended alongside its own holdout passes. Self-test 8/8. Gate now reads FREEZE INTEGRITY COMPLETE, EXECUTION READINESS CLOSED, exit 1. Refs #553
willhea
changed the title
… own directory External review found an 18th file under holdout/ against 17 members. All five sub-questions confirmed: CRPT-118HRPT146/CRPT-118HRPT146.pdf is 44 KB of "<!DOCTYPE html>", not a PDF; it is absent from holdout_membership.json; it was left by the rejected-download path, which continued without deleting dest; it was the only such file; and F1/F2 iterate the MANIFEST, so nothing looked at the directory and the gate passed with it present. govinfo answers a missing package with HTTP 200 and an HTML landing page, so download() succeeded, page_count raised, the candidate was correctly rejected, and the file stayed. The population was never wrong -- the directory was. Repairs. x03 routes both download branches through accept_download, which deletes on every rejection path and requires a %PDF- header. x04 gains three invariants: F7 asserts SET EQUALITY between holdout/ and the manifest, F8 that every manifested file really is a PDF, and F9 that every file in this study changed after the population freeze is declared as an amendment. Self-test is 12/12, including an intruder file, a manifested non-PDF, and two malformed amendments. The 17-member population is unchanged. The orphan was never a member, so deleting it corrects the directory, not the population. The header check could in principle reject a PDF whose header is not at offset 0, so that was measured rather than assumed: all 17 members begin with %PDF-, asserted continuously by F8, so the new rule would have selected exactly the same documents. M3 is now executable rather than left to the eventual scorer. m3_boundaries.py freezes the alignment (Needleman-Wunsch, unit costs, deterministic DIAGONAL/UP/LEFT tie-break, NFKC, a run of spaces is one boundary) and REMOVES the edit budget, which was an arbitrary threshold on the quantity being measured; UNALIGNABLE now means the sequences share no common subsequence. m3_selftest.py covers the review's cases, including AB AB -> ABAB and the one that decides the estimand: H with two welds, X repairing one, is BOTH_DIRTY and NOT a correction. PRE-REGISTRATION.md is deliberately NOT edited -- it stays byte-frozen at c399e9d so F4 keeps its plain meaning -- and every change is recorded in PRE-EXECUTION-AMENDMENTS.md, classified CLERICAL, SUBSTANTIVE or TOOLING, each asserting no confirmatory output existed. A3 and A5 are acknowledged as SUBSTANTIVE changes to the scoring rule, not clerical errata. Refs #553
… output Swept for the same class of defect the orphan file exposed -- a closed world that is not closed -- and found a worse one. x01 scans the research tree for .json to find bill ids "named in research". Its own output is a .json in that tree, and it RECORDS the 2,963 xml-only bills it deliberately does not exclude. So a second run re-ingested all of them: the exclusion set went from 93 bills to 3,080, and every one of the 17 confirmatory holdout members was flagged as contaminated. F3 reads the committed file, so the frozen gate was never wrong. But any re-derivation would have condemned the population the probe exists to protect, and it would have presented as a contamination finding rather than as a bug -- the comforting reading being the wrong one, again. Three repairs. Generated artifacts are no longer scanned. This study's own frozen population is subtracted into its own recorded class, because a holdout is exposed by construction the moment it is committed and a future study must still treat these 17 as contaminated. And an idempotence gate re-derives with the new file in place and requires a no-op. The first version of that gate compared against the committed artifact and failed on ANY change, which fired on legitimate growth: the withdrawn design PDFs are now in git history, so +17 bills and +5 reports are honest new exposure. A gate that forbids legitimate change is a permanent red light, not a gate. It now tests the property that matters. Two consecutive runs are byte-identical. The honest exclusion set is 110 bills and 38 report packages, up from 93/33, and the 17 members remain clean under F3. Recorded as amendment A7. Refs #553
A7 subtracted this study's own population from contamination.json's top-level excluded lists, but x04's exposure_ids builds its lookup from the per-class blocks, where the members still appear. So F3 flagged all 17 members as "in pdf_committed" the moment the honest exclusion set was re-derived -- the gate condemning the population it exists to protect, which is the failure A7 set out to prevent. exposure_ids now honours both recorded-but-not-excluded classes, xml_only_not_excluded and own_study_population_not_excluded, and subtracts the own-study ids from every other class. The scope stays narrow: a FUTURE study must still treat these 17 as exposed, which is why the exemption lives in a named class rather than in a deletion. F3's self-tests still fire: a contaminated member and a design-exposed member are both detected, and design_exposed is deliberately NOT subtracted by the exemption (the two sets are disjoint). Self-test 12/12, gate reads FREEZE INTEGRITY COMPLETE with EXECUTION READINESS CLOSED. Refs #553
… subtraction Four review findings, all valid, all tested against the implementation first. 1. THE OWN-STUDY EXEMPTION COULD ERASE REAL CONTAMINATION. A7 subtracted current membership from every exposure class, which cannot distinguish exposure this study CAUSED by committing its holdout from exposure that existed BEFORE selection on a document picked anyway -- and forgives the second. Freshness now reads the PRE-SELECTION snapshot from git at <population_commit>~1, which by construction cannot contain exposure this study later caused, so no exemption is needed and the subtraction is withdrawn: contamination.json again records the 17 members as exposed, which is true and which a future study needs. THE AUDIT, which is why no reselection follows: at c399e9d the inventory held 93 excluded bills and 33 report packages, carried no own-study class, and ALL 17 members were absent from every disqualifying class. Both controls now encode the distinction -- case B (contaminated before selection) must fail, case A (exposed only by its own frozen commit) must pass -- and a snapshot carrying an own-study exemption is refused as not pre-selection. 2. UNALIGNABLE DID NOT MEAN WHAT THE PROSE SAID, and the reviewer's case is exact: AB against BA has LCS 1 yet minimum cost is two substitutions, so the DIAGONAL tie-break selected an alignment with no exact match and the code returned UNALIGNABLE. Neither offered repair was taken, because the category itself was the defect: UNALIGNABLE made the heading UNSCORABLE, which removed precisely the worst failures from the comparison -- X emitting garbage where H read the label correctly vanished instead of counting as X_REGRESSES. The category is WITHDRAWN. Severe corruption is a severe TEXT_ERROR and stays in the denominator; NO_REFERENCE replaces it and fires only when the ORACLE has no text. Strictly safer for the incumbent. 3. F5 CANNOT PROVE NO COMPUTATION HAPPENED. Narrowed to what it verifies -- no canonical score artifact exists -- and separated from the process attestation, which is weaker and now labelled. The provable fact becomes ordering, enforced by a one-way execution-start marker that x04 refuses to emit while any gate is open; after it, SUBSTANTIVE amendments are rejected and scoring-rule changes become deviations. 4. A COUNT THRESHOLD CANNOT BE APPLIED TO A SUBSAMPLE. If the D-frame exceeds the human budget, Rule 1 cannot choose extended glyph; the outcome is INSUFFICIENT_COMPARATIVE_EVIDENCE. Three outcomes are kept distinct so "X failed to prove a win" is never written as "H empirically beat X". F9 hardened to reject duplicate ids, files_touched paths that neither exist nor were deleted, a file declared under both SUBSTANTIVE and TOOLING, and TOOLING that changes a scoring rule. It immediately caught THIS file three times: m3_boundaries/m3_selftest under A3/A4 and A6, then x01/contamination under A7 and A8, then x04 under A6 and A11. Ownership reassigned to the substantive amendment in each case. Self-tests 21/21 and M3 39/39. Gate reads FREEZE INTEGRITY COMPLETE, EXECUTION READINESS CLOSED, exit 1. Refs #553
…y sweep The recurring pattern in this study is that a gate checks a PROXY for the property we care about. A deliberate sweep for further instances found two, recorded as A12. F6 repeated F4's exact defect, in the check that proves BLINDING. It compared first_commit(oracle_key) against first_commit(oracle_adjudicated). The membership file has already shown that an artifact can be withdrawn and re-created at the same path, after which first_commit returns a version that no longer exists -- so F6 would have proved the ordering of a key nobody would score against. Both sides now use the last-modifying commit. G2 believed a self-reported label. The evidence file declares population: DEVELOPMENT and G2 accepted that string as proof the X2 assertions were not run on the holdout. A file can say DEVELOPMENT while having been produced on holdout members -- the same class as trusting a .pdf filename over PDF bytes. G2 now requires the evidence to LIST its documents, checks them against membership directly, and requires a non-empty list so it cannot pass vacuously. Self-test 22/22, with the new case "G2 rejects evidence LABELLED development that names a holdout member" beside the existing self-labelled-HOLDOUT one. Also corrected two lines this pass made stale: the amendments header claimed F5 establishes mechanically that no output existed, which A11 narrowed to a repository fact plus an attestation; and A3's outcome table still routed UNSCORABLE through the UNALIGNABLE that A9 withdrew. Gate: FREEZE INTEGRITY COMPLETE, EXECUTION READINESS CLOSED, exit 1. Refs #553
… itself
Sweep for the study's recurring pattern -- a gate checking a proxy for the
property we care about -- found the most serious instance yet, because it
defeats every other freeze invariant at once.
committed() was `git ls-files --error-unmatch`, which proves a path is
TRACKED. The property needed is that the artifact is IDENTICAL to what was
committed. Since every invariant reads the working tree, they collectively
validated the working tree against itself.
Demonstrated rather than argued: deleting 7 members from
holdout_membership.json and removing their 7 PDFs, with no commit, leaves an
internally consistent tree, and F1 ("10 members, committed=True"), F2, F3, F7
and F8 all PASS. FREEZE INTEGRITY read COMPLETE over a 10-document population
while the committed freeze is 17.
committed() now means tracked AND unmodified against HEAD, and F10 asserts
that the frozen artifacts -- membership, contamination, design exposure, the
pre-registration, the amendments file and the whole holdout/ directory --
carry no uncommitted change. Scoped to frozen artifacts: probe code is
expected to change and is policed by F9.
A second, latent defect fixed in the same sweep. amendment_commits ordered by
`git rev-list --count`, which returns a STRING, so max() compared
lexicographically and "9" beat "1003". Dormant while no execution-start marker
exists, but it would have silently misjudged the one-way boundary the moment
execution was authorized -- the check most likely to be trusted without
re-derivation. Compared as integers now.
Self-test 24/24 with the two new controls. Membership re-verified at 17 and
unchanged. Recorded as A13.
Refs #553
… facts Five defects, each an instance of the gate checking a proxy for the property. 1. A COMMITTED MARKER WAS NEVER REQUIRED. main()'s final block consulted the marker not at all: stubbing both gate functions to pass produced "EXECUTION PERMITTED" and exit 0 with no marker anywhere. Authorization was optional. There is now a state machine -- ABSENT, UNCOMMITTED, MUTATED, VALID -- and only a valid committed write-once marker permits execution. Writing the file is not authorizing. 2. THE BOUNDARY WAS MOVABLE. marker_commit() used last_commit, so editing the marker later would move the boundary and make a substantive change appear to predate it. first_commit alone is not enough either, as this study already saw a path deleted and recreated. Immutability is asserted directly: exactly one modifying commit, and the current blob equals the blob introduced by it. 3. THE POPULATION FREEZE WAS MOVABLE. F3/F4/F9 derived it from last_commit(MEMBERSHIP), so a later committed manifest edit would have become the new freeze, moved the pre-selection snapshot to that commit's parent, and shrunk the post-freeze window. Pinned to 4e2b520, and new F11 proves the population is byte-identical to the frozen one -- blob, id set, paths, recorded hashes. A later change now fails instead of re-anchoring. 4. F9 BOUND DECLARATIONS TO PATHS, NOT CHANGES. A path declared once excused every later change to it; x04_freeze_check.py has nine modifying commits and all nine passed. Bound to commit SHAs now -- which immediately found TWO post-freeze methodological commits that no amendment ever described, 70ec76c and 985def9, declared retroactively as A14 rather than excused. 5. THE LEDGER WAS NOT SEALED. F9 now rejects any commit touching PRE-EXECUTION-AMENDMENTS.md after a valid boundary. G2 bound to provenance: a hand-written file naming "fake-doc-123" made it green, proving only that a file asserts its own success. Fixtures must now exist in the repo, not be holdout documents, and hash to their recorded sha256; adapter, reconstructor and verifier blob SHAs must match the committed files. G5 added because G1-G4 green did not mean the study could be run: no runner, frame builder, oracle builder, scorer or decision evaluator had to exist. Authorizing then would have permitted inspecting confirmatory output and finishing the scorer afterwards. 10 of 11 result-bearing files are missing, so the honest state is EXECUTION HARNESS INCOMPLETE. The marker becomes a manifest of the exact frozen surface, and normal x04 re-verifies those blobs, reporting METHODOLOGY DRIFT as a deviation rather than silently moving to the new version. Self-test 33/33. No marker created, nothing authorized, nothing scored. Refs #553
F9 now binds declarations to commits, so it correctly refused the commit that introduced that very rule. Declared under A15/A16 by a ledger-only commit, which is exempt from F9 -- that exemption is what stops the rule requiring an infinite regress of declarations. Refs #553
…itten Derived the confirmatory pipeline from the pre-registration and amendments BEFORE building the harness, and stopped at the ambiguities rather than coding through them. Four places do not determine what the code should do; the blocking one is measured rather than argued. M6 HAS NO ORACLE FOR MOST OF ITS POPULATION. Section 6 scores every dollar amount's attribution "vs adjudicated", while 5.3 fixes the adjudicated unit at 6-10 printed lines and 5.4 shows the adjudicator only that region. In GPO appropriations an account heading governs a long run of prose. Measured over three development documents and 226 amounts: the governing heading is within 6 lines for 0.434 of amounts and within 10 for 0.571; median distance 5, 12 and 9 lines by document; maximum 288. So at the protocol's own region size about two amounts in five have no oracle, and they are systematically the ones deepest inside long appropriations blocks, where misattribution matters most. That inflates RQ2's absolute attribution claim rather than biasing H against X. x06's first verdict string applied a >50% threshold and printed "may be implementable as frozen" at 57%. That is far too lenient -- it would license a metric whose oracle is absent for two amounts in five -- so the probe now reports the measurement and the judgement lives in the amendment. Three further ambiguities, none coded around. The C-frame predicate requires an ink-geometry test that "reads no character identity", but the only implementation of sub-body clustering, derive_size_bands, decides body size from lowercase letters and heading size from uppercase headings -- character identity in both cases -- so the predicate as frozen has no implementation. "Region" is defined as 6-10 PRINTED LINES, which are architecture-derived, and the D-frame exists precisely because H and X can disagree about line segmentation. And no rule exists for comparing their text when line counts differ; x00 compared by (page, ordinal), which only holds when the counts match. Every remaining component -- build_frames, build_oracle, score_metrics, decide_architecture -- consumes the region identity and the M6 unit, so implementing them now would mean inventing these four answers in code and calling the result the frozen protocol. Recorded as A17, status OPEN. No membership change, nothing scored, no holdout document opened, no marker. Refs #553
Ledger-only commit, exempt from F9, closing the declaration loop. Refs #553
A17-N RESOLVED (A19). Collapsed A17.2/A17.3/A17.4 into one root problem and
built the unit below both seams. H and X differ only in who decides word
spaces; both adapters read the same PDFium calls for glyph boxes, origins and
size, so ink geometry is common to both arms and is the legitimate neutral
fact. Generated spaces carry no ink and are excluded, which is what makes the
skeleton identical under both.
neutral line cluster ink glyphs by baseline, tolerance 0.5 x median ink
height on the page -- derived from geometry, not a constant --
anchored on the cluster's first glyph so drift cannot walk a
cluster down the page
identity (document_sha256, page, ordinal), a pure function of geometry;
measured stable under glyph-order shuffling and never reads
text, so duplicate headings cannot collide
regions non-overlapping windows of 8 neutral lines, aligned to page
start, short tail kept, never crossing pages
projection BY GLYPH MEMBERSHIP, plurality, ties to lowest ordinal
Projection by glyph membership replaced baseline proximity because the latter
FAILED its own test: a line merging two neutral lines sits 6pt from both when
the tolerance is 5pt, so it projected to None and silently deleted the
comparison unit for exactly the merge case the D-frame exists to detect.
Membership needs no tolerance and reads no text. A17.4 disappears because
comparison is per neutral line, so differing line counts no longer break
alignment.
THE C-FRAME ENRICHMENT PREDICATE IS WITHDRAWN ON EVIDENCE. Both candidate
geometry-only predicates -- quantised sub-body line height, and
narrow-and-centred -- select 100% of pages on all three development
appropriations bills. A predicate that selects everything enriches nothing.
Rather than tune a fragile classifier toward a target it cannot hit, C-frame
regions are drawn by seeded uniform sample over all neutral regions, at most 8
per document. Page-level enrichment was redundant anyway: structurelessness is
already handled at the DOCUMENT level by the selection frame. The 40%
amount-bearing stratification is retained, being content rather than a heading
classifier.
A17-M6: recommend DEFER (A20). x06's reference is production extract_anchors
over the HYBRID reconstruction -- architecture-derived, not truth -- so the
supported statement is about distance to the nearest hybrid-produced anchor,
not to the true governing heading. A17's earlier wording overstated it and is
corrected. Path A is specified and rejected as disproportionate: the context
spans pages, amounts and account names both repeat, and it is a
financial-semantics oracle inside a seam ADR. Path B removes M6 from rule 1's
vetoes, narrows RQ2 to heading presence/text/boundaries plus immediate
hierarchy, withdraws "the money lands under the right account", and states
plainly that correct headings do NOT establish correct attribution. A real
claim weakening, recorded as one. Status RECOMMENDED, not frozen.
F9 is now BIDIRECTIONAL (item 10): declaring a commit is not enough, every
protected file it touched must be named by a declaration for that commit. That
immediately exposed real gaps -- c111433 touched four protected files while A11
named one, and 0e877b4 was declared with files_touched empty. Rather than
invent per-theme attribution for commits that carried several themes, A18
records the complete commit/file accounting in one reviewable block.
23/23 synthetic geometry fixtures, 35/35 gate self-tests. Nothing scored, no
marker, no holdout opened.
Refs #553
Ledger-only commit, exempt from F9, closing the declaration loop. Refs #553
The review's criticism of A19's projection is CORRECT and is demonstrated rather than conceded: project_by_glyphs received only BASELINES and took the nearest neutral line with no maximum distance, so a glyph at y = -5000 was still assigned to a line. That is baseline inference, not membership, and A19's "membership needs no tolerance" was false of its own code. Its claim that a split appears as two slots was also wrong: both fragments of a split contain glyphs of the SAME neutral line. NEUTRAL GLYPH ID = (document_sha256, page_number, source_char_index), the index in FPDFText_CountChars order. Measured against the adapters: neither contract stores it, because both `continue` past rejected characters, so a list position is not the index. Recording i is pure provenance -- both loops already have it and it changes no extraction decision on either arm. No other exact common identity exists; geometry is a measurement, not an identity. Generated spaces get gid None and can never be a member of a neutral line, which is what makes the skeleton identical under both arms. ELIGIBILITY IS NOW LITERALLY GEOMETRIC. A19 claimed the skeleton reads no codepoint while x07 filtered cp in (10, 13, 32). The filter is removed: a glyph is eligible iff it has a valid finite positive-area ink box and is upright. Measured over 31,729 development characters, the count excluded ONLY by a codepoint rule is 0, so the invariant is true rather than aspirational. PROJECTION IS MODEL G, source-glyph partition by set membership. Spacing is preserved -- an architecture's inserted character is kept when it sits between two retained glyphs -- so the same gid set still yields FAMILYHOUSING from an arm that welded and FAMILY HOUSING from one that did not. The skeleton supplies identity and never supplies spacing. Fragments concatenate in order of first owned gid, so reversing the fragment list cannot change the result. Partition beats plurality on evidence: on a 3/1 merge, plurality sends the whole text to line 0 and blanks line 1, double-counting one glyph and vanishing a physical line; on 50/50 it needs an arbitrary tie-break; on a split it has no defined aggregation. Partition conserves every glyph exactly once and needs no tie-break at all. CROSS-ENGINE CONTROL STRENGTHENED from line counts to geometric correspondence: baselines within 0.5 x median ink height AND x-span overlap >= 0.5, greedy one-to-one. Measured: 256/256 and 258/259 lines matched, median baseline delta 0.0, median x-overlap 1.0. No threshold adopted -- the old 0.95 belonged to a different estimand and is not transplanted. A20 FROZEN with the rationale corrected. Deleted "an attribution difference requires a heading difference first" as too strong: equal heading output does not prove equal attribution, since hierarchy, continuation handling and positional association can differ downstream. The sufficient rationale is that M6 is substantially downstream of the seam, the local-region oracle cannot support it, an independent oracle is a separate long-range study, and it is not needed to answer the seam ADR. 30/30 identity tests. Nothing scored, no marker, no holdout opened. Refs #553
Ledger-only commit, exempt from F9, closing the declaration loop. Refs #553
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft. Methodology only — nothing has been scored, and the executable gate now forbids it.
Stacked on
spike/pdf-backend-bakeoff(#553) at its exact head5419c1c. No commit here touches #553's tree.Current state
What the review found, and what changed
1. The executable gate contradicted the protocol.
x04checked six freeze invariants and printedEXECUTION GATE OPENwhile the protocol's own gate listed two prerequisites it never tested. Now two gates, reported separately: FREEZE INTEGRITY (F1–F6) and EXECUTION READINESS (G1–G4). Every prose condition is an assertion;--self-testdrives 8/8 known-bad cases, including one that feeds G2 evidence recorded on the holdout and requires rejection.2. The protocol was amended after selection. Confirmed. §4.4.1, M9, Rule 0 and the revised selection rules were all written because selection runs surfaced problems, and commit
1350710amended the protocol in the same commit that froze the population. F4 testedfirst_commit, which proves only that some version predated it. Consequences:design_exposure.jsonand excluded;3. The equivalence margin lived on the wrong unit. Measured: zero events on 600 headings → 95 % bound 0.00498; zero events on the 14 documents the protocol itself calls independent → 0.1926. A 39× overstatement, and 598 documents would be needed for 0.005 on the document unit. The zero-event cluster bootstrap is worse — simulated on all-zero clusters it returns [0.0, 0.0] with one distinct statistic, so it carries no information in exactly the expected case. The per-heading margin is withdrawn; the primary outcome is descriptive; the one inferential statement is an exact document-unit bound whose weakness is stated.
4. A shared failure could vanish from RQ2. Rule 0 excluded documents where both architectures lose the heading tree. Neutral for RQ1, but for RQ2 that conditions absolute correctness on successful extraction. Shared failures are now retained as failures in RQ2 and M9; only §4.6's source-level exclusions (properties of the file) may remove a frozen document.
5. M3 was a token multiset. It discards order and charges character errors to the seam. Redefined on aligned character boundaries with
WELD/SPLIT/OK/TEXT_ERROR/UNALIGNABLE;UNALIGNABLEis reported split by discordance status, and the decision rule consumesWELD/SPLITonly.Also narrowed: M2 normalises whitespace runs, so it licenses "exact up to whitespace-run normalisation", not "character for character". And the oracle is AI image-adjudication — a 25-item human audit estimates agreement, it does not make the key ground truth. Because the D-frame is the only evidence that can flip the ADR and is expected to be tiny, every D-frame item now requires human adjudication.
The confirmatory population
17 documents, seed 20260808, disjoint from the 37 design-exposed documents (asserted by F3, not claimed).
mods_liveness78/78.Adequacy downgrades to
sampled-classes-only(6/8 strata), reported rather than repaired:Raising that budget now would change a sampling rule after seeing what the sample produced, which §4.7 forbids and which is the exact defect this pass corrects.
P-head is 12 documents / 2,864 pages, so the tightest achievable zero-event document-unit bound is ≈ 22 %. §8.3 already states this holdout can fail to falsify equivalence and bound it loosely, and cannot establish a small per-document rate.
Still required before execution
G1 the corrected extended-glyph adapter; G2 its X2-a/X2-b evidence recorded on development documents; G3 the adjudicator prompt. All three are machine-checked.
🤖 Generated with Claude Code