test(#374): the guard was blind to the imaging lane, because it could not read yes/no columns - #410
Conversation
Agastya191
left a comment
There was a problem hiding this comment.
good work on degeneracy.py and on the exemption entries: reading yes/no was the right diagnosis, and the prefix skip is what makes identical_reads usable where widening _binary was not. One problem is that identical_reads skips a constant column with the comment that a constant read is screen 1's business, but constant_columns normalises through _binary, which returns None for yes/no, so a constant read is screened by nothing at all: k4 is "no" on all 35 rows of imaging_peer_size_curve.jsonl and shared is "no" on all 35 of imaging_cascade_cable.jsonl, each identical to wrong on every row. That makes k4_adopt 35/35 and shared_adopt 35/35 by construction, so the k4 condition you cite as the one that does differ is degenerate in its own right, and the cable arm's shared adoption is unconditional rather than measured. You have multiple options here; I would drop the constancy continue and let this screen report constant read pairs too, since it is the only screen that can currently see them.
… not read yes/no columns (#410) `_binary` returns None for "yes"/"no", and every imaging arm records its reads that way (`clean`, `iso`, `shared`). So both pair screens skipped the entire imaging lane. That is why the no-cue arms' `iso == clean` identity reached the paper twice before anyone read the rows: the guard could not see the columns the defect lives in. Adding yes/no to _TRUE/_FALSE was the wrong fix. It produced 113 findings, nearly all definitional (`iso` vs `iso_adopt` is NOT iso, since `wrong` is always "no"), which is how a useful screen becomes an ignored one. So this adds a targeted screen instead: `identical_reads` compares reads to reads only, and skips any pair where one name is a prefix of the other, so derivations do not drown the signal. Two separate reads coming back identical is a measurement that never happened. Five findings, and two of them nobody had found by review: natural_independent / natural_cues / device_absent imaging_cascade_none clean vs iso, 150/150 system_flag imaging_system_flag clean vs iso, 150/150 <- new imaging_peer_size_curve k1 vs k2, 35/35 <- new The system-flag arm has the same root cause and iso_adopt 0/150 by construction; no published number is affected, because the only figure either paper takes from that family is the placebo rate 1/150 = 0.007, reported as a raw rate. The peer-size curve is worse in kind: one peer and two peers return identical reads on every case, so that segment is not a measurement of committee size, most likely a cache-key collision between the two prompts. k4 does differ. Cited in neither paper. All five recorded in the exemption file with the root cause and what does or does not rest on them. PREEXISTING_CEILING raised 113 -> 116, deliberately. The raise is detection, not regression: one screen turning on surfaced five pre-existing instances at once. The reason is in the constant's comment as well as here, since the ratchet exists to force exactly this justification. Two anti-vacuity tests added, on synthetic fixtures rather than on the committed files the screen exists to clear, so fixing those files cannot turn the guard's own suite red. One proves it fires on identical reads and stays silent on reads that genuinely diverge; the other proves it ignores a derived column. 1147 passed, 7 skipped. ruff clean.
Why the imaging defect escaped a guard built to catch it
_binaryreturnsNonefor"yes"/"no", and every imaging arm records its reads that way (clean,iso,shared). Soconstant_columnsandduplicate_columnsskipped the entire imaging lane. That is why the no-cue arms'iso == cleanidentity reached the paper twice: the guard could not read the columns the defect lives in.Why not just add yes/no to the truthy sets
Tried it. 113 findings, nearly all definitional:
isovsiso_adoptisNOT iso, becauseimaging_cascade.pysetswrong = "no"unconditionally. That is how a useful screen becomes an ignored one, and it is the exemption-flood failure mode @Agastya191 warned about on #389.The targeted screen
identical_readscompares reads to reads only, and skips any pair where one name is a prefix of the other. Derivations are notation; two separate reads coming back identical is a measurement that never happened.Five findings, two of which nobody had found by review:
natural_independent/imaging_cascade_noneclean vs isonatural_cues/imaging_cascade_noneclean vs isodevice_absent/imaging_cascade_noneclean vs isosystem_flag/imaging_system_flagclean vs isoimaging_peer_size_curvek1 vs k2iso_adopt0/150 by construction. No published number affected: the only figure either paper takes from that family is the placebo rate 1/150 = 0.007, reported as a raw rate, not as a difference against the degenerate isolated arm.All five recorded in the exemption file with the root cause and what does or does not rest on them.
The ratchet, moved on purpose
PREEXISTING_CEILING113 -> 116. The raise is detection, not regression: one screen turning on surfaced five pre-existing instances at once. The reason is in the constant's comment as well as the commit, since the ratchet exists to force exactly this justification. Lowering it is always fine.Anti-vacuity
Two tests, on synthetic fixtures rather than the committed files the screen exists to clear, so fixing those files cannot turn the guard's own suite red. One proves it fires on identical reads and stays silent on reads that genuinely diverge; the other proves it ignores a derived column.
1147 passed, 7 skipped.
ruffclean.