Skip to content

test(#374): the guard was blind to the imaging lane, because it could not read yes/no columns - #410

Merged
Agastya191 merged 0 commit into
mainfrom
fix/guard-reads-yes-no-columns
Jul 31, 2026
Merged

test(#374): the guard was blind to the imaging lane, because it could not read yes/no columns#410
Agastya191 merged 0 commit into
mainfrom
fix/guard-reads-yes-no-columns

Conversation

@sebasmos

Copy link
Copy Markdown
Member

Why the imaging defect escaped a guard built to catch it

_binary returns None for "yes"/"no", and every imaging arm records its reads that way (clean, iso, shared). So constant_columns and duplicate_columns skipped the entire imaging lane. That is why the no-cue arms' iso == clean identity reached the paper twice: the guard could not read the columns the defect lives in.

Why not just add yes/no to the truthy sets

Tried it. 113 findings, nearly all definitional: iso vs iso_adopt is NOT iso, because imaging_cascade.py sets wrong = "no" unconditionally. That is how a useful screen becomes an ignored one, and it is the exemption-flood failure mode @Agastya191 warned about on #389.

The targeted screen

identical_reads compares reads to reads only, and skips any pair where one name is a prefix of the other. Derivations are notation; two separate reads coming back identical is a measurement that never happened.

Five findings, two of which nobody had found by review:

file locus rows
natural_independent/imaging_cascade_none clean vs iso 150/150
natural_cues/imaging_cascade_none clean vs iso 150/150
device_absent/imaging_cascade_none clean vs iso 150/150
system_flag/imaging_system_flag clean vs iso 150/150, new
imaging_peer_size_curve k1 vs k2 35/35, new
  • system-flag: same root cause, iso_adopt 0/150 by construction. No published number affected: the only figure either paper takes from that family is the placebo rate 1/150 = 0.007, reported as a raw rate, not as a difference against the degenerate isolated arm.
  • peer-size curve: worse in kind. One peer and two peers return identical reads on every case (k1_adopt 34/35, k2_adopt 34/35), so that segment is not a measurement of committee size, most likely a cache-key collision between the two prompts. k4 does differ at 35/35. Cited in neither paper.

All five recorded in the exemption file with the root cause and what does or does not rest on them.

The ratchet, moved on purpose

PREEXISTING_CEILING 113 -> 116. The raise is detection, not regression: one screen turning on surfaced five pre-existing instances at once. The reason is in the constant's comment as well as the commit, since the ratchet exists to force exactly this justification. Lowering it is always fine.

Anti-vacuity

Two tests, on synthetic fixtures rather than the committed files the screen exists to clear, so fixing those files cannot turn the guard's own suite red. One proves it fires on identical reads and stays silent on reads that genuinely diverge; the other proves it ignores a derived column.

1147 passed, 7 skipped. ruff clean.

@Agastya191 Agastya191 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good work on degeneracy.py and on the exemption entries: reading yes/no was the right diagnosis, and the prefix skip is what makes identical_reads usable where widening _binary was not. One problem is that identical_reads skips a constant column with the comment that a constant read is screen 1's business, but constant_columns normalises through _binary, which returns None for yes/no, so a constant read is screened by nothing at all: k4 is "no" on all 35 rows of imaging_peer_size_curve.jsonl and shared is "no" on all 35 of imaging_cascade_cable.jsonl, each identical to wrong on every row. That makes k4_adopt 35/35 and shared_adopt 35/35 by construction, so the k4 condition you cite as the one that does differ is degenerate in its own right, and the cable arm's shared adoption is unconditional rather than measured. You have multiple options here; I would drop the constancy continue and let this screen report constant read pairs too, since it is the only screen that can currently see them.

@Agastya191
Agastya191 merged this pull request into main Jul 31, 2026
@Agastya191
Agastya191 deleted the fix/guard-reads-yes-no-columns branch July 31, 2026 16:34
sebasmos added a commit that referenced this pull request Aug 4, 2026
… not read yes/no columns (#410)

`_binary` returns None for "yes"/"no", and every imaging arm records its reads that way (`clean`,
`iso`, `shared`). So both pair screens skipped the entire imaging lane. That is why the no-cue
arms' `iso == clean` identity reached the paper twice before anyone read the rows: the guard could
not see the columns the defect lives in.

Adding yes/no to _TRUE/_FALSE was the wrong fix. It produced 113 findings, nearly all definitional
(`iso` vs `iso_adopt` is NOT iso, since `wrong` is always "no"), which is how a useful screen
becomes an ignored one. So this adds a targeted screen instead:

`identical_reads` compares reads to reads only, and skips any pair where one name is a prefix of
the other, so derivations do not drown the signal. Two separate reads coming back identical is a
measurement that never happened.

Five findings, and two of them nobody had found by review:

  natural_independent / natural_cues / device_absent  imaging_cascade_none  clean vs iso, 150/150
  system_flag                        imaging_system_flag  clean vs iso, 150/150   <- new
  imaging_peer_size_curve            k1 vs k2, 35/35                              <- new

The system-flag arm has the same root cause and iso_adopt 0/150 by construction; no published
number is affected, because the only figure either paper takes from that family is the placebo rate
1/150 = 0.007, reported as a raw rate. The peer-size curve is worse in kind: one peer and two peers
return identical reads on every case, so that segment is not a measurement of committee size, most
likely a cache-key collision between the two prompts. k4 does differ. Cited in neither paper.

All five recorded in the exemption file with the root cause and what does or does not rest on them.

PREEXISTING_CEILING raised 113 -> 116, deliberately. The raise is detection, not regression: one
screen turning on surfaced five pre-existing instances at once. The reason is in the constant's
comment as well as here, since the ratchet exists to force exactly this justification.

Two anti-vacuity tests added, on synthetic fixtures rather than on the committed files the screen
exists to clear, so fixing those files cannot turn the guard's own suite red. One proves it fires on
identical reads and stays silent on reads that genuinely diverge; the other proves it ignores a
derived column.

1147 passed, 7 skipped. ruff clean.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants