Skip to content

fix(gate): lex the cross-product detector's alt blocks as CODE, not raw text (#76)#5718

Merged
matthewevans merged 1 commit into
mainfrom
ship/crossprod-lexer
Jul 13, 2026
Merged

fix(gate): lex the cross-product detector's alt blocks as CODE, not raw text (#76)#5718
matthewevans merged 1 commit into
mainfrom
ship/crossprod-lexer

Conversation

@matthewevans

Copy link
Copy Markdown
Member

Family (D) of the parser-combinator gate finds an alt((...)) block's end by
counting parentheses over RAW text. That count is a lex, and a lex that reads
literals and comments as code is the ceiling class that has now bitten this
repo's census scanner three times (raw strings #5704, lifetimes #5705,
backslash-continued strings #5715). Here it has teeth in BOTH directions,
because the gate it feeds is commit-blocking:

FALSE HIT an alt(( written inside a COMMENT opens a phantom block, which
swallows the ordinary tag() bindings below it and blocks a good
commit over an alt that does not exist.
FALSE MISS a comment carrying )) exhausts the counter's depth headroom and
truncates a REAL block: the arms are never collected and a genuine
cross-product alt ships past the gate built to stop it. This is
string-matching parsing landing on main with a green gate.

Both are witnessed and committed as tests. In-tree today the bug is live-by-luck:
9 of 2,853 alt blocks in crates/engine/src/parser have their end line computed
wrong (all of them phantom blocks opened by an alt(( inside a comment), but
zero FLAG DECISIONS change. Equivalence sweep old-vs-new over every block, with
every line treated as added: 0 decision changes, and the block count falls
2,853 -> 2,844, exactly the 9 phantoms and nothing else. Positive control on the
sweep harness: it reports a decision change on both witnesses, so the zero is a
finding rather than a dead probe.

DEPENDENCY DIRECTION, deliberately narrow: family (D) now imports the census
module's lexer (scripts/zone_authority_census.py). Single authority for
CODE-STREAM lexing -- explicitly NOT for literal matching. The detector reads
STRUCTURE (where does alt(( open, where do its parens balance?) from the
stripped code stream, and CONTENT (which literals are the arms? is there an
allow-noncombinator annotation?) from the raw text. Two views, one grammar.

The same commit documents why the gate's OTHER families must NOT follow. (A),
(B), (E) and (F) match ON the string literal (.contains(", "lit" =>,
== "long", name: "..."); routing them through strip_noncode would delete the
very quote they key on and BLIND the gate -- it would pass string-matching parser
code forever while reporting green. Measured during #76: 194 real match arms go
unmatched that way. Their exposure to this class is the benign mirror image, a
false hit that is loud and already escape-hatched (// allow-noncombinator:);
#76 measured 10 such lines in the whole parser scope, all doc comments describing
the forbidden pattern. A comment block at the grep definitions pins this where a
future "hardener" would edit.

The detector's seam suite runs from the gate (section D0), ahead of the scan it
protects -- the same shape as section (B0) of check-engine-authorities.sh, and
for the same reason: a lexing regression here does not FAIL this gate, it
silently mis-scopes it. Watched red (a deliberately broken suite fails the gate)
and green. Zone and draw baselines are untouched and byte-identical; this change
is gate-internal.

Audit outcome for the other scanners in #76: scripts/audit-parser.py is NOT A
MEMBER (it never opens a .rs file -- its only input is card-data.json), and
scripts/gen-test-fixture.py is benign-directional (over-harvest is fixture bloat;
a missed card is loud at test runtime). Neither is changed.

…aw text (#76)

Family (D) of the parser-combinator gate finds an `alt((...))` block's end by
counting parentheses over RAW text. That count is a lex, and a lex that reads
literals and comments as code is the ceiling class that has now bitten this
repo's census scanner three times (raw strings #5704, lifetimes #5705,
backslash-continued strings #5715). Here it has teeth in BOTH directions,
because the gate it feeds is commit-blocking:

  FALSE HIT   an `alt((` written inside a COMMENT opens a phantom block, which
              swallows the ordinary tag() bindings below it and blocks a good
              commit over an `alt` that does not exist.
  FALSE MISS  a comment carrying `))` exhausts the counter's depth headroom and
              truncates a REAL block: the arms are never collected and a genuine
              cross-product alt ships past the gate built to stop it. This is
              string-matching parsing landing on main with a green gate.

Both are witnessed and committed as tests. In-tree today the bug is live-by-luck:
9 of 2,853 alt blocks in crates/engine/src/parser have their end line computed
wrong (all of them phantom blocks opened by an `alt((` inside a comment), but
zero FLAG DECISIONS change. Equivalence sweep old-vs-new over every block, with
every line treated as added: 0 decision changes, and the block count falls
2,853 -> 2,844, exactly the 9 phantoms and nothing else. Positive control on the
sweep harness: it reports a decision change on both witnesses, so the zero is a
finding rather than a dead probe.

DEPENDENCY DIRECTION, deliberately narrow: family (D) now imports the census
module's lexer (scripts/zone_authority_census.py). Single authority for
CODE-STREAM lexing -- explicitly NOT for literal matching. The detector reads
STRUCTURE (where does `alt((` open, where do its parens balance?) from the
stripped code stream, and CONTENT (which literals are the arms? is there an
allow-noncombinator annotation?) from the raw text. Two views, one grammar.

The same commit documents why the gate's OTHER families must NOT follow. (A),
(B), (E) and (F) match ON the string literal (`.contains("`, `"lit" =>`,
`== "long"`, `name: "..."`); routing them through strip_noncode would delete the
very quote they key on and BLIND the gate -- it would pass string-matching parser
code forever while reporting green. Measured during #76: 194 real match arms go
unmatched that way. Their exposure to this class is the benign mirror image, a
false hit that is loud and already escape-hatched (`// allow-noncombinator:`);
#76 measured 10 such lines in the whole parser scope, all doc comments describing
the forbidden pattern. A comment block at the grep definitions pins this where a
future "hardener" would edit.

The detector's seam suite runs from the gate (section D0), ahead of the scan it
protects -- the same shape as section (B0) of check-engine-authorities.sh, and
for the same reason: a lexing regression here does not FAIL this gate, it
silently mis-scopes it. Watched red (a deliberately broken suite fails the gate)
and green. Zone and draw baselines are untouched and byte-identical; this change
is gate-internal.

Audit outcome for the other scanners in #76: scripts/audit-parser.py is NOT A
MEMBER (it never opens a .rs file -- its only input is card-data.json), and
scripts/gen-test-fixture.py is benign-directional (over-harvest is fixture bloat;
a missed card is loud at test runtime). Neither is changed.
@matthewevans
matthewevans enabled auto-merge July 13, 2026 03:51
@gemini-code-assist

Copy link
Copy Markdown
Contributor

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

@matthewevans
matthewevans added this pull request to the merge queue Jul 13, 2026
Merged via the queue into main with commit d1e7198 Jul 13, 2026
13 checks passed
@matthewevans
matthewevans deleted the ship/crossprod-lexer branch July 13, 2026 04:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant