fix(census): carry backslash-continued non-raw strings across lines in strip_noncode#5715
Merged
Merged
Conversation
…n strip_noncode The census lexer closed a non-raw string only if its quote closed on the same line. Rust's STRING_CONTINUE says otherwise: a `\` immediately before the newline escapes it and the literal RUNS ON -- which is how every wrapped `assert!` message in the tree is written. Scanned as code, the continuation body was the raw-string ceiling (#5704) and the lifetime ceiling (#5705) a third time, and it failed in all three of their directions at once: a `//` in the body opened a phantom comment (`Assault // Battery` is real message text in this tree), a `{` LEAKED into the brace counter, and an `add_to_zone(` in a message was a FALSE HIT. Population, measured (instrument: an independent Rust lexer over the census scopes, 434 scanned files / 919,557 lines): 1,114 continued-string lines in the files the census scans, 872 more in the name-excluded ones. On production lines the old lexer emitted 204 string-body lines as code and leaked 160 braces out of them. The leak was live-by-luck: they balanced, so no skip region moved -- an unbalanced one desyncs brace tracking and either raises CensusError or silently swallows the production code after it. ScanState now carries the third multi-line construct alongside block-comment depth and raw-string `#` count. The three are mutually exclusive by construction: a `\` is inert inside a raw string (no escapes at all), and either string inside a block comment is comment text. Old-vs-new whole-scope sweep: 204 code-text divergences, ALL of them string body that is no longer scanned as code; 0 lines added to or removed from the production stream; 0 hit-set changes; 0 CensusError either way. Both baselines (zone, draw-replacement) byte-identical, `--write` idempotent, all three gates in check-engine-authorities.sh green. Throughput 18.8 MB/s CPU (best-of-5, process_time, 5 largest engine/src files), up from 17.9: a continuation line now short-circuits instead of being scanned character by character. Tests: 11 red-first cases in the seam suite (8 failures + 1 CensusError against the old lexer) covering false hit, leaked brace, phantom line comment, phantom block comment, 3-line continuation, cfg(test) region toggle and desync, the b/c prefixed forms, and the minimal pairs an escaped backslash and a raw string make.
matthewevans
enabled auto-merge
July 13, 2026 03:20
Contributor
|
Warning You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again! |
andriypolanski
pushed a commit
to andriypolanski/phase
that referenced
this pull request
Jul 13, 2026
…aw text (phase-rs#76) (phase-rs#5718) Family (D) of the parser-combinator gate finds an `alt((...))` block's end by counting parentheses over RAW text. That count is a lex, and a lex that reads literals and comments as code is the ceiling class that has now bitten this repo's census scanner three times (raw strings phase-rs#5704, lifetimes phase-rs#5705, backslash-continued strings phase-rs#5715). Here it has teeth in BOTH directions, because the gate it feeds is commit-blocking: FALSE HIT an `alt((` written inside a COMMENT opens a phantom block, which swallows the ordinary tag() bindings below it and blocks a good commit over an `alt` that does not exist. FALSE MISS a comment carrying `))` exhausts the counter's depth headroom and truncates a REAL block: the arms are never collected and a genuine cross-product alt ships past the gate built to stop it. This is string-matching parsing landing on main with a green gate. Both are witnessed and committed as tests. In-tree today the bug is live-by-luck: 9 of 2,853 alt blocks in crates/engine/src/parser have their end line computed wrong (all of them phantom blocks opened by an `alt((` inside a comment), but zero FLAG DECISIONS change. Equivalence sweep old-vs-new over every block, with every line treated as added: 0 decision changes, and the block count falls 2,853 -> 2,844, exactly the 9 phantoms and nothing else. Positive control on the sweep harness: it reports a decision change on both witnesses, so the zero is a finding rather than a dead probe. DEPENDENCY DIRECTION, deliberately narrow: family (D) now imports the census module's lexer (scripts/zone_authority_census.py). Single authority for CODE-STREAM lexing -- explicitly NOT for literal matching. The detector reads STRUCTURE (where does `alt((` open, where do its parens balance?) from the stripped code stream, and CONTENT (which literals are the arms? is there an allow-noncombinator annotation?) from the raw text. Two views, one grammar. The same commit documents why the gate's OTHER families must NOT follow. (A), (B), (E) and (F) match ON the string literal (`.contains("`, `"lit" =>`, `== "long"`, `name: "..."`); routing them through strip_noncode would delete the very quote they key on and BLIND the gate -- it would pass string-matching parser code forever while reporting green. Measured during phase-rs#76: 194 real match arms go unmatched that way. Their exposure to this class is the benign mirror image, a false hit that is loud and already escape-hatched (`// allow-noncombinator:`); phase-rs#76 measured 10 such lines in the whole parser scope, all doc comments describing the forbidden pattern. A comment block at the grep definitions pins this where a future "hardener" would edit. The detector's seam suite runs from the gate (section D0), ahead of the scan it protects -- the same shape as section (B0) of check-engine-authorities.sh, and for the same reason: a lexing regression here does not FAIL this gate, it silently mis-scopes it. Watched red (a deliberately broken suite fails the gate) and green. Zone and draw baselines are untouched and byte-identical; this change is gate-internal. Audit outcome for the other scanners in phase-rs#76: scripts/audit-parser.py is NOT A MEMBER (it never opens a .rs file -- its only input is card-data.json), and scripts/gen-test-fixture.py is benign-directional (over-harvest is fixture bloat; a missed card is loud at test runtime). Neither is changed. Co-authored-by: matthewevans <matthewevans@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The census lexer closed a non-raw string only if its quote closed on the same
line. Rust's STRING_CONTINUE says otherwise: a
\immediately before the newlineescapes it and the literal RUNS ON -- which is how every wrapped
assert!messagein the tree is written. Scanned as code, the continuation body was the raw-string
ceiling (#5704) and the lifetime ceiling (#5705) a third time, and it failed in
all three of their directions at once: a
//in the body opened a phantom comment(
Assault // Batteryis real message text in this tree), a{LEAKED into thebrace counter, and an
add_to_zone(in a message was a FALSE HIT.Population, measured (instrument: an independent Rust lexer over the census
scopes, 434 scanned files / 919,557 lines): 1,114 continued-string lines in the
files the census scans, 872 more in the name-excluded ones. On production lines
the old lexer emitted 204 string-body lines as code and leaked 160 braces out of
them. The leak was live-by-luck: they balanced, so no skip region moved -- an
unbalanced one desyncs brace tracking and either raises CensusError or silently
swallows the production code after it.
ScanState now carries the third multi-line construct alongside block-comment depth
and raw-string
#count. The three are mutually exclusive by construction: a\is inert inside a raw string (no escapes at all), and either string inside a block
comment is comment text.
Old-vs-new whole-scope sweep: 204 code-text divergences, ALL of them string body
that is no longer scanned as code; 0 lines added to or removed from the production
stream; 0 hit-set changes; 0 CensusError either way. Both baselines (zone,
draw-replacement) byte-identical,
--writeidempotent, all three gates incheck-engine-authorities.sh green. Throughput 18.8 MB/s CPU (best-of-5,
process_time, 5 largest engine/src files), up from 17.9: a continuation line now
short-circuits instead of being scanned character by character.
Tests: 11 red-first cases in the seam suite (8 failures + 1 CensusError against
the old lexer) covering false hit, leaked brace, phantom line comment, phantom
block comment, 3-line continuation, cfg(test) region toggle and desync, the b/c
prefixed forms, and the minimal pairs an escaped backslash and a raw string make.