automata: fix leftmost-first miss in the reverse suffix optimization#1373
Open
RECHE23 wants to merge 1 commit into
Open
automata: fix leftmost-first miss in the reverse suffix optimization#1373RECHE23 wants to merge 1 commit into
RECHE23 wants to merge 1 commit into
Conversation
The reverse suffix optimization prefilters on the longest common suffix and processes candidates in order of match end, yielding the first reverse+forward confirmed match. That is only correct when the leftmost starting match also has the smallest candidate end. When one exact suffix literal (an entire match of one alternation branch) is a proper suffix of another suffix literal, a match can be nested inside the tail of an overlapping match that ends later but starts earlier: the first confirmed candidate is then not leftmost, and the (correct in itself) anti-quadratic min_start clamp makes the missed match permanent. For example, 'a|.aa' on '-#aa' reported [(2,3), (3,4)] instead of the leftmost-first [(1,4)]. Fix conservatively: decline the optimization when any exact suffix literal is a proper suffix of another suffix literal, falling back to the core engine. Common beneficiaries (single branches, all-inexact seqs such as '\w+@\w+', non-nested alternations like 'foo|bar') keep the optimization. Found by a differential fuzzer (REAL-regex vs this crate); minimized by hand.
RECHE23
added a commit
to RECHE23/real-regex
that referenced
this pull request
Jul 5, 2026
The upstream fix is up as a PR (rust-lang/regex#1373). The root cause is sharper than "prefilter resume": the reverse-suffix optimization orders candidates by their END, but when one alternation branch's exact literal is a proper suffix of another branch's match, ordering by end breaks the leftmost-by-START rule (the [E("a"), I("aa")] shape). fuzz/known_rust_bugs/README and the crate README's divergence note now point at #1373 and carry the corrected root cause — the "found by REAL's differential fuzzer" line now lives in their tracker.
RECHE23
added a commit
to RECHE23/real-regex
that referenced
this pull request
Jul 5, 2026
…s record — INNER-LITERAL IL.3
The theme closer. Re-running the duel with the route on gives the honest verdict, and one deferral turned the
headline line to parity: the reverse DFA is now built LAZILY, on the first candidate — a no-match haystack (the
literal never appears) pays only the memmem and never builds a reverse it would not use. The date
`\d{4}-\d{2}-\d{2}` no-match line went from ~201x rust to 0.020 vs 0.013 ns/B — 1.6x, the V0 target of <=2x met.
The duel gains the sparse rows the V0 debt named (date-sparse 4.2x, email-sparse 7.2x — the reverse+confirm
per match trails rust's tighter loop but sits far from 201x) and the key= flagship (1.9x). The class / digit /
field rows are unchanged (REAL still 5-6x ahead): the route only fires for a required inner literal, and the
exhaustive confirms it byte-identical to the core (serious=0, route on, 3.21M cases). BENCHMARKS.md §E.5
records the table.
design.dox §7.8 documents the full arc (extract -> boundary -> prefix compile -> reverse -> loop), the two
traps (the reverse DFA's dangling byte-program span; the dynamic VM's own state type leaving the route inert)
and the lesson verbatim — the reverse bound advances only on a yield, the linearity guard is the forward
backstop, not the reverse bound; it read as a missing heuristic but was a one-line wiring bug. It cites
rust-lang/regex#1373 (their sibling leftmost bug, found by REAL's fuzzer — the symmetry is instructive) and
names the alternation-sibling and multi-literal follow-ups. Full gate green.
RECHE23
added a commit
to RECHE23/real-regex
that referenced
this pull request
Jul 5, 2026
…3rd differential fuzzer, crate perf)
The first large train of the accumulation rhythm. Bumps the five version surfaces (pyproject, __init__,
version.hpp, Cargo.toml, Cargo.lock; CMake derives) to 2026.7.22 and adds the release notes.
Headline: the inner-literal prefilter decapitates the date line — `\d{4}-\d{2}-\d{2}` no-match went from 201x
of the regex crate to 1.9x (sparse haystacks 30-100x better, key= now beats the crate, dense email at parity,
class/digit/field unchanged; exhaustive serious=0 with the route on). Plus the third differential fuzzer
(real-regex vs the regex crate) and its four fixed divergence classes (empty-match iteration, $ end-only flag,
class-set-syntax declines, and the upstream rust-lang/regex#1373 it found), and find_iter now outruns the
crate on span throughput. Full notes in docs/release-notes/v2026.7.22.md. Full gate green.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The reverse suffix optimization can skip a match that starts earlier than the matches it reports, violating leftmost-first semantics.
Repro (1.12.4, default config)
[1,4)(".aa" matching#aa) is a match and starts leftmost. (Original fuzz find:\0|.\0\0on"\0\0\0\0\n\x0b\0\0\0", missing(5,8); minimized by hand.)Root cause
With
RUST_LOG=trace: suffixes extract asSeq[E("a"), I("aa")], the LCS prefilter ismemchr('a'), andReverseSuffixis engaged. Candidates are processed in order of match end, and the first reverse+forward confirmed match is yielded. That is only correct when the leftmost-starting match also has the smallest candidate end at/after the resume point. Here branch 1's entire match (E("a")) is a proper suffix of branch 2's suffix ("aa"): a branch-1 match nests inside the tail of an overlapping branch-2 match that ends later but starts earlier. Candidate end 3 confirms(2,3)and is yielded; the true leftmost[1,4)(candidate end 4) is then behind the (correct in itself) anti-quadraticmin_startclamp, so the miss is permanent.The fix
Conservative: decline the optimization in
ReverseSuffix::newwhen any exact suffix literal is a proper suffix of another suffix literal — the precise nesting that breaks the end-order ⇒ start-order assumption — falling back to the core engine. Common beneficiaries keep the optimization (single-branch suffixes, all-inexact seqs such as\w+@\w+, non-nested alternations likefoo|bar).A complete fix that keeps the optimization for these patterns would need candidate look-ahead across ends before yielding when overlap is possible; happy to explore that as a follow-up if you'd prefer.
Validation
tests/regression_fuzz.rs(fails before, passes after).cargo test --test integration(66) andcargo test -p regex-automata --lib(137): pass.Found by a differential fuzzer (REAL-regex vs this crate).