diff --git a/doc/plans/glob-perf-nfa-improvements.md b/doc/plans/glob-perf-nfa-improvements.md new file mode 100644 index 0000000..e325b42 --- /dev/null +++ b/doc/plans/glob-perf-nfa-improvements.md @@ -0,0 +1,177 @@ +# glob_perf — NFA improvement opportunities + +Investigation roadmap drawn from +[DataDog/experimental/users/dangermike/glob_perf](https://github.com/DataDog/experimental/tree/main/users/dangermike/glob_perf). +glob_perf is a cross-language benchmark of multi-pattern glob matching +(Go/Java/Rust, 64 – 65 536 patterns). reggie is single-pattern compile-to-bytecode, +so trie/multi-pattern machinery does **not** port; the NFA-execution +micro-optimizations do. + +This is a **research roadmap with pointers**, not a build plan. Designs +are sketched; implementation should be brainstormed/planned per item +before code is written. + +## Headline result from glob_perf + +`lazydfa` (NFA-trie + lazily materialized DFA cache) is the consistent +winner across languages — beats both pure NFA (`fasttrie`) and Intel +Hyperscan once warm. Java 256-pattern `static/randomkey`: lazydfa +214 ns/op vs fasttrie 300, hyperscan 5101. Hyperscan loses on build +time (~13 s at 16 k patterns) — informative non-goal for reggie's +compile-once model. + +## glob_perf source pointers + +(GitHub paths under `users/dangermike/glob_perf/`) + +| File | What to read it for | +|---|---| +| `README.md` | Methodology, hit/miss split, headline numbers. | +| `README_rules.md` | Glob grammar — irrelevant for porting. | +| `java/src/main/java/.../LazyDFA.java` | Reference impl of R1+R2+R5 below. | +| `java/src/main/java/.../FastTrie.java` | SoA `int[]` keys + binary search reference (R4). | +| `docs/` | Cross-language perf write-up. | + +## Recommendations + +### R1 — Lazy DFA layer over NFA + +**Problem.** `OPTIMIZED_NFA` (chosen by `PatternAnalyzer.java:726, 741`) +recomputes `closure(stateSet, c)` for every input character. Eager +subset construction (`automaton/SubsetConstructor.java`) is too +expensive for many of these patterns — that's why we route to NFA in +the first place. + +**Idea.** Intern NFA state-sets to DFA states **on first encounter**, +cap the cache (e.g. 4 k states), fall back to plain NFA stepping when +the cache fills. + +**Pointers.** +- New generator next to + `reggie-codegen/src/main/java/com/datadoghq/reggie/codegen/codegen/NFABytecodeGenerator.java` +- Strategy switch in + `reggie-codegen/src/main/java/com/datadoghq/reggie/codegen/analysis/PatternAnalyzer.java` + — gate by state count and lookaround presence. +- Reference: `glob_perf/.../LazyDFA.java` lines around `217-224` for + the state-set interning key (sorted IDs packed into 2-chars-per-int + string + `putIfAbsent`). + +**Caveats.** +- Cache must be bounded; glob_perf documents a 300× hit-path regression + at 65 k patterns when the cache stops fitting. +- reggie matchers may be single-threaded — confirm in + `reggie-runtime/src/main/java/com/datadoghq/reggie/runtime/ReggieMatcher.java` + before adopting glob_perf's `AtomicReferenceArray`. Plain `int[]` / + `Object[]` recovers a documented 2-3× Java-vs-Go gap. + +### R2 — Per-state ASCII transition table + +**Idea.** Each cached DFA state gets an `int[128]` of target state IDs, +indexed by `c & 0x7F` after a `c < 128` test. Two sentinels: +- `-1` (or `null` in `Object[]` form) = uncached → compute and fill. +- A `DEAD` constant = computed-dead → reject without retry. + +**Pointers.** +- Emit in the new lazy-DFA generator (paired with R1). +- Retrofittable to `DFASwitchBytecodeGenerator.java` (states 50–300) + if profiling shows the switch dispatch is slower than the table. +- Reference: `LazyDFA.java` lines around `31, 321-326`. + +### R3 — Inline `long[]` bitset ops in generated bytecode + +**Idea.** Verify generated NFA bytecode uses inlined `LOR`/`LAND`/ +`LSHL` for state-set updates on small-state patterns rather than +`INVOKEVIRTUAL StateSet.add`. + +**Pointers.** +- `NFABytecodeGenerator.java` already selects between primitive `long` + / dual-long / `BitSet` / `SparseSet` paths — check the small-state + branch emits direct bitwise ops, not method calls. +- Reference: `LazyDFA.java` lines around `363-365` (`bset/bclr/bhas`). +- Runtime helpers: + `reggie-runtime/src/main/java/com/datadoghq/reggie/runtime/SparseSet.java`, + `StateSet.java`. + +### R4 — Struct-of-arrays automaton representation + +**Idea.** Store NFA/DFA transitions as parallel `int[] keys` + +`int[] targets`, binary-searched. ~16 keys per cache line vs ~4 for +interleaved `(char,int)` records. glob_perf's `FastTrie` Java impl +wins precisely on this. + +**Pointers.** +- `reggie-codegen/src/main/java/com/datadoghq/reggie/codegen/automaton/NFA.java` +- `reggie-codegen/src/main/java/com/datadoghq/reggie/codegen/automaton/DFA.java` +- Reference: `FastTrie.java` lines around `54-55, 183-194`. + +**Caveat.** This is a compile-time representation change. Audit all +readers (subset constructor, bytecode generators) for assumptions +about edge layout before refactoring. + +### R5 — Per-matcher thread-local scratch + +**Idea.** Reuse `nextBuf` / bitset buffers across `matches()` calls so +steady-state allocs/op drop to 0. + +**Pointers.** +- Base class: + `reggie-runtime/src/main/java/com/datadoghq/reggie/runtime/ReggieMatcher.java` +- Hook: generated `` allocates a `Scratch` once; `matches()` + resets, never reallocates. +- Reference: `LazyDFA.java` lines around `43-51, 89` (`ThreadLocal`). +- **Confirm thread-model first** — if matchers are not shared across + threads, plain instance fields beat `ThreadLocal`. + +### R6 — Literal fast-path at matcher entry + +**Idea.** For `prefix(.*)` shaped patterns, run a tight `charAt` loop +over the literal prefix before any NFA setup. + +**Pointers.** +- Existing related generators: + `FixedSequenceBytecodeGenerator.java`, + `LinearPatternBytecodeGenerator.java`. +- Analyzer hook: `PatternAnalyzer.java` — verify the + `literal-prefix + free-tail` shape is detected and routed before + falling into the NFA strategies. +- Reference: `LazyDFA.java` `matchLiteral` around lines `347-358`. + +### R7 — Benchmark methodology import + +**Idea.** Adopt glob_perf's **hit-vs-miss corpus split** and +**bounded-cache worst-case reporting** in `reggie-benchmark`. Today +benchmarks report blended ns/op; the split would have caught +lazydfa's 300× hit-path regression at scale, and the analogous shape +of regression could occur in reggie if R1 ships unbounded. + +**Pointers.** +- `reggie-benchmark/` JMH suites. +- Add: explicit `*_hit` / `*_miss` variants for any new lazy-DFA + benchmark. Assert DFA cache size stays bounded. + +## Suggested ordering + +1. **R5** (thread-model audit + ThreadLocal/instance scratch) — + cheapest, unlocks zero-alloc baseline that the rest measures against. +2. **R3** (inline bitset ops audit) — verification, not redesign; + small risk, possibly small win. +3. **R7** (benchmark split) — required infrastructure before R1. +4. **R1 + R2** (lazy DFA + ASCII table) — the headline change. + Highest expected payoff, highest implementation effort, requires + R7 to keep honest. +5. **R6** (literal fast path) — independent; ship whenever convenient. +6. **R4** (SoA automaton) — broad refactor; defer until R1 lands and + indicates the cache-locality win is worth the disruption. + +## Watch-outs (consolidated) + +- **Lazy DFA cache must be bounded.** glob_perf's own data is the + cautionary tale. +- **Confirm matcher thread model** before adopting `AtomicReferenceArray` + or `ThreadLocal` — plain arrays/fields are 2-3× faster if matchers + are single-thread. +- **Hyperscan-style SIMD is not transferable** to one-shot compilation + (13 s build time at 16 k patterns documented). Out of scope. +- glob_perf optimizes the **multi-pattern problem**; the trie/`hasWild`/ + star-collapse code is irrelevant to reggie's single-pattern compile- + to-bytecode model. Do **not** port that machinery. diff --git a/docs/sphinx/specs/2026-05-29-when-the-static-cache-is-shared-by-mul.md b/docs/sphinx/specs/2026-05-29-when-the-static-cache-is-shared-by-mul.md new file mode 100644 index 0000000..ad1e7fb --- /dev/null +++ b/docs/sphinx/specs/2026-05-29-when-the-static-cache-is-shared-by-mul.md @@ -0,0 +1,156 @@ +# Spec: LazyDFA PR #67 — five-item fix batch + +**Date:** 2026-05-29 +**Branch:** feat-lazy-dfa-r1-r2 +**Items:** 3325667007, 3325673306, 3325673350, 3325673394, 3325673423 + +--- + +## Item 3325667007 — Acquire/release semantics for int[] element writes in LazyDFACache + +### What must change and why + +`LazyDFACache.cacheEntry` has two branches. The first branch (table is null) already +uses `TABLES_VH.setRelease` to publish the newly-allocated `int[128]`, establishing +happens-before with the `TABLES_VH.getAcquire` call in the hot loop. The second branch +(table already exists) writes `table[c] = value` as a plain array store. On weakly-ordered +platforms (ARM/RISC-V), a reader thread can see the new DFA state id in `table[c]` before +the per-state data (`nfaStateSets[newId]`, `accepting[newId]`) written by +`computeIfAbsent` on the writer thread has been committed to memory. This can produce +null reads or silent wrong results in `lookupOrCompute` when the reader tries to dereference +`nfaStateSets[state]`. + +### Correct behaviour + +Element writes to an existing ASCII table must use release semantics, and the corresponding +reads in the hot loop (both the runtime `matches()` method and the generated inlined version) +must use acquire semantics. + +Changes: + +1. **`LazyDFACache`**: add a second `static final VarHandle INT_ARRAY_VH` for `int[].class` + next to the existing `TABLES_VH`. In `cacheEntry`'s else-branch replace the plain + `table[c] = value` with `INT_ARRAY_VH.setRelease(table, c, value)`. In the + `matches()` hot loop replace `table[c]` with + `(int) INT_ARRAY_VH.getAcquire(table, c)`. + +2. **`LazyDFABytecodeGenerator.generateMatchesMethod`**: the inlined hot loop currently + reads `table[c]` with a plain `IALOAD`. Replace it with a VarHandle invocation: + `GETSTATIC LazyDFACache.INT_ARRAY_VH`, push `table` (ALOAD 7) and `c` (ILOAD 6), + then `INVOKEVIRTUAL VarHandle.getAcquire "([II)I"`. + +### Constraints + +- `INT_ARRAY_VH` must be `static final` and initialised in the existing `static {}` block + alongside `TABLES_VH`. +- On x86/TSO, `setRelease`/`getAcquire` for `int[]` elements compile to plain + store/load — zero overhead. +- The writer thread's own read `table = asciiTables[state]` inside `cacheEntry` to check + for null does NOT need acquire semantics because it is on the writer thread. +- `INT_ARRAY_VH` must be `package-private` (`static final VarHandle INT_ARRAY_VH`) not + private, so the generated hot-loop bytecode (same package) can `GETSTATIC` it. + +--- + +## Item 3325673306 — FrozenState benchmark corpus must exercise the frozen path + +### What must change and why + +`FrozenState.setup()` uses a 36-character alphabet +(`abcdefghijklmnopqrstuvwxyz0123456789`) to fill the cache. Most generated strings contain +non-`[ab]` characters that hit DEAD on the first step and add only 1-2 DFA states. With +10 000 iterations the cache is unlikely to reach 4096 states, so `frozenPath` measures +normal cached-DEAD-rejection rather than the post-freeze NFA fallback path that the +benchmark claims to measure. + +### Correct behaviour + +Change the warm-up alphabet from 36 chars to `"ab"` only. With the pattern +`(?:a+b+|b+a+){75}` and only `a`/`b` inputs, every character forces a genuine NFA-derived +DFA transition, ensuring state explosion fills the cap quickly. No assertion is required; +the change is sufficient because the reachable DFA state-sets from random `a`/`b` inputs +for this pattern exceed 4096. + +The comment above the warm-up loop should describe the intent clearly. + +--- + +## Item 3325673350 — Processor test for LAZY_DFA end-to-end path + +### What must change and why + +No existing test in `reggie-processor` exercises the LAZY_DFA code path emitted by +`ReggieMatcherBytecodeGenerator`. The delegating `matches()` path (which avoids +package-private access) and the `findMatchFrom` delegation are entirely untested in the +processor module. + +### Correct behaviour + +Add a `testLazyDfaStrategy` test to +`reggie-processor/.../processor/ReggieMatcherBytecodeGeneratorTest.java`. + +Requirements: +- Compile the pattern `(?:a+b+|b+a+){75}` via the existing `compile()` helper. This + pattern is known to route to `LAZY_DFA`. +- Assert `matches()` accepts `"ab".repeat(75)` and rejects `"ab".repeat(74) + "b"`. +- Assert `find()` returns `true` for `"xx" + "ab".repeat(75) + "yy"` and `false` for `"xx"`. +- The test must use only public `ReggieMatcher` API via reflection (the same pattern as + every other test in that file). + +--- + +## Item 3325673394 — LazyDFABytecodeGeneratorTest missing coverage for match/matchBounded/findMatchFrom + +### What must change and why + +`LazyDFABytecodeGeneratorTest` only calls `matches()`. The methods `match()`, +`matchBounded()`, and `findMatchFrom()` generated by `LazyDFABytecodeGenerator` are not +exercised. + +### Correct behaviour + +Add three tests to `LazyDFABytecodeGeneratorTest` (runtime module): + +1. `testMatchMethod`: call `match("ab".repeat(75))`, assert result is non-null, + `result.start(0) == 0`, `result.end(0) == 150`. +2. `testMatchBoundedMethod`: call `matchBounded("xxab".repeat(38) + "xx", 2, 78)` — + the substring `[2,78)` is `"ab".repeat(38)` which does NOT match (requires 75 groups). + Use a correct bounded input: `"xx" + "ab".repeat(75)`, start=2, end=152. Assert + non-null and offsets. +3. `testFindMatchFromMethod`: call `findMatchFrom("xx" + "ab".repeat(75) + "yy", 0)`, + assert non-null result, `result.start(0) == 2`, `result.end(0) == 152`. + +Use `RuntimeCompiler.compile(LARGE_NFA_PATTERN)` to get the matcher, then reflectively +invoke the methods. + +For `match` and `matchBounded` and `findMatchFrom`, use the `MatchResult` interface type +and call `start(int)` / `end(int)` via reflection. + +--- + +## Item 3325673423 — Spec doc: fix incorrect "c & 0x7F" description + +### What must change and why + +`docs/superpowers/specs/2026-05-28-lazy-dfa-design.md` line 18 says the ASCII transition +table is "indexed by `c & 0x7F`". The actual code uses a `c < 128` guard; characters +with `c >= 128` bypass the table entirely and fall through to the NFA step. Masking +non-ASCII chars with `0x7F` would alias them to unrelated ASCII transitions, which is +wrong. + +### Correct behaviour + +Replace every occurrence of "indexed by `c & 0x7F`" in the spec with accurate language: +the table covers ASCII characters (`c < 128`) only; non-ASCII characters (`c >= 128`) +bypass the table and fall through to the NFA step. + +--- + +## Cross-cutting constraints + +- All changes are in these modules: `reggie-runtime`, `reggie-codegen`, `reggie-benchmark`, + `reggie-processor`, and one doc file. +- No new external dependencies. +- Run `./gradlew spotlessApply` before committing. +- Build: `./gradlew :reggie-runtime:compileJava :reggie-codegen:compileJava --no-daemon` +- Test: `./gradlew :reggie-runtime:test :reggie-codegen:test --no-daemon` diff --git a/docs/superpowers/specs/2026-05-28-lazy-dfa-design.md b/docs/superpowers/specs/2026-05-28-lazy-dfa-design.md new file mode 100644 index 0000000..d8a69a6 --- /dev/null +++ b/docs/superpowers/specs/2026-05-28-lazy-dfa-design.md @@ -0,0 +1,309 @@ +# Lazy DFA (R1 + R2) — Design Spec + +**Date:** 2026-05-28 +**Branch:** fix/anchor-semantics (spike off this; implementation goes on a new branch) +**Source:** `doc/plans/glob-perf-nfa-improvements.md` recommendations R1 and R2 + +--- + +## Problem + +`OPTIMIZED_NFA` patterns recompute `closure(stateSet, c)` for every input character. +For patterns with ≥300 NFA states where eager subset construction is too expensive, +matching cost scales with NFA state count on every character, every call. + +R1 + R2 add a lazily-materialized DFA cache over the NFA execution: + +- **R1** — Intern NFA state-sets to DFA state IDs on first encounter; cap at 4 096 states; freeze and fall back to plain NFA stepping when full. +- **R2** — Each cached DFA state gets an `int[128]` ASCII transition table covering ASCII characters only (`c < 128`); non-ASCII characters (`c ≥ 128`) bypass the table and fall through to the NFA step. Warm-path cost is one array read per character. + +--- + +## Scope + +Patterns that qualify for `LAZY_DFA` strategy (`PatternAnalyzer.isLazyDFAEligible`): + +- NFA state count ≥ 300 (only patterns that hit the `OPTIMIZED_NFA` fallback are eligible) +- No capturing groups (`nfa.getGroupCount() == 0` / `CapturingGroupDetector` AST check) +- No anchors (no NFA state has a non-null `AnchorType`, covering `^`, `$`, `\A`, `\Z`, `\z`, `\b`, multiline variants) +- No lookaround assertions — these already route to `OPTIMIZED_NFA_WITH_LOOKAROUND`, not `OPTIMIZED_NFA` +- No backreferences — these route to `OPTIMIZED_NFA_WITH_BACKREFS` + +All other patterns remain on their current strategy unchanged. + +--- + +## Architecture + +``` +compile-time runtime +────────────────────────────────── ─────────────────────────────────────── +PatternAnalyzer LazyDFACache (reggie-runtime) + NFA≥300, no lookaround/backref → ├─ state-set interning map + routes to LAZY_DFA strategy │ ConcurrentHashMap + ├─ per-DFA-state ASCII tables +LazyDFABytecodeGenerator │ Object[] asciiTables + emits: │ asciiTables[id] = int[128] or null + ① static final LazyDFACache CACHE ├─ cap = 4096; volatile boolean frozen + ② int[] nfaStep(int[] states, int c) └─ int[][] nfaStateSets (backing NFA state-sets) + ③ boolean matches(String input) + → CACHE.matches(input, this::nfaStep, ACCEPT_STATE_IDS) +``` + +The generated class owns NFA mechanics only (steps and accept-set checking). +`LazyDFACache` owns caching policy only (interning, cap, ASCII tables). +Neither layer knows the other's internals beyond the `NfaStep` functional interface. + +Note on state-set representation: ≥300-state patterns use `SparseSet`/`int[]` internally +(not a single `long`), so the interface uses `int[]` snapshots (sorted arrays of active NFA +state IDs) rather than bitset longs. The `nfaStep` method allocates a new `int[]` per call, +but this only occurs on cache misses — the warm path never calls it. + +--- + +## Components + +### `LazyDFACache` — new class, `reggie-runtime` + +``` +Functional interface (reggie-runtime) + @FunctionalInterface interface NfaStep { + int[] apply(int[] currentStates, int c); + } + +Fields + ConcurrentHashMap stateIndex + Object[] asciiTables // asciiTables[dfaStateId] = int[128]; null until first write + int[][] nfaStateSets // nfaStateSets[dfaStateId] = sorted int[] of active NFA state IDs + boolean[] accepting // accepting[dfaStateId] = true iff this DFA state is a final state + AtomicInteger nextId // next DFA state ID to assign + volatile boolean frozen // set true when nextId >= CAP + int[] acceptStateIds // NFA state IDs that are accept states (from generated class) + +Constants + int CAP = 4096 + int UNCACHED = -1 + int DEAD = -2 + int FALLBACK = -3 // frozen and no cached state: switch to NFA mode for remainder + +Key method + boolean matches(String input, NfaStep nfaStep) + dfaState = 0 + for pos = 0 to input.length()-1: + char c = input.charAt(pos) + asciiTable = asciiTables[dfaState] + int next = (asciiTable != null && c < 128) ? asciiTable[c] : UNCACHED + if next == UNCACHED: + next = lookupOrCompute(dfaState, c, nfaStep) + if next == DEAD: return false + if next == FALLBACK: return nfaFallbackMatch(input, pos, nfaStateSets[dfaState], nfaStep) + dfaState = next + return accepting[dfaState] + + private int lookupOrCompute(int state, int c, NfaStep nfaStep) + int[] nextNFASet = nfaStep.apply(nfaStateSets[state], c) + if nextNFASet.length == 0: return DEAD + key = new StateSetKey(nextNFASet) + Integer id = stateIndex.get(key) + if id == null && !frozen: + id = stateIndex.computeIfAbsent(key, k -> nextId.getAndIncrement()) + if id >= CAP: frozen = true + else: + nfaStateSets[id] = nextNFASet + accepting[id] = containsAny(nextNFASet, acceptStateIds) + if id == null: return FALLBACK + // populate ASCII table entry (idempotent — same key always maps to same id) + int[] table = (int[]) asciiTables[state] + if table == null: + table = new int[128] + Arrays.fill(table, UNCACHED) + asciiTables[state] = table // plain write; idempotent race is safe + if c < 128: table[c] = id + return id + + private boolean nfaFallbackMatch(String input, int fromPos, int[] nfaStateSet, NfaStep nfaStep) + // Called once the cache is frozen and a new transition was encountered. + // Runs pure NFA stepping (allocating per step) for the remainder of the input. + int[] states = nfaStep.apply(nfaStateSet, input.charAt(fromPos)) + for pos = fromPos + 1 to input.length()-1: + if states.length == 0: return false + states = nfaStep.apply(states, input.charAt(pos)) + return containsAny(states, acceptStateIds) +``` + +`StateSetKey` wraps a sorted `int[]` of active NFA state IDs and implements `equals`/`hashCode` +based on array contents. An alternative encoding (glob_perf's String-packing trick: each state +ID packed as a `char` in a `String`) may be chosen during implementation for a faster hash. + +### `LazyDFABytecodeGenerator` — new class, `reggie-codegen` + +Parallels `NFABytecodeGenerator`. Emits three artifacts into the generated class: + +1. `private static final LazyDFACache CACHE` — one instance per compiled pattern class, + initialised with: the ε-closure of the NFA start state (as a sorted int[]), and + the accept state IDs (as a sorted int[]). + +2. `int[] nfaStep(int[] states, int c)` — given a sorted int[] of active NFA state IDs and + a character, returns a new sorted int[] of the next NFA state IDs (including ε-closure). + Same transition logic the NFA generator currently inlines into `matches()`, extracted as + a package-private method callable by `LazyDFACache` on cache misses. + Allocates on each call; this is acceptable because it is only called on cache misses. + +3. `public boolean matches(String input)` — delegates: + ```java + return CACHE.matches(input, this::nfaStep); + ``` + +### `PatternAnalyzer` — addendum to routing logic + +After the existing DFA/NFA routing decision: + +```java +if (strategy == Strategy.OPTIMIZED_NFA + && nfa.getStateCount() >= 300 + && !nfa.hasLookaround() + && !nfa.hasBackreferences()) { + strategy = Strategy.LAZY_DFA; +} +``` + +This check is placed **after** all existing DFA threshold and anchor-dilution checks, so +those patterns are unaffected. + +--- + +## Data Flow + +### Cold path (first encounter of a state × char pair) + +``` +matches("abc...") + dfaState = 0 // start state; nfaStateSets[0] = ε-closure of NFA start (int[]) + c = 'a' + asciiTables[0] == null // not yet allocated + → lookupOrCompute(0, 'a', nfaStep) + nextNFASet = nfaStep.apply(nfaStateSets[0], 'a') // int[] allocation on miss + stateIndex.computeIfAbsent(key, ...) → id = 1 + nfaStateSets[1] = nextNFASet + accepting[1] = containsAny(nextNFASet, acceptStateIds) + asciiTables[0] = new int[128]; asciiTables[0]['a'] = 1 + dfaState = 1 + c = 'b' + asciiTables[1] == null // new state, no table yet + → lookupOrCompute(1, 'b', nfaStep) → id = 2 + ... +``` + +### Warm path (all transitions cached) + +``` + c = 'a' + asciiTables[dfaState]['a'] → 3 // single int[128] read, zero allocation + dfaState = 3 +``` + +### Frozen path (cap reached, FALLBACK sentinel triggers NFA mode) + +``` + dfaState = 2048, c = '?' + asciiTables[2048] == null, frozen == true + next = lookupOrCompute(2048, '?', nfaStep) + nextNFASet = nfaStep.apply(nfaStateSets[2048], '?') + stateIndex.get(key) → null, frozen → return FALLBACK + next == FALLBACK + → nfaFallbackMatch(input, pos, nfaStateSets[2048], nfaStep) + runs pure NFA stepping for remainder of input (correct but allocating) +``` + +--- + +## Thread-Safety + +- `stateIndex`: `ConcurrentHashMap` — safe concurrent interning via `computeIfAbsent`. +- **ASCII table publication** (`asciiTables` slot): the initial `int[128]` array is published + with `TABLES_VH.setRelease(asciiTables, state, t)` (array-element VarHandle with release + semantics); readers use `TABLES_VH.getAcquire(asciiTables, dfaState)`. This establishes a + happens-before on weakly-ordered platforms (ARM/RISC-V) and compiles to plain load/store on + x86/TSO. +- **ASCII table entry updates** (subsequent writes to an already-published slot): written with + `INT_ARRAY_VH.setRelease(table, c, value)` and read with `INT_ARRAY_VH.getAcquire(table, c)`. + This pairs with the `nfaStateSets[newId]`/`accepting[newId]` initialization done inside + `computeIfAbsent` on the writer thread, ensuring those writes are visible to any reader that + subsequently observes the new DFA state id via `getAcquire`. Idempotent: same key always + maps to the same id, so racing writes produce the same value. +- `frozen`: `volatile boolean` — guarantees all threads see the freeze once set. +- `nextId`: `AtomicInteger` — safe increments under concurrent interning. + +`ReggieMatcher` instances are single-threaded per instance (confirmed from code inspection); +`CACHE` is shared but the above invariants make it safe without instance-level locking. + +--- + +## TDD Test Plan + +### Layer 1 — `LazyDFACache` (write tests first) + +File: `reggie-runtime/src/test/java/.../LazyDFACacheTest.java` + +| Test | What it verifies | +|------|-----------------| +| `testCacheMissInternsNewState` | First call on (startState, char) allocates DFA state ID 1 | +| `testCacheHitUsesAsciiTable` | Second call to same (state, char) reads int[128], no map lookup | +| `testDeadStateEarlyExit` | Non-matching NFA step (returns 0L) maps to DEAD, match returns false | +| `testFreezeAtCap` | After 4096 interned states, `frozen=true`, `FALLBACK` returned for new transitions | +| `testFallbackMatchCorrect` | After freeze, `nfaFallbackMatch` produces same accept/reject decision as unfrozen NFA | +| `testConcurrentInterning` | Two threads racing on same cache key get consistent, deduplicated IDs | +| `testAcceptStateRecognition` | State whose NFA set intersects acceptMask is recognised as accepting | +| `testNonAsciiCharFallsBackToNfaStep` | `c ≥ 128` always takes the slow path (no int[128] read out of bounds) | + +### Layer 2 — `LazyDFABytecodeGenerator` (write tests first) + +File: `reggie-codegen/src/test/java/.../LazyDFABytecodeGeneratorTest.java` + +| Test | What it verifies | +|------|-----------------| +| `testGeneratedClassMatchesNFAForSameInputs` | LAZY_DFA and OPTIMIZED_NFA produce identical results on 500 random strings | +| `testNfaStepMethodPresent` | Generated class has `nfaStep(long, int)` via reflection | +| `testCacheIsSharedAcrossInstances` | Two instances of same generated class share the same `CACHE` reference | +| `testCacheIsNotSharedAcrossPatterns` | Two different patterns have different `CACHE` objects | + +### Layer 3 — `PatternAnalyzer` routing (write tests first) + +File: `reggie-codegen/src/test/java/.../PatternAnalyzerLazyDFATest.java` + +| Test | What it verifies | +|------|-----------------| +| `testRouteToLazyDFAWhenNFALarge` | Pattern with ≥300 NFA states, no lookaround → `LAZY_DFA` | +| `testDoNotRouteWithLookahead` | Same pattern + `(?=...)` → `OPTIMIZED_NFA` | +| `testDoNotRouteWithBackref` | Same pattern + `\1` → `OPTIMIZED_NFA` | +| `testDoNotRouteWhenNFASmall` | 250-state NFA → `OPTIMIZED_NFA` | + +### Layer 4 — Benchmarks (JMH, `reggie-benchmark`) + +New class: `LazyDFABenchmark.java` + +| Benchmark | Input corpus | Measures | +|-----------|-------------|----------| +| `hitPath` | Repeated short inputs from a fixed set (all DFA transitions cached after first pass) | Warm-path throughput: `int[128]` read per char | +| `missPath` | Fresh random strings each iteration (DFA cache always cold) | Cold-path cost: `nfaStep` + interning overhead | +| `frozenPath` | Fill cache to 4096, then repeat large-corpus matching | Freeze+fallback performance | + +All three variants report explicit `_hit` / `_miss` / `_frozen` suffixes (per R7 methodology). +Baseline: same patterns run through `NFAFallbackBenchmark` for apples-to-apples comparison. + +--- + +## Watch-outs + +- **Cap must be enforced.** glob_perf documents a 300× hit-path regression when the DFA + cache is unbounded at 65k patterns. The 4096 cap is non-negotiable. +- **NFA state count for ≥300-state patterns.** These patterns use `SparseSet` / `int[]` + state-set representation, not a single `long`. `StateSetKey` must handle both. +- **`nfaStep` returns a new `int[]` on every call.** This allocation is acceptable because + it only occurs on cache misses (cold path). Once the cache is warm, `nfaStep` is never + called. The `FALLBACK` path (post-freeze) is similarly allocation-heavy but is a + degraded-mode last resort. +- **`this::nfaStep` lambda allocation on each `matches()` call.** If profiling shows this + matters, store the `NfaStep` instance as a `final` field on the matcher instance. +- **Do not port trie/multi-pattern machinery from glob_perf.** reggie is single-pattern + compile-to-bytecode; only the cache/table micro-optimizations apply. diff --git a/reggie-benchmark/src/main/java/com/datadoghq/reggie/benchmark/LazyDFABenchmark.java b/reggie-benchmark/src/main/java/com/datadoghq/reggie/benchmark/LazyDFABenchmark.java new file mode 100644 index 0000000..8c210ac --- /dev/null +++ b/reggie-benchmark/src/main/java/com/datadoghq/reggie/benchmark/LazyDFABenchmark.java @@ -0,0 +1,186 @@ +/* + * Copyright 2026-Present Datadog, Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.datadoghq.reggie.benchmark; + +import com.datadoghq.reggie.runtime.LazyDFACache; +import com.datadoghq.reggie.runtime.ReggieMatcher; +import com.datadoghq.reggie.runtime.RuntimeCompiler; +import java.lang.reflect.Field; +import java.lang.reflect.Method; +import java.util.Arrays; +import java.util.Random; +import java.util.concurrent.TimeUnit; +import java.util.regex.Pattern; +import org.openjdk.jmh.annotations.*; + +/** + * Hit/miss/frozen benchmarks for the Lazy DFA cache (R1+R2). Per R7 methodology: explicit _hit / + * _miss / _frozen variants. Baseline: compare against NFAFallbackBenchmark for the same patterns. + */ +@BenchmarkMode(Mode.Throughput) +@OutputTimeUnit(TimeUnit.MILLISECONDS) +@State(Scope.Thread) +@Warmup(iterations = 3, time = 1) +@Measurement(iterations = 5, time = 1) +@Fork(1) +public class LazyDFABenchmark { + + // ~685 NFA states, no groups/anchors — DFA state explosion via interleaved a+/b+ alternation + // causes StateExplosionException → OPTIMIZED_NFA → LAZY_DFA. + // Note: deterministic patterns like (?:[a-z][0-9]){200} now route to DFA_TABLE instead. + private static final String PATTERN = "(?:a+b+|b+a+){75}"; + // Positive match: 75 repetitions of "ab" — each "ab" satisfies one (a+b+) group + private static final String MATCH_INPUT = "ab".repeat(75); + + private ReggieMatcher lazyMatcher; + // JDK baseline — same pattern, same inputs, java.util.regex NFA + private Pattern jdkPattern; + private String[] missInputs; + private int missIndex; + // Hard-miss inputs: all-[ab] strings that fail late in the pattern, + // forcing real NFA/DFA traversal rather than immediate first-char rejection. + private String[] hardMissInputs; + private int hardMissIndex; + + @Setup(Level.Trial) + public void setup() { + RuntimeCompiler.clearCache(); + lazyMatcher = RuntimeCompiler.compile(PATTERN); + jdkPattern = Pattern.compile(PATTERN); + // Warm up the DFA cache + for (int i = 0; i < 50; i++) lazyMatcher.matches(MATCH_INPUT); + // Build diverse miss inputs (random chars — tests early-exit behavior) + Random rng = new Random(12345); + missInputs = new String[1000]; + String chars = "abcdefghijklmnopqrstuvwxyz0123456789!@#$"; + for (int i = 0; i < missInputs.length; i++) { + int len = 300 + rng.nextInt(200); + StringBuilder sb = new StringBuilder(len); + for (int j = 0; j < len; j++) sb.append(chars.charAt(rng.nextInt(chars.length()))); + missInputs[i] = sb.toString(); + } + // Build hard-miss inputs: all [ab] chars, fail after 60-74 complete groups. + // Forces real NFA/DFA traversal before rejection — no early-exit on first char. + hardMissInputs = new String[1000]; + for (int i = 0; i < hardMissInputs.length; i++) { + // 60-74 complete (a+b+) groups, then 1-5 trailing 'a's without a closing 'b'. + int completeGroups = 60 + (i % 15); + int trailingAs = 1 + (i % 5); + hardMissInputs[i] = "ab".repeat(completeGroups) + "a".repeat(trailingAs); + } + } + + /** Warm path: all DFA transitions cached → single int[128] read per char. */ + @Benchmark + public boolean hitPath() { + return lazyMatcher.matches(MATCH_INPUT); + } + + /** + * Steady-state early-rejection throughput: diverse random inputs that mostly contain characters + * outside the pattern alphabet ([ab]). After the first pass over the 1,000-input pool the DEAD + * transitions for those characters are cached in the ASCII table, so subsequent calls measure + * cached-DEAD lookups rather than cold NFA-step+interning overhead. Use {@code hardMissPath} for + * a fair late-failing comparison where both engines traverse the automaton before rejecting. + */ + @Benchmark + public boolean missPath() { + return lazyMatcher.matches(missInputs[(missIndex++ & 0x7FFF_FFFF) % missInputs.length]); + } + + /** JDK baseline — same pattern, fixed matching input, java.util.regex NFA. */ + @Benchmark + public boolean jdkHitBaseline() { + return jdkPattern.matcher(MATCH_INPUT).matches(); + } + + /** JDK baseline — same diverse miss inputs as missPath. */ + @Benchmark + public boolean jdkMissBaseline() { + return jdkPattern + .matcher(missInputs[(missIndex++ & 0x7FFF_FFFF) % missInputs.length]) + .matches(); + } + + /** + * Hard-miss path: all-[ab] inputs that fail after 60-74 complete groups. Forces real NFA + * traversal before rejection — a fair comparison against jdkHardMissBaseline. + */ + @Benchmark + public boolean hardMissPath() { + return lazyMatcher.matches( + hardMissInputs[(hardMissIndex++ & 0x7FFF_FFFF) % hardMissInputs.length]); + } + + /** JDK hard-miss baseline — same late-failing all-[ab] inputs. */ + @Benchmark + public boolean jdkHardMissBaseline() { + return jdkPattern + .matcher(hardMissInputs[(hardMissIndex++ & 0x7FFF_FFFF) % hardMissInputs.length]) + .matches(); + } + + /** Frozen path: cache at cap, all transitions use NFA fallback. */ + @State(Scope.Thread) + public static class FrozenState { + ReggieMatcher matcher; + String[] frozenInputs; + int idx; + + @Setup(Level.Trial) + public void setup() { + RuntimeCompiler.clearCache(); + matcher = RuntimeCompiler.compile(PATTERN); + Random rng = new Random(99999); + // Use only 'a'/'b' so every warm-up step forces a real NFA-derived DFA transition. + // A 36-char alphabet hits DEAD after one step and adds too few states to fill the cap, + // causing frozenPath to measure normal cached-DEAD-rejection rather than NFA fallback. + String alpha = "ab"; + // Fill the cache to trigger freeze + for (int i = 0; i < 10_000; i++) { + StringBuilder sb = new StringBuilder(400); + for (int j = 0; j < 400; j++) sb.append(alpha.charAt(rng.nextInt(alpha.length()))); + matcher.matches(sb.toString()); + } + // Assert the cache is actually frozen before measuring, so frozenPath truly exercises + // the NFA-fallback path and not the normal DFA cache. isFrozen() is package-private, + // so we access it via reflection. + try { + Field cacheField = matcher.getClass().getDeclaredField("CACHE"); + cacheField.setAccessible(true); + LazyDFACache cache = (LazyDFACache) cacheField.get(null); + Method isFrozen = LazyDFACache.class.getDeclaredMethod("isFrozen"); + isFrozen.setAccessible(true); + if (!(Boolean) isFrozen.invoke(cache)) { + throw new IllegalStateException( + "LazyDFACache not frozen after warm-up — frozenPath would measure the wrong path." + + " Increase warm-up iterations or check the pattern's DFA state count."); + } + } catch (ReflectiveOperationException e) { + throw new IllegalStateException("Could not verify frozen state", e); + } + // Fixed always-matching input: measures full 400-char NFA traversal after freeze, + // not early rejection on random non-matching strings. + frozenInputs = new String[500]; + Arrays.fill(frozenInputs, MATCH_INPUT); + } + } + + @Benchmark + public boolean frozenPath(FrozenState s) { + return s.matcher.matches(s.frozenInputs[s.idx++ % s.frozenInputs.length]); + } +} diff --git a/reggie-codegen/src/main/java/com/datadoghq/reggie/codegen/analysis/PatternAnalyzer.java b/reggie-codegen/src/main/java/com/datadoghq/reggie/codegen/analysis/PatternAnalyzer.java index efa5753..e7239fe 100644 --- a/reggie-codegen/src/main/java/com/datadoghq/reggie/codegen/analysis/PatternAnalyzer.java +++ b/reggie-codegen/src/main/java/com/datadoghq/reggie/codegen/analysis/PatternAnalyzer.java @@ -801,6 +801,7 @@ && dfaHasAcceptingStateWithTransitions(dfa)) { } // Try standard DFA construction (lookaround already handled above) + MatchingStrategyResult result; try { SubsetConstructor constructor = new SubsetConstructor(); DFA dfa = constructor.buildDFA(nfa); @@ -839,17 +840,43 @@ && dfaHasAcceptingStateWithTransitions(dfa)) { return new MatchingStrategyResult( MatchingStrategy.DFA_TABLE, dfa, null, false, requiredLiterals); } else { - // Large DFA state spaces can still be too expensive as tables. Fall back to NFA simulation - // when the compressed table would exceed the configured memory budget or the DFA uses - // features not yet supported by the table backend. - return new MatchingStrategyResult( - MatchingStrategy.OPTIMIZED_NFA, null, null, false, requiredLiterals); + // Large DFA not eligible for table backend; fall through to LAZY_DFA promotion check. + result = + new MatchingStrategyResult( + MatchingStrategy.OPTIMIZED_NFA, null, null, false, requiredLiterals); } } catch (StateExplosionException e) { - // DFA too large, use optimized NFA - return new MatchingStrategyResult( - MatchingStrategy.OPTIMIZED_NFA, null, null, false, requiredLiterals); - } + // DFA too large to construct; fall through to LAZY_DFA promotion check. + result = + new MatchingStrategyResult( + MatchingStrategy.OPTIMIZED_NFA, null, null, false, requiredLiterals); + } + // Promote large anchor-free group-free NFA patterns to the lazy DFA strategy. + if (result.strategy == MatchingStrategy.OPTIMIZED_NFA + && nfa != null + && isLazyDFAEligible(nfa, ast)) { + result = + new MatchingStrategyResult( + MatchingStrategy.LAZY_DFA, + result.dfa, + result.patternInfo, + result.useTaggedDFA, + result.requiredLiterals, + result.lookaheadGreedyInfo, + result.usePosixLastMatch); + } + return result; + } + + private boolean isLazyDFAEligible(NFA nfa, RegexNode ast) { + return nfa.getStates().size() >= 300 + && !hasCapturingGroups(ast) + && nfa.getStates().stream().noneMatch(s -> s.anchor != null); + } + + private boolean hasCapturingGroups(RegexNode node) { + CapturingGroupDetector detector = new CapturingGroupDetector(); + return node.accept(detector); } private boolean isDFATableEligible(DFA dfa) { @@ -1937,6 +1964,7 @@ public enum MatchingStrategy { DFA_SWITCH_WITH_GROUPS, // 20-300 states - switch with inline group tracking DFA_TABLE, // >300 states - table-driven OPTIMIZED_NFA, // DFA state explosion - optimized NFA + LAZY_DFA, // Large anchor-free group-free NFA - on-the-fly DFA construction OPTIMIZED_NFA_WITH_BACKREFS, // Has backreferences OPTIMIZED_NFA_WITH_LOOKAROUND, // Has variable-width lookaround assertions (NFA for all) HYBRID_DFA_LOOKAHEAD, // Hybrid: DFA for lookahead sub-patterns, NFA for main pattern @@ -3573,6 +3601,76 @@ private BackreferencePatternInfo detectGreedyAnyBackrefPattern(List c totalGroupCount); } + /** Visitor to detect capturing groups in AST. */ + private static class CapturingGroupDetector implements RegexVisitor { + @Override + public Boolean visitLiteral(LiteralNode node) { + return false; + } + + @Override + public Boolean visitCharClass(CharClassNode node) { + return false; + } + + @Override + public Boolean visitConcat(ConcatNode node) { + return node.children.stream().anyMatch(child -> child.accept(this)); + } + + @Override + public Boolean visitAlternation(AlternationNode node) { + return node.alternatives.stream().anyMatch(alt -> alt.accept(this)); + } + + @Override + public Boolean visitQuantifier(QuantifierNode node) { + return node.child.accept(this); + } + + @Override + public Boolean visitGroup(GroupNode node) { + return node.capturing || node.child.accept(this); + } + + @Override + public Boolean visitAnchor(AnchorNode node) { + return false; + } + + @Override + public Boolean visitBackreference(BackreferenceNode node) { + return false; + } + + @Override + public Boolean visitAssertion(AssertionNode node) { + return node.subPattern != null && node.subPattern.accept(this); + } + + @Override + public Boolean visitSubroutine(SubroutineNode node) { + return false; + } + + @Override + public Boolean visitConditional(ConditionalNode node) { + boolean hasThen = node.thenBranch.accept(this); + boolean hasElse = node.elseBranch != null && node.elseBranch.accept(this); + return hasThen || hasElse; + } + + @Override + public Boolean visitBranchReset(BranchResetNode node) { + for (RegexNode alt : node.alternatives) { + if (alt.accept(this)) { + return true; + } + } + return false; + } + } + /** Visitor to detect backreferences in AST. */ private static class BackrefDetector implements RegexVisitor { @Override diff --git a/reggie-codegen/src/main/java/com/datadoghq/reggie/codegen/codegen/LazyDFABytecodeGenerator.java b/reggie-codegen/src/main/java/com/datadoghq/reggie/codegen/codegen/LazyDFABytecodeGenerator.java new file mode 100644 index 0000000..9d3f8b5 --- /dev/null +++ b/reggie-codegen/src/main/java/com/datadoghq/reggie/codegen/codegen/LazyDFABytecodeGenerator.java @@ -0,0 +1,867 @@ +/* + * Copyright 2026-Present Datadog, Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.datadoghq.reggie.codegen.codegen; + +import static com.datadoghq.reggie.codegen.codegen.BytecodeUtil.pushInt; +import static org.objectweb.asm.Opcodes.*; + +import com.datadoghq.reggie.codegen.automaton.CharSet; +import com.datadoghq.reggie.codegen.automaton.NFA; +import java.util.ArrayDeque; +import java.util.ArrayList; +import java.util.Deque; +import java.util.HashSet; +import java.util.List; +import java.util.Set; +import org.objectweb.asm.ClassWriter; +import org.objectweb.asm.Label; +import org.objectweb.asm.MethodVisitor; + +/** Emits static NFA data arrays, {@code nfaStep}, and a lazy-DFA {@code matches} method. */ +public class LazyDFABytecodeGenerator { + + private static final String SPARSE_SET = "com/datadoghq/reggie/runtime/SparseSet"; + private static final String LAZY_CACHE = "com/datadoghq/reggie/runtime/LazyDFACache"; + private static final String NFA_STEP = "com/datadoghq/reggie/runtime/NfaStep"; + private static final String ARRAYS = "java/util/Arrays"; + + // Mirrors of LazyDFACache.UNCACHED/DEAD/FALLBACK. Cannot import directly because + // reggie-codegen does not depend on reggie-runtime at compile time. Update both if changed. + private static final int UNCACHED = -1; + private static final int DEAD = -2; + private static final int FALLBACK = -3; + + /** + * Mirrors {@code LazyDFACache.NEEDS_INT_ARRAY_ACQUIRE}. On x86/TSO the hot loop emits a plain + * {@code IALOAD} for the ASCII table element read; on weakly-ordered platforms it emits {@code + * INT_ARRAY_VH.getAcquire} to pair with the release write in {@code cacheEntry}. Computed once at + * class-load time so the generated bytecode is fixed per JVM invocation. + */ + private static final boolean NEEDS_INT_ARRAY_ACQUIRE; + + static { + String arch = System.getProperty("os.arch", ""); + NEEDS_INT_ARRAY_ACQUIRE = + !arch.equals("x86") + && !arch.equals("x86_64") + && !arch.equals("amd64") + && !arch.equals("i386"); + } + + private final NFA nfa; + private final int stateCount; + private final int[][] transitions; // transitions[id] = flat [min, max, target, ...] + private final int[][] epsClosure; // epsClosure[id] = sorted int[] of reachable IDs + private final int[] startSet; // ε-closure of start state + private final int[] acceptIds; // sorted accept state IDs + + public LazyDFABytecodeGenerator(NFA nfa) { + this.nfa = nfa; + this.stateCount = nfa.getStates().size(); + this.transitions = buildTransitions(nfa); + this.epsClosure = buildEpsClosure(nfa); + this.startSet = epsClosure[nfa.getStartState().id]; + this.acceptIds = nfa.getAcceptStates().stream().mapToInt(s -> s.id).sorted().toArray(); + } + + /** Declare + initialize all static fields and emit {@code }. */ + public void generateStaticFields(ClassWriter cw, String className) { + cw.visitField(ACC_PRIVATE | ACC_STATIC | ACC_FINAL, "NFA_STATE_COUNT", "I", null, stateCount) + .visitEnd(); + cw.visitField(ACC_PRIVATE | ACC_STATIC | ACC_FINAL, "NFA_TRANSITIONS", "[[I", null, null) + .visitEnd(); + cw.visitField(ACC_PRIVATE | ACC_STATIC | ACC_FINAL, "NFA_EPS_CLOSURES", "[[I", null, null) + .visitEnd(); + cw.visitField(ACC_PRIVATE | ACC_STATIC | ACC_FINAL, "NFA_START_SET", "[I", null, null) + .visitEnd(); + cw.visitField(ACC_PRIVATE | ACC_STATIC | ACC_FINAL, "NFA_ACCEPT_IDS", "[I", null, null) + .visitEnd(); + cw.visitField(ACC_PRIVATE | ACC_STATIC | ACC_FINAL, "CACHE", "L" + LAZY_CACHE + ";", null, null) + .visitEnd(); + + MethodVisitor clinit = cw.visitMethod(ACC_STATIC, "", "()V", null, null); + clinit.visitCode(); + + // For large NFAs, split array init into helper methods to avoid 64 KB method limit. + emitSplitInt2DArrayInit(cw, clinit, className, "NFA_TRANSITIONS", transitions, "[[I"); + emitSplitInt2DArrayInit(cw, clinit, className, "NFA_EPS_CLOSURES", epsClosure, "[[I"); + emitInt1DArrayInit(clinit, className, "NFA_START_SET", startSet, "[I"); + emitInt1DArrayInit(clinit, className, "NFA_ACCEPT_IDS", acceptIds, "[I"); + + // CACHE = new LazyDFACache(NFA_START_SET, NFA_ACCEPT_IDS) + clinit.visitTypeInsn(NEW, LAZY_CACHE); + clinit.visitInsn(DUP); + clinit.visitFieldInsn(GETSTATIC, className, "NFA_START_SET", "[I"); + clinit.visitFieldInsn(GETSTATIC, className, "NFA_ACCEPT_IDS", "[I"); + clinit.visitMethodInsn(INVOKESPECIAL, LAZY_CACHE, "", "([I[I)V", false); + clinit.visitFieldInsn(PUTSTATIC, className, "CACHE", "L" + LAZY_CACHE + ";"); + + clinit.visitInsn(RETURN); + clinit.visitMaxs(0, 0); + clinit.visitEnd(); + } + + /** Emits the {@code nfaStep(int[], int): int[]} instance method. */ + public void generateNfaStepMethod(ClassWriter cw, String className) { + // slot 0=this, 1=states[], 2=c, 3=current, 4=next, 5=outer-i/si/sz/copy-i, + // 6=stateId/result, 7=trans/copy-loop-i, 8=j, 9=eps, 10=eps-index + MethodVisitor mv = cw.visitMethod(0, "nfaStep", "([II)[I", null, null); + mv.visitCode(); + + // SparseSet current = new SparseSet(NFA_STATE_COUNT) + mv.visitTypeInsn(NEW, SPARSE_SET); + mv.visitInsn(DUP); + mv.visitFieldInsn(GETSTATIC, className, "NFA_STATE_COUNT", "I"); + mv.visitMethodInsn(INVOKESPECIAL, SPARSE_SET, "", "(I)V", false); + mv.visitVarInsn(ASTORE, 3); + + // SparseSet next = new SparseSet(NFA_STATE_COUNT) + mv.visitTypeInsn(NEW, SPARSE_SET); + mv.visitInsn(DUP); + mv.visitFieldInsn(GETSTATIC, className, "NFA_STATE_COUNT", "I"); + mv.visitMethodInsn(INVOKESPECIAL, SPARSE_SET, "", "(I)V", false); + mv.visitVarInsn(ASTORE, 4); + + // for (int i = 0; i < states.length; i++) current.add(states[i]) + pushInt(mv, 0); + mv.visitVarInsn(ISTORE, 5); + Label loopPopStart = new Label(); + Label loopPopEnd = new Label(); + mv.visitLabel(loopPopStart); + mv.visitVarInsn(ILOAD, 5); + mv.visitVarInsn(ALOAD, 1); + mv.visitInsn(ARRAYLENGTH); + mv.visitJumpInsn(IF_ICMPGE, loopPopEnd); + mv.visitVarInsn(ALOAD, 3); + mv.visitVarInsn(ALOAD, 1); + mv.visitVarInsn(ILOAD, 5); + mv.visitInsn(IALOAD); + mv.visitMethodInsn(INVOKEVIRTUAL, SPARSE_SET, "add", "(I)V", false); + mv.visitIincInsn(5, 1); + mv.visitJumpInsn(GOTO, loopPopStart); + mv.visitLabel(loopPopEnd); + + // for (int si = 0; si < current.size(); si++) + pushInt(mv, 0); + mv.visitVarInsn(ISTORE, 5); + Label loopSiStart = new Label(); + Label loopSiEnd = new Label(); + mv.visitLabel(loopSiStart); + mv.visitVarInsn(ILOAD, 5); + mv.visitVarInsn(ALOAD, 3); + mv.visitMethodInsn(INVOKEVIRTUAL, SPARSE_SET, "size", "()I", false); + mv.visitJumpInsn(IF_ICMPGE, loopSiEnd); + + // int stateId = current.get(si) + mv.visitVarInsn(ALOAD, 3); + mv.visitVarInsn(ILOAD, 5); + mv.visitMethodInsn(INVOKEVIRTUAL, SPARSE_SET, "get", "(I)I", false); + mv.visitVarInsn(ISTORE, 6); + + // int[] trans = NFA_TRANSITIONS[stateId] + mv.visitFieldInsn(GETSTATIC, className, "NFA_TRANSITIONS", "[[I"); + mv.visitVarInsn(ILOAD, 6); + mv.visitInsn(AALOAD); + mv.visitVarInsn(ASTORE, 7); + + // for (int j = 0; j < trans.length; j += 3) + pushInt(mv, 0); + mv.visitVarInsn(ISTORE, 8); + Label loopJStart = new Label(); + Label loopJEnd = new Label(); + mv.visitLabel(loopJStart); + mv.visitVarInsn(ILOAD, 8); + mv.visitVarInsn(ALOAD, 7); + mv.visitInsn(ARRAYLENGTH); + mv.visitJumpInsn(IF_ICMPGE, loopJEnd); + + // if (c >= trans[j] && c <= trans[j+1]) + Label skipTransition = new Label(); + mv.visitVarInsn(ILOAD, 2); // c + mv.visitVarInsn(ALOAD, 7); // trans + mv.visitVarInsn(ILOAD, 8); // j + mv.visitInsn(IALOAD); // trans[j] + mv.visitJumpInsn(IF_ICMPLT, skipTransition); + mv.visitVarInsn(ILOAD, 2); // c + mv.visitVarInsn(ALOAD, 7); // trans + mv.visitVarInsn(ILOAD, 8); // j + mv.visitInsn(ICONST_1); + mv.visitInsn(IADD); + mv.visitInsn(IALOAD); // trans[j+1] + mv.visitJumpInsn(IF_ICMPGT, skipTransition); + + // int[] eps = NFA_EPS_CLOSURES[trans[j+2]] + mv.visitFieldInsn(GETSTATIC, className, "NFA_EPS_CLOSURES", "[[I"); + mv.visitVarInsn(ALOAD, 7); // trans + mv.visitVarInsn(ILOAD, 8); // j + mv.visitInsn(ICONST_2); + mv.visitInsn(IADD); + mv.visitInsn(IALOAD); // trans[j+2] + mv.visitInsn(AALOAD); + mv.visitVarInsn(ASTORE, 9); + + // for (int ei = 0; ei < eps.length; ei++) next.add(eps[ei]) + pushInt(mv, 0); + mv.visitVarInsn(ISTORE, 10); + Label loopEpsStart = new Label(); + Label loopEpsEnd = new Label(); + mv.visitLabel(loopEpsStart); + mv.visitVarInsn(ILOAD, 10); + mv.visitVarInsn(ALOAD, 9); + mv.visitInsn(ARRAYLENGTH); + mv.visitJumpInsn(IF_ICMPGE, loopEpsEnd); + mv.visitVarInsn(ALOAD, 4); // next + mv.visitVarInsn(ALOAD, 9); // eps + mv.visitVarInsn(ILOAD, 10); + mv.visitInsn(IALOAD); + mv.visitMethodInsn(INVOKEVIRTUAL, SPARSE_SET, "add", "(I)V", false); + mv.visitIincInsn(10, 1); + mv.visitJumpInsn(GOTO, loopEpsStart); + mv.visitLabel(loopEpsEnd); + + mv.visitLabel(skipTransition); + mv.visitIincInsn(8, 3); // j += 3 + mv.visitJumpInsn(GOTO, loopJStart); + mv.visitLabel(loopJEnd); + + mv.visitIincInsn(5, 1); // si++ + mv.visitJumpInsn(GOTO, loopSiStart); + mv.visitLabel(loopSiEnd); + + // int sz = next.size() + mv.visitVarInsn(ALOAD, 4); + mv.visitMethodInsn(INVOKEVIRTUAL, SPARSE_SET, "size", "()I", false); + mv.visitVarInsn(ISTORE, 5); + + // int[] result = new int[sz] + mv.visitVarInsn(ILOAD, 5); + mv.visitIntInsn(NEWARRAY, T_INT); + mv.visitVarInsn(ASTORE, 6); + + // for (int i = 0; i < sz; i++) result[i] = next.get(i) + pushInt(mv, 0); + mv.visitVarInsn(ISTORE, 7); + Label loopCopyStart = new Label(); + Label loopCopyEnd = new Label(); + mv.visitLabel(loopCopyStart); + mv.visitVarInsn(ILOAD, 7); + mv.visitVarInsn(ILOAD, 5); + mv.visitJumpInsn(IF_ICMPGE, loopCopyEnd); + mv.visitVarInsn(ALOAD, 6); + mv.visitVarInsn(ILOAD, 7); + mv.visitVarInsn(ALOAD, 4); + mv.visitVarInsn(ILOAD, 7); + mv.visitMethodInsn(INVOKEVIRTUAL, SPARSE_SET, "get", "(I)I", false); + mv.visitInsn(IASTORE); + mv.visitIincInsn(7, 1); + mv.visitJumpInsn(GOTO, loopCopyStart); + mv.visitLabel(loopCopyEnd); + + // Arrays.sort(result) + mv.visitVarInsn(ALOAD, 6); + mv.visitMethodInsn(INVOKESTATIC, ARRAYS, "sort", "([I)V", false); + + mv.visitVarInsn(ALOAD, 6); + mv.visitInsn(ARETURN); + mv.visitMaxs(0, 0); + mv.visitEnd(); + } + + /** + * Emits {@code public int[] apply(int[], int)} — the {@link NfaStep} interface method. Delegates + * to {@code nfaStep} so the generated class satisfies the {@code NfaStep} contract without + * requiring INVOKEDYNAMIC/LambdaMetafactory (which is problematic in hidden classes). + */ + /** + * Emits {@code public boolean matches(String input)} as a plain delegation to the public {@code + * CACHE.matches(input, this)} call. Unlike {@link #generateMatchesMethod}, this version accesses + * no package-private {@code LazyDFACache} members and is safe to emit for AOT classes that live + * outside {@code com.datadoghq.reggie.runtime}. + */ + public void generateMatchesDelegatingMethod(ClassWriter cw, String className) { + MethodVisitor mv = cw.visitMethod(ACC_PUBLIC, "matches", "(Ljava/lang/String;)Z", null, null); + mv.visitCode(); + mv.visitFieldInsn(GETSTATIC, className, "CACHE", "L" + LAZY_CACHE + ";"); + mv.visitVarInsn(ALOAD, 1); // input + mv.visitVarInsn(ALOAD, 0); // this (implements NfaStep) + mv.visitMethodInsn( + INVOKEVIRTUAL, LAZY_CACHE, "matches", "(Ljava/lang/String;L" + NFA_STEP + ";)Z", false); + mv.visitInsn(IRETURN); + mv.visitMaxs(0, 0); + mv.visitEnd(); + } + + public void generateApplyMethod(ClassWriter cw, String className) { + MethodVisitor mv = cw.visitMethod(ACC_PUBLIC, "apply", "([II)[I", null, null); + mv.visitCode(); + mv.visitVarInsn(ALOAD, 0); + mv.visitVarInsn(ALOAD, 1); + mv.visitVarInsn(ILOAD, 2); + mv.visitMethodInsn(INVOKEVIRTUAL, className, "nfaStep", "([II)[I", false); + mv.visitInsn(ARETURN); + mv.visitMaxs(0, 0); + mv.visitEnd(); + } + + /** + * Emits {@code public boolean matches(String input)} with the hot loop inlined — eliminates the + * {@code LazyDFACache.matches()} call frame and exposes the per-character ASCII table read + * directly to the JIT for better optimization. The ASCII table element read uses {@code + * INT_ARRAY_VH.getAcquire} on weakly-ordered platforms ({@link #NEEDS_INT_ARRAY_ACQUIRE}) and a + * plain {@code IALOAD} on x86/TSO where TSO provides acquire semantics for free. + * + *

Equivalent Java: + * + *

+   *   public boolean matches(String input) {
+   *     if (input == null) return false;
+   *     LazyDFACache cache = CACHE;
+   *     int dfaState = 0;
+   *     for (int pos = 0, len = input.length(); pos < len; pos++) {
+   *       int c = input.charAt(pos);
+   *       int[] table = cache.asciiTables[dfaState];
+   *       int next = (table != null && c < 128)
+   *           ? (int) LazyDFACache.INT_ARRAY_VH.getAcquire(table, c) : LazyDFACache.UNCACHED;
+   *       if (next == LazyDFACache.UNCACHED) next = cache.lookupOrCompute(dfaState, c, this);
+   *       if (next == LazyDFACache.DEAD) return false;
+   *       if (next == LazyDFACache.FALLBACK)
+   *         return cache.nfaFallbackMatch(input, pos, cache.nfaStateSets[dfaState], this);
+   *       dfaState = next;
+   *     }
+   *     return cache.accepting[dfaState];
+   *   }
+   * 
+ * + * Variable layout: 0=this, 1=input, 2=cache, 3=dfaState, 4=len, 5=pos, 6=c, 7=table, 8=next + */ + public void generateMatchesMethod(ClassWriter cw, String className) { + if (!className.startsWith("com/datadoghq/reggie/runtime/")) { + throw new IllegalArgumentException( + "generateMatchesMethod emits GETFIELD on package-private LazyDFACache members; " + + "className must be in com/datadoghq/reggie/runtime/ but got: " + + className + + ". Use generateMatchesDelegatingMethod for classes in other packages."); + } + MethodVisitor mv = cw.visitMethod(ACC_PUBLIC, "matches", "(Ljava/lang/String;)Z", null, null); + mv.visitCode(); + + // if (input == null) return false + Label notNull = new Label(); + mv.visitVarInsn(ALOAD, 1); + mv.visitJumpInsn(IFNONNULL, notNull); + mv.visitInsn(ICONST_0); + mv.visitInsn(IRETURN); + mv.visitLabel(notNull); + + // LazyDFACache cache = CACHE (slot 2) + mv.visitFieldInsn(GETSTATIC, className, "CACHE", "L" + LAZY_CACHE + ";"); + mv.visitVarInsn(ASTORE, 2); + + // int dfaState = 0 (slot 3) + mv.visitInsn(ICONST_0); + mv.visitVarInsn(ISTORE, 3); + + // int len = input.length() (slot 4) + mv.visitVarInsn(ALOAD, 1); + mv.visitMethodInsn(INVOKEVIRTUAL, "java/lang/String", "length", "()I", false); + mv.visitVarInsn(ISTORE, 4); + + // int pos = 0 (slot 5) + mv.visitInsn(ICONST_0); + mv.visitVarInsn(ISTORE, 5); + + Label loopStart = new Label(), loopEnd = new Label(); + mv.visitLabel(loopStart); + // if (pos >= len) break + mv.visitVarInsn(ILOAD, 5); + mv.visitVarInsn(ILOAD, 4); + mv.visitJumpInsn(IF_ICMPGE, loopEnd); + + // int c = input.charAt(pos) (slot 6) + mv.visitVarInsn(ALOAD, 1); + mv.visitVarInsn(ILOAD, 5); + mv.visitMethodInsn(INVOKEVIRTUAL, "java/lang/String", "charAt", "(I)C", false); + mv.visitVarInsn(ISTORE, 6); + + // int[] table = (int[]) TABLES_VH.getAcquire(cache.asciiTables, dfaState) (slot 7) + // getAcquire establishes happens-before with the setRelease in LazyDFACache.cacheEntry, + // so readers always see a fully-initialized table on all platforms including ARM. + mv.visitFieldInsn(GETSTATIC, LAZY_CACHE, "TABLES_VH", "Ljava/lang/invoke/VarHandle;"); + mv.visitVarInsn(ALOAD, 2); + mv.visitFieldInsn(GETFIELD, LAZY_CACHE, "asciiTables", "[[I"); + mv.visitVarInsn(ILOAD, 3); + mv.visitMethodInsn( + INVOKEVIRTUAL, "java/lang/invoke/VarHandle", "getAcquire", "([[II)[I", false); + mv.visitVarInsn(ASTORE, 7); + + // int next = (table != null && c < 128) ? : UNCACHED (slot 8) + // On weakly-ordered platforms (ARM/RISC-V): INT_ARRAY_VH.getAcquire pairs with the + // setRelease in LazyDFACache.cacheEntry, making nfaStateSets/accepting visible. + // On x86/TSO: TSO gives loads acquire semantics for free — plain IALOAD is sufficient. + Label slowPath = new Label(), afterTableRead = new Label(); + mv.visitVarInsn(ALOAD, 7); + mv.visitJumpInsn(IFNULL, slowPath); // table == null → slow path + mv.visitVarInsn(ILOAD, 6); + pushInt(mv, 128); + mv.visitJumpInsn(IF_ICMPGE, slowPath); // c >= 128 → slow path + if (NEEDS_INT_ARRAY_ACQUIRE) { + // ARM/RISC-V: acquire read via VarHandle + mv.visitFieldInsn(GETSTATIC, LAZY_CACHE, "INT_ARRAY_VH", "Ljava/lang/invoke/VarHandle;"); + mv.visitVarInsn(ALOAD, 7); // table + mv.visitVarInsn(ILOAD, 6); // c + mv.visitMethodInsn( + INVOKEVIRTUAL, "java/lang/invoke/VarHandle", "getAcquire", "([II)I", false); + } else { + // x86/TSO: plain array element load — zero VarHandle overhead + mv.visitVarInsn(ALOAD, 7); // table + mv.visitVarInsn(ILOAD, 6); // c + mv.visitInsn(IALOAD); + } + mv.visitVarInsn(ISTORE, 8); + mv.visitJumpInsn(GOTO, afterTableRead); + mv.visitLabel(slowPath); + pushInt(mv, UNCACHED); + mv.visitVarInsn(ISTORE, 8); + mv.visitLabel(afterTableRead); + + // if (next == UNCACHED) next = cache.lookupOrCompute(dfaState, c, this) + Label notUncached = new Label(); + mv.visitVarInsn(ILOAD, 8); + pushInt(mv, UNCACHED); + mv.visitJumpInsn(IF_ICMPNE, notUncached); + mv.visitVarInsn(ALOAD, 2); // cache + mv.visitVarInsn(ILOAD, 3); // dfaState + mv.visitVarInsn(ILOAD, 6); // c + mv.visitVarInsn(ALOAD, 0); // this (NfaStep) + mv.visitMethodInsn( + INVOKEVIRTUAL, LAZY_CACHE, "lookupOrCompute", "(IIL" + NFA_STEP + ";)I", false); + mv.visitVarInsn(ISTORE, 8); + mv.visitLabel(notUncached); + + // if (next == DEAD) return false + mv.visitVarInsn(ILOAD, 8); + pushInt(mv, DEAD); + Label notDead = new Label(); + mv.visitJumpInsn(IF_ICMPNE, notDead); + mv.visitInsn(ICONST_0); + mv.visitInsn(IRETURN); + mv.visitLabel(notDead); + + // if (next == FALLBACK) return cache.nfaFallbackMatch(input, pos, cache.nfaStateSets[dfaState], + // this) + mv.visitVarInsn(ILOAD, 8); + pushInt(mv, FALLBACK); + Label notFallback = new Label(); + mv.visitJumpInsn(IF_ICMPNE, notFallback); + mv.visitVarInsn(ALOAD, 2); // cache + mv.visitVarInsn(ALOAD, 1); // input + mv.visitVarInsn(ILOAD, 5); // pos + mv.visitVarInsn(ALOAD, 2); // cache (for nfaStateSets) + mv.visitFieldInsn(GETFIELD, LAZY_CACHE, "nfaStateSets", "[[I"); + mv.visitVarInsn(ILOAD, 3); // dfaState + mv.visitInsn(AALOAD); // cache.nfaStateSets[dfaState] + mv.visitVarInsn(ALOAD, 0); // this (NfaStep) + mv.visitMethodInsn( + INVOKEVIRTUAL, + LAZY_CACHE, + "nfaFallbackMatch", + "(Ljava/lang/String;I[IL" + NFA_STEP + ";)Z", + false); + mv.visitInsn(IRETURN); + mv.visitLabel(notFallback); + + // dfaState = next + mv.visitVarInsn(ILOAD, 8); + mv.visitVarInsn(ISTORE, 3); + + // pos++; goto loopStart + mv.visitIincInsn(5, 1); + mv.visitJumpInsn(GOTO, loopStart); + mv.visitLabel(loopEnd); + + // return cache.accepting[dfaState] + mv.visitVarInsn(ALOAD, 2); + mv.visitFieldInsn(GETFIELD, LAZY_CACHE, "accepting", "[Z"); + mv.visitVarInsn(ILOAD, 3); + mv.visitInsn(BALOAD); + mv.visitInsn(IRETURN); + mv.visitMaxs(0, 0); + mv.visitEnd(); + } + + /** + * Emits {@code public MatchResult matchBounded(String input, int start, int end)}: delegates to + * {@code CACHE.matchesBounded(input, start, end, this)} — no substring allocation. This String + * overload is called internally by the NFA-delegated {@code findMatchFrom} method, which scans + * all end positions; using {@code matchesBounded} avoids O(n²) substring copies on large inputs. + * Variable layout: 0=this, 1=input, 2=start, 3=end. + */ + public void generateMatchBoundedStringMethod(ClassWriter cw, String className) { + String matchResultImpl = "com/datadoghq/reggie/runtime/MatchResultImpl"; + String matchResult = "com/datadoghq/reggie/runtime/MatchResult"; + MethodVisitor mv = + cw.visitMethod( + ACC_PUBLIC, "matchBounded", "(Ljava/lang/String;II)L" + matchResult + ";", null, null); + mv.visitCode(); + + // if (!CACHE.matchesBounded(input, start, end, this)) return null + mv.visitFieldInsn(GETSTATIC, className, "CACHE", "L" + LAZY_CACHE + ";"); + mv.visitVarInsn(ALOAD, 1); + mv.visitVarInsn(ILOAD, 2); + mv.visitVarInsn(ILOAD, 3); + mv.visitVarInsn(ALOAD, 0); // this (NfaStep) + mv.visitMethodInsn( + INVOKEVIRTUAL, + LAZY_CACHE, + "matchesBounded", + "(Ljava/lang/String;IIL" + NFA_STEP + ";)Z", + false); + Label matched = new Label(); + mv.visitJumpInsn(IFNE, matched); + mv.visitInsn(ACONST_NULL); + mv.visitInsn(ARETURN); + mv.visitLabel(matched); + + // new MatchResultImpl(input, {start}, {end}, 0) — starts[0]=start, ends[0]=end + mv.visitTypeInsn(NEW, matchResultImpl); + mv.visitInsn(DUP); + mv.visitVarInsn(ALOAD, 1); // original input + // starts = new int[]{start} + mv.visitInsn(ICONST_1); + mv.visitIntInsn(NEWARRAY, T_INT); + mv.visitInsn(DUP); + mv.visitInsn(ICONST_0); + mv.visitVarInsn(ILOAD, 2); // starts[0] = start + mv.visitInsn(IASTORE); + // ends = new int[]{end} + mv.visitInsn(ICONST_1); + mv.visitIntInsn(NEWARRAY, T_INT); + mv.visitInsn(DUP); + mv.visitInsn(ICONST_0); + mv.visitVarInsn(ILOAD, 3); // ends[0] = end + mv.visitInsn(IASTORE); + mv.visitInsn(ICONST_0); + mv.visitMethodInsn( + INVOKESPECIAL, matchResultImpl, "", "(Ljava/lang/String;[I[II)V", false); + mv.visitInsn(ARETURN); + mv.visitMaxs(0, 0); + mv.visitEnd(); + } + + /** + * Emits {@code public MatchResult match(String input)}: returns a full-input MatchResultImpl if + * {@code matches(input)} is true, otherwise null. No capturing groups (group-free pattern). + */ + public void generateMatchMethod(ClassWriter cw, String className) { + // Descriptor: (Ljava/lang/String;)Lcom/datadoghq/reggie/runtime/MatchResult; + String matchResultImpl = "com/datadoghq/reggie/runtime/MatchResultImpl"; + String matchResult = "com/datadoghq/reggie/runtime/MatchResult"; + MethodVisitor mv = + cw.visitMethod( + ACC_PUBLIC, "match", "(Ljava/lang/String;)L" + matchResult + ";", null, null); + mv.visitCode(); + + // if (!matches(input)) return null; + mv.visitVarInsn(ALOAD, 0); + mv.visitVarInsn(ALOAD, 1); + mv.visitMethodInsn(INVOKEVIRTUAL, className, "matches", "(Ljava/lang/String;)Z", false); + Label matched = new Label(); + mv.visitJumpInsn(IFNE, matched); + mv.visitInsn(ACONST_NULL); + mv.visitInsn(ARETURN); + mv.visitLabel(matched); + + // new MatchResultImpl(input, {0}, {input.length()}, 0) + // starts[0]=0, ends[0]=input.length() — group 0 spans the whole input. + mv.visitTypeInsn(NEW, matchResultImpl); + mv.visitInsn(DUP); + mv.visitVarInsn(ALOAD, 1); // input + // starts = new int[]{0} + mv.visitInsn(ICONST_1); + mv.visitIntInsn(NEWARRAY, T_INT); + mv.visitInsn(DUP); + mv.visitInsn(ICONST_0); + mv.visitInsn(ICONST_0); // starts[0] = 0 + mv.visitInsn(IASTORE); + // ends = new int[]{input.length()} + mv.visitInsn(ICONST_1); + mv.visitIntInsn(NEWARRAY, T_INT); + mv.visitInsn(DUP); + mv.visitInsn(ICONST_0); + mv.visitVarInsn(ALOAD, 1); + mv.visitMethodInsn(INVOKEVIRTUAL, "java/lang/String", "length", "()I", false); + mv.visitInsn(IASTORE); // ends[0] = input.length() + mv.visitInsn(ICONST_0); + mv.visitMethodInsn( + INVOKESPECIAL, matchResultImpl, "", "(Ljava/lang/String;[I[II)V", false); + mv.visitInsn(ARETURN); + mv.visitMaxs(0, 0); + mv.visitEnd(); + } + + /** + * Emits {@code public boolean matchesBounded(CharSequence input, int start, int end)}: extracts + * the subsequence as a String and delegates to {@code matches()}. + */ + public void generateMatchesBoundedMethod(ClassWriter cw, String className) { + MethodVisitor mv = + cw.visitMethod(ACC_PUBLIC, "matchesBounded", "(Ljava/lang/CharSequence;II)Z", null, null); + mv.visitCode(); + // this.matches(input.subSequence(start, end).toString()) + mv.visitVarInsn(ALOAD, 0); + mv.visitVarInsn(ALOAD, 1); + mv.visitVarInsn(ILOAD, 2); + mv.visitVarInsn(ILOAD, 3); + mv.visitMethodInsn( + INVOKEINTERFACE, + "java/lang/CharSequence", + "subSequence", + "(II)Ljava/lang/CharSequence;", + true); + mv.visitMethodInsn( + INVOKEINTERFACE, "java/lang/CharSequence", "toString", "()Ljava/lang/String;", true); + mv.visitMethodInsn(INVOKEVIRTUAL, className, "matches", "(Ljava/lang/String;)Z", false); + mv.visitInsn(IRETURN); + mv.visitMaxs(0, 0); + mv.visitEnd(); + } + + /** + * Emits {@code public MatchResult matchBounded(CharSequence input, int start, int end)}: returns + * a MatchResultImpl for the bounded region if it matches, otherwise null. + */ + public void generateMatchBoundedMethod(ClassWriter cw, String className) { + String matchResultImpl = "com/datadoghq/reggie/runtime/MatchResultImpl"; + String matchResult = "com/datadoghq/reggie/runtime/MatchResult"; + MethodVisitor mv = + cw.visitMethod( + ACC_PUBLIC, + "matchBounded", + "(Ljava/lang/CharSequence;II)L" + matchResult + ";", + null, + null); + mv.visitCode(); + + // String sub = input.subSequence(start, end).toString() + mv.visitVarInsn(ALOAD, 1); + mv.visitVarInsn(ILOAD, 2); + mv.visitVarInsn(ILOAD, 3); + mv.visitMethodInsn( + INVOKEINTERFACE, + "java/lang/CharSequence", + "subSequence", + "(II)Ljava/lang/CharSequence;", + true); + mv.visitMethodInsn( + INVOKEINTERFACE, "java/lang/CharSequence", "toString", "()Ljava/lang/String;", true); + mv.visitVarInsn(ASTORE, 4); // sub + + // if (!matches(sub)) return null; + mv.visitVarInsn(ALOAD, 0); + mv.visitVarInsn(ALOAD, 4); + mv.visitMethodInsn(INVOKEVIRTUAL, className, "matches", "(Ljava/lang/String;)Z", false); + Label matchedBounded = new Label(); + mv.visitJumpInsn(IFNE, matchedBounded); + mv.visitInsn(ACONST_NULL); + mv.visitInsn(ARETURN); + mv.visitLabel(matchedBounded); + + // new MatchResultImpl(input.toString(), {start}, {end}, 0) + // starts[0]=start, ends[0]=end — absolute offsets into the original input. + mv.visitTypeInsn(NEW, matchResultImpl); + mv.visitInsn(DUP); + mv.visitVarInsn(ALOAD, 1); // original input (CharSequence) + mv.visitMethodInsn( + INVOKEINTERFACE, "java/lang/CharSequence", "toString", "()Ljava/lang/String;", true); + // starts = new int[]{start} + mv.visitInsn(ICONST_1); + mv.visitIntInsn(NEWARRAY, T_INT); + mv.visitInsn(DUP); + mv.visitInsn(ICONST_0); + mv.visitVarInsn(ILOAD, 2); // starts[0] = start + mv.visitInsn(IASTORE); + // ends = new int[]{end} + mv.visitInsn(ICONST_1); + mv.visitIntInsn(NEWARRAY, T_INT); + mv.visitInsn(DUP); + mv.visitInsn(ICONST_0); + mv.visitVarInsn(ILOAD, 3); // ends[0] = end + mv.visitInsn(IASTORE); + mv.visitInsn(ICONST_0); + mv.visitMethodInsn( + INVOKESPECIAL, matchResultImpl, "", "(Ljava/lang/String;[I[II)V", false); + mv.visitInsn(ARETURN); + mv.visitMaxs(0, 0); + mv.visitEnd(); + } + + // ── private helpers ────────────────────────────────────────────────────── + + // Maximum bytecode bytes per helper method (conservative, well under 64 KB limit). + private static final int MAX_HELPER_BYTES = 40_000; + + /** + * Emits a 2-D int array field initializer. For large arrays the bytecode is split across multiple + * private static helper methods (chunks of states) to stay within the 64 KB JVM method limit. + */ + private static void emitSplitInt2DArrayInit( + ClassWriter cw, + MethodVisitor clinit, + String className, + String field, + int[][] data, + String desc) { + // Estimate total bytecode. Each row of length L costs roughly (10 + 8*L) bytes. + long totalEstimate = 0; + for (int[] row : data) { + totalEstimate += 10L + 8L * row.length; + } + + if (totalEstimate <= MAX_HELPER_BYTES) { + // Small enough to inline directly into + emitInt2DArrayInit(clinit, className, field, data, desc); + return; + } + + // Allocate the outer array and store it in the field first. + pushInt(clinit, data.length); + clinit.visitTypeInsn(ANEWARRAY, "[I"); + clinit.visitFieldInsn(PUTSTATIC, className, field, desc); + + // Split into variable-size chunks, each within MAX_HELPER_BYTES. + int chunkStart = 0; + int chunkIndex = 0; + while (chunkStart < data.length) { + // Find end of this chunk. + long chunkBytes = 0; + int chunkEnd = chunkStart; + while (chunkEnd < data.length) { + long rowBytes = 10L + 8L * data[chunkEnd].length; + if (chunkBytes + rowBytes > MAX_HELPER_BYTES && chunkEnd > chunkStart) break; + chunkBytes += rowBytes; + chunkEnd++; + } + + String helperName = "init" + field + "_" + chunkIndex; + emitPartialArrayHelper(cw, className, helperName, field, data, chunkStart, chunkEnd); + clinit.visitMethodInsn(INVOKESTATIC, className, helperName, "()V", false); + + chunkStart = chunkEnd; + chunkIndex++; + } + } + + /** Emits a private static helper that fills a range of an already-stored 2-D array field. */ + private static void emitPartialArrayHelper( + ClassWriter cw, + String className, + String methodName, + String field, + int[][] data, + int from, + int to) { + MethodVisitor mv = cw.visitMethod(ACC_PRIVATE | ACC_STATIC, methodName, "()V", null, null); + mv.visitCode(); + for (int i = from; i < to; i++) { + mv.visitFieldInsn(GETSTATIC, className, field, "[[I"); + pushInt(mv, i); + int[] row = data[i]; + pushInt(mv, row.length); + mv.visitIntInsn(NEWARRAY, T_INT); + for (int j = 0; j < row.length; j++) { + mv.visitInsn(DUP); + pushInt(mv, j); + pushInt(mv, row[j]); + mv.visitInsn(IASTORE); + } + mv.visitInsn(AASTORE); + } + mv.visitInsn(RETURN); + mv.visitMaxs(0, 0); + mv.visitEnd(); + } + + private static void emitInt1DArrayInit( + MethodVisitor mv, String className, String field, int[] data, String desc) { + pushInt(mv, data.length); + mv.visitIntInsn(NEWARRAY, T_INT); + for (int i = 0; i < data.length; i++) { + mv.visitInsn(DUP); + pushInt(mv, i); + pushInt(mv, data[i]); + mv.visitInsn(IASTORE); + } + mv.visitFieldInsn(PUTSTATIC, className, field, desc); + } + + private static void emitInt2DArrayInit( + MethodVisitor mv, String className, String field, int[][] data, String desc) { + pushInt(mv, data.length); + mv.visitTypeInsn(ANEWARRAY, "[I"); + for (int i = 0; i < data.length; i++) { + mv.visitInsn(DUP); + pushInt(mv, i); + int[] row = data[i]; + pushInt(mv, row.length); + mv.visitIntInsn(NEWARRAY, T_INT); + for (int j = 0; j < row.length; j++) { + mv.visitInsn(DUP); + pushInt(mv, j); + pushInt(mv, row[j]); + mv.visitInsn(IASTORE); + } + mv.visitInsn(AASTORE); + } + mv.visitFieldInsn(PUTSTATIC, className, field, desc); + } + + private static int[][] buildTransitions(NFA nfa) { + int n = nfa.getStates().size(); + int[][] result = new int[n][]; + for (NFA.NFAState state : nfa.getStates()) { + List triples = new ArrayList<>(); + for (NFA.Transition t : state.getTransitions()) { + for (CharSet.Range r : t.chars.getRanges()) { + triples.add((int) r.start); + triples.add((int) r.end); + triples.add(t.target.id); + } + } + result[state.id] = triples.stream().mapToInt(Integer::intValue).toArray(); + } + return result; + } + + private static int[][] buildEpsClosure(NFA nfa) { + int n = nfa.getStates().size(); + int[][] result = new int[n][]; + for (NFA.NFAState state : nfa.getStates()) { + Set closure = new HashSet<>(); + Deque worklist = new ArrayDeque<>(); + worklist.add(state); + while (!worklist.isEmpty()) { + NFA.NFAState s = worklist.poll(); + if (closure.add(s.id)) { + for (NFA.NFAState eps : s.getEpsilonTransitions()) { + if (eps.anchor == null) worklist.add(eps); + } + } + } + result[state.id] = closure.stream().mapToInt(Integer::intValue).sorted().toArray(); + } + return result; + } +} diff --git a/reggie-codegen/src/test/java/com/datadoghq/reggie/codegen/analysis/PatternAnalyzerLazyDFATest.java b/reggie-codegen/src/test/java/com/datadoghq/reggie/codegen/analysis/PatternAnalyzerLazyDFATest.java new file mode 100644 index 0000000..887f51a --- /dev/null +++ b/reggie-codegen/src/test/java/com/datadoghq/reggie/codegen/analysis/PatternAnalyzerLazyDFATest.java @@ -0,0 +1,83 @@ +/* + * Copyright 2026-Present Datadog, Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.datadoghq.reggie.codegen.analysis; + +import static org.junit.jupiter.api.Assertions.*; + +import com.datadoghq.reggie.codegen.ast.RegexNode; +import com.datadoghq.reggie.codegen.automaton.NFA; +import com.datadoghq.reggie.codegen.automaton.ThompsonBuilder; +import com.datadoghq.reggie.codegen.parsing.RegexParser; +import org.junit.jupiter.api.Test; + +class PatternAnalyzerLazyDFATest { + + private PatternAnalyzer.MatchingStrategyResult analyze(String pattern) throws Exception { + RegexParser parser = new RegexParser(); + RegexNode ast = parser.parse(pattern); + ThompsonBuilder builder = new ThompsonBuilder(); + NFA nfa = builder.build(ast, 0); + return new PatternAnalyzer(ast, nfa).analyzeAndRecommend(); + } + + @Test + void testRouteToLazyDFAWhenNFALarge() throws Exception { + // (?:a+b+|b+a+){75} has ~685 NFA states, no groups/anchors. + // DFA state explosion via interleaved a+/b+ alternation → StateExplosionException + // → OPTIMIZED_NFA → isLazyDFAEligible (NFA ≥300, no groups/anchors) → LAZY_DFA. + PatternAnalyzer.MatchingStrategyResult r = analyze("(?:a+b+|b+a+){75}"); + assertEquals(PatternAnalyzer.MatchingStrategy.LAZY_DFA, r.strategy); + } + + @Test + void testDoNotRouteWhenNFASmall() throws Exception { + // (?:a?){50} has ~100 NFA states, no capturing groups, no anchors — below the 300-state + // threshold. Uses non-capturing group so the assertion exercises the size guard specifically + // (a capturing group would also be excluded by the group check). + PatternAnalyzer.MatchingStrategyResult r = analyze("(?:a?){50}"); + assertNotEquals(PatternAnalyzer.MatchingStrategy.LAZY_DFA, r.strategy); + } + + @Test + void testDoNotRouteWithLookahead() throws Exception { + // Lookahead → OPTIMIZED_NFA_WITH_LOOKAROUND, not LAZY_DFA + PatternAnalyzer.MatchingStrategyResult r = analyze("(?=[a-z])(?:a+b+|b+a+){75}"); + assertNotEquals(PatternAnalyzer.MatchingStrategy.LAZY_DFA, r.strategy); + } + + @Test + void testDoNotRouteWithAnchor() throws Exception { + // Anchored pattern must not route to LAZY_DFA + PatternAnalyzer.MatchingStrategyResult r = analyze("^(?:a+b+|b+a+){75}"); + assertNotEquals(PatternAnalyzer.MatchingStrategy.LAZY_DFA, r.strategy); + } + + @Test + void testDoNotRouteWithBackref() throws Exception { + // Pattern with backreference → OPTIMIZED_NFA_WITH_BACKREFS, not LAZY_DFA + PatternAnalyzer.MatchingStrategyResult r = analyze("((?:[a-z][0-9]){100})\\1"); + assertNotEquals(PatternAnalyzer.MatchingStrategy.LAZY_DFA, r.strategy); + } + + @Test + void testDoNotRouteWithCapturingGroup() throws Exception { + // Capturing group (no backref) — directly exercises the hasCapturingGroups guard. + // The backref test exits via OPTIMIZED_NFA_WITH_BACKREFS before isLazyDFAEligible is + // consulted, so it does not cover this path. + PatternAnalyzer.MatchingStrategyResult r = analyze("((?:a+b+|b+a+){75})"); + assertNotEquals(PatternAnalyzer.MatchingStrategy.LAZY_DFA, r.strategy); + } +} diff --git a/reggie-processor/src/main/java/com/datadoghq/reggie/processor/ReggieMatcherBytecodeGenerator.java b/reggie-processor/src/main/java/com/datadoghq/reggie/processor/ReggieMatcherBytecodeGenerator.java index 8bc82db..48d7e68 100644 --- a/reggie-processor/src/main/java/com/datadoghq/reggie/processor/ReggieMatcherBytecodeGenerator.java +++ b/reggie-processor/src/main/java/com/datadoghq/reggie/processor/ReggieMatcherBytecodeGenerator.java @@ -37,6 +37,7 @@ import com.datadoghq.reggie.codegen.codegen.FixedRepetitionBackrefBytecodeGenerator; import com.datadoghq.reggie.codegen.codegen.FixedSequenceBytecodeGenerator; import com.datadoghq.reggie.codegen.codegen.GreedyCharClassBytecodeGenerator; +import com.datadoghq.reggie.codegen.codegen.LazyDFABytecodeGenerator; import com.datadoghq.reggie.codegen.codegen.LinearPatternBytecodeGenerator; import com.datadoghq.reggie.codegen.codegen.LiteralAlternationTrieGenerator; import com.datadoghq.reggie.codegen.codegen.MultiGroupGreedyBytecodeGenerator; @@ -105,14 +106,19 @@ public byte[] generate() throws Exception { // 4. Generate bytecode based on strategy ClassWriter cw = new ClassWriter(ClassWriter.COMPUTE_FRAMES | ClassWriter.COMPUTE_MAXS); - // Class declaration + // Class declaration — LAZY_DFA generated class implements NfaStep directly so the cache + // can call nfaStep() without INVOKEDYNAMIC (which is unsupported in hidden classes). + String[] ifaces = + strategy == PatternAnalyzer.MatchingStrategy.LAZY_DFA + ? new String[] {"com/datadoghq/reggie/runtime/NfaStep"} + : null; cw.visit( V21, ACC_PUBLIC | ACC_FINAL | ACC_SUPER, className, null, "com/datadoghq/reggie/runtime/ReggieMatcher", - null); + ifaces); // Generate constructor (with NFA state initialization for NFA-based strategies) boolean needsNFAState = @@ -121,7 +127,8 @@ public byte[] generate() throws Exception { || strategy == PatternAnalyzer.MatchingStrategy.OPTIMIZED_NFA_WITH_LOOKAROUND || strategy == PatternAnalyzer.MatchingStrategy.HYBRID_DFA_LOOKAHEAD || strategy == PatternAnalyzer.MatchingStrategy.SPECIALIZED_MULTIPLE_LOOKAHEADS - || strategy == PatternAnalyzer.MatchingStrategy.SPECIALIZED_LITERAL_LOOKAHEADS; + || strategy == PatternAnalyzer.MatchingStrategy.SPECIALIZED_LITERAL_LOOKAHEADS + || strategy == PatternAnalyzer.MatchingStrategy.LAZY_DFA; boolean needsRecursiveDescent = strategy == PatternAnalyzer.MatchingStrategy.RECURSIVE_DESCENT; generateConstructor(cw, needsNFAState, needsRecursiveDescent, nfa, nameMap); @@ -498,6 +505,40 @@ public byte[] generate() throws Exception { nestedGen.generate(cw); break; + case LAZY_DFA: + { + LazyDFABytecodeGenerator lazyGen = new LazyDFABytecodeGenerator(nfa); + lazyGen.generateStaticFields(cw, getJavaClassName()); + lazyGen.generateNfaStepMethod(cw, getJavaClassName()); + lazyGen.generateApplyMethod(cw, getJavaClassName()); + // Use the delegating (non-inlined) matches() — the AOT class lives in the user's package + // and cannot access package-private LazyDFACache members that the inlined version uses. + lazyGen.generateMatchesDelegatingMethod(cw, getJavaClassName()); + lazyGen.generateMatchMethod(cw, getJavaClassName()); + // CharSequence abstract methods required by ReggieMatcher. + lazyGen.generateMatchesBoundedMethod(cw, getJavaClassName()); + lazyGen.generateMatchBoundedMethod(cw, getJavaClassName()); + // Remaining methods delegate to the standard NFA generator. + NFABytecodeGenerator nfaDelegate = + new NFABytecodeGenerator( + nfa, + null, + null, + result.requiredLiterals, + result.lookaheadGreedyInfo, + false, + caseInsensitive); + // String overload of matchBounded called internally by findMatchFrom/findMatch. + lazyGen.generateMatchBoundedStringMethod(cw, getJavaClassName()); + nfaDelegate.generateFindMethod(cw, getJavaClassName()); + nfaDelegate.generateFindFromMethod(cw, getJavaClassName()); + nfaDelegate.generateFindLongestMatchEndMethod(cw, getJavaClassName()); + nfaDelegate.generateFindMatchMethod(cw, getJavaClassName()); + nfaDelegate.generateFindMatchFromMethod(cw, getJavaClassName()); + nfaDelegate.generateFindBoundsFromMethod(cw, getJavaClassName()); + break; + } + default: throw new IllegalStateException("Unknown strategy: " + strategy); } diff --git a/reggie-processor/src/test/java/com/datadoghq/reggie/processor/ReggieMatcherBytecodeGeneratorTest.java b/reggie-processor/src/test/java/com/datadoghq/reggie/processor/ReggieMatcherBytecodeGeneratorTest.java index 9899db9..8ff08c7 100644 --- a/reggie-processor/src/test/java/com/datadoghq/reggie/processor/ReggieMatcherBytecodeGeneratorTest.java +++ b/reggie-processor/src/test/java/com/datadoghq/reggie/processor/ReggieMatcherBytecodeGeneratorTest.java @@ -480,6 +480,19 @@ void testLookbehindBeforeUnboundedQuantifier() throws Exception { assertEquals("3X", replaceAllStar.invoke(matcherStar, "3abc", "X")); } + @Test + void testLazyDfaStrategy() throws Exception { + // (?:a+b+|b+a+){75} has ~685 NFA states and no groups/anchors → routes to LAZY_DFA. + // The processor emits the delegating matches() path (no package-private access). + Object matcher = compile("(?:a+b+|b+a+){75}", "LazyDfaMatcher"); + Method matches = matcher.getClass().getMethod("matches", String.class); + Method find = matcher.getClass().getMethod("find", String.class); + assertTrue((Boolean) matches.invoke(matcher, "ab".repeat(75))); + assertFalse((Boolean) matches.invoke(matcher, "ab".repeat(74) + "b")); + assertTrue((Boolean) find.invoke(matcher, "xx" + "ab".repeat(75) + "yy")); + assertFalse((Boolean) find.invoke(matcher, "xx")); + } + /** Custom ClassLoader for loading generated bytecode in tests. */ private static class TestClassLoader extends ClassLoader { public Class defineClass(String name, byte[] bytecode) { diff --git a/reggie-runtime/src/main/java/com/datadoghq/reggie/runtime/LazyDFACache.java b/reggie-runtime/src/main/java/com/datadoghq/reggie/runtime/LazyDFACache.java new file mode 100644 index 0000000..34f4b20 --- /dev/null +++ b/reggie-runtime/src/main/java/com/datadoghq/reggie/runtime/LazyDFACache.java @@ -0,0 +1,244 @@ +/* + * Copyright 2026-Present Datadog, Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.datadoghq.reggie.runtime; + +import java.lang.invoke.MethodHandles; +import java.lang.invoke.VarHandle; +import java.util.Arrays; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.atomic.AtomicInteger; + +/** + * Lazily-materialized DFA cache over NFA execution. + * + *

On first encounter of a (DFA-state, character) pair the NFA is stepped and the resulting + * state-set is interned as a new DFA state; on subsequent encounters the cached state id is + * returned directly from a per-state {@code int[128]} ASCII transition table. The cache is bounded + * at {@link #DEFAULT_CAP} DFA states; once full it freezes and remaining inputs fall back to plain + * NFA stepping. + * + *

The lazily-materialized DFA technique and the state-set interning key idea are adapted from + * Mike Snoyman's {@code lazydfa} implementation in the {@code glob_perf} benchmark suite ({@code + * DataDog/experimental/users/dangermike/glob_perf}). + */ +public final class LazyDFACache { + + /** + * Array-element VarHandle for {@code int[][]} slots. Used to publish newly filled {@code + * int[128]} tables with release semantics and to read them with acquire semantics, establishing a + * happens-before between the writer (in {@link #cacheEntry}) and any reader (in {@link #matches} + * or in the generated inlined hot loop) on all JMM-conformant platforms including ARM. + */ + static final VarHandle TABLES_VH; + + /** + * Array-element VarHandle for {@code int[]} slots. Used to publish individual transition entries + * with release semantics in {@link #cacheEntry} and to read them with acquire semantics on + * weakly-ordered platforms (ARM/RISC-V). On x86/TSO (Total Store Order), plain loads already + * carry acquire semantics, so the acquire read is skipped to avoid VarHandle dispatch overhead. + */ + static final VarHandle INT_ARRAY_VH; + + /** + * {@code true} on weakly-ordered architectures (ARM, RISC-V) where element-level acquire reads + * are required to pair with {@link #INT_ARRAY_VH} release writes. {@code false} on x86/TSO where + * plain array loads are sufficient and the VarHandle overhead is avoided. + */ + static final boolean NEEDS_INT_ARRAY_ACQUIRE; + + static { + try { + TABLES_VH = MethodHandles.arrayElementVarHandle(int[][].class); + INT_ARRAY_VH = MethodHandles.arrayElementVarHandle(int[].class); + } catch (Exception e) { + throw new ExceptionInInitializerError(e); + } + String arch = System.getProperty("os.arch", ""); + // x86 and x86_64 use TSO — plain loads carry acquire semantics at no extra cost. + // Everything else (aarch64, riscv, ppc, ...) requires explicit acquire reads. + NEEDS_INT_ARRAY_ACQUIRE = + !arch.equals("x86") + && !arch.equals("x86_64") + && !arch.equals("amd64") + && !arch.equals("i386"); + } + + static final int DEFAULT_CAP = 4096; + static final int UNCACHED = -1; + static final int DEAD = -2; + static final int FALLBACK = -3; + + private final ConcurrentHashMap stateIndex; + // package-private so the generated class (same package) can inline the hot loop + final int[][] asciiTables; // asciiTables[id] = int[128] or null + final int[][] nfaStateSets; // nfaStateSets[id] = sorted NFA state IDs + final boolean[] accepting; + private final int[] acceptStateIds; + private final AtomicInteger nextId; + private volatile boolean frozen; + private final int cap; + + public LazyDFACache(int[] startStateSet, int[] acceptStateIds) { + this(startStateSet, acceptStateIds, DEFAULT_CAP); + } + + // package-private for tests + LazyDFACache(int[] startStateSet, int[] acceptStateIds, int cap) { + this.cap = cap; + this.acceptStateIds = acceptStateIds; + this.stateIndex = new ConcurrentHashMap<>(); + this.asciiTables = new int[cap][]; + this.nfaStateSets = new int[cap][]; + this.accepting = new boolean[cap]; + this.nextId = new AtomicInteger(1); // 0 = start state + nfaStateSets[0] = startStateSet; + accepting[0] = containsAny(startStateSet, acceptStateIds); + stateIndex.put(new StateSetKey(startStateSet), 0); + } + + public boolean matches(String input, NfaStep nfaStep) { + if (input == null) return false; + int dfaState = 0; + for (int pos = 0; pos < input.length(); pos++) { + int c = input.charAt(pos); + int[] table = (int[]) TABLES_VH.getAcquire(asciiTables, dfaState); + int next = + (table != null && c < 128) + ? (NEEDS_INT_ARRAY_ACQUIRE ? (int) INT_ARRAY_VH.getAcquire(table, c) : table[c]) + : UNCACHED; + if (next == UNCACHED) { + next = lookupOrCompute(dfaState, c, nfaStep); + } + if (next == DEAD) return false; + if (next == FALLBACK) return nfaFallbackMatch(input, pos, nfaStateSets[dfaState], nfaStep); + dfaState = next; + } + return accepting[dfaState]; + } + + int lookupOrCompute(int state, int c, NfaStep nfaStep) { + int[] nextSet = nfaStep.apply(nfaStateSets[state], c); + if (nextSet.length == 0) { + cacheEntry(state, c, DEAD); + return DEAD; + } + + StateSetKey key = new StateSetKey(nextSet); + Integer id = stateIndex.get(key); + + if (id == null && !frozen) { + id = + stateIndex.computeIfAbsent( + key, + k -> { + int newId = nextId.getAndIncrement(); + if (newId < cap) { + nfaStateSets[newId] = k.getStates(); + accepting[newId] = containsAny(k.getStates(), acceptStateIds); + return newId; + } + return null; // over cap: don't insert, keeps map bounded at cap entries + }); + if (id == null) { + frozen = true; + return FALLBACK; + } + } + if (id == null) return FALLBACK; + + cacheEntry(state, c, id); + return id; + } + + void cacheEntry(int state, int c, int value) { + if (c >= 128) return; + int[] table = asciiTables[state]; + if (table == null) { + int[] t = new int[128]; + Arrays.fill(t, UNCACHED); + t[c] = value; + // setRelease: all writes to t[] above are visible to any thread that subsequently + // observes the non-null slot via getAcquire. This establishes a proper happens-before + // edge on weakly-ordered platforms (ARM/RISC-V), preventing a stale 0 from being + // treated as DFA state 0. + TABLES_VH.setRelease(asciiTables, state, t); + } else { + // setRelease: pairs with getAcquire in the hot loop; ensures nfaStateSets/accepting + // initialization (written on the computeIfAbsent thread) is visible to the reader + // that sees this new id on ARM/RISC-V. Idempotent: same key always maps to same value. + INT_ARRAY_VH.setRelease(table, c, value); + } + } + + /** + * Bounded lazy-DFA match over {@code input[start, end)} without copying the substring. Used by + * the String overload of {@code matchBounded} so that the delegated {@code findMatchFrom} loop + * never allocates a substring for each candidate region. + */ + public boolean matchesBounded(String input, int start, int end, NfaStep nfaStep) { + int dfaState = 0; + for (int pos = start; pos < end; pos++) { + int c = input.charAt(pos); + int[] table = (int[]) TABLES_VH.getAcquire(asciiTables, dfaState); + int next = + (table != null && c < 128) + ? (NEEDS_INT_ARRAY_ACQUIRE ? (int) INT_ARRAY_VH.getAcquire(table, c) : table[c]) + : UNCACHED; + if (next == UNCACHED) { + next = lookupOrCompute(dfaState, c, nfaStep); + } + if (next == DEAD) return false; + if (next == FALLBACK) { + return nfaFallbackMatchBounded(input, pos, end, nfaStateSets[dfaState], nfaStep); + } + dfaState = next; + } + return accepting[dfaState]; + } + + boolean nfaFallbackMatch(String input, int fromPos, int[] nfaSet, NfaStep nfaStep) { + int[] states = nfaStep.apply(nfaSet, input.charAt(fromPos)); + for (int pos = fromPos + 1; pos < input.length(); pos++) { + if (states.length == 0) return false; + states = nfaStep.apply(states, input.charAt(pos)); + } + return states.length > 0 && containsAny(states, acceptStateIds); + } + + private boolean nfaFallbackMatchBounded( + String input, int fromPos, int end, int[] nfaSet, NfaStep nfaStep) { + int[] states = nfaStep.apply(nfaSet, input.charAt(fromPos)); + for (int pos = fromPos + 1; pos < end; pos++) { + if (states.length == 0) return false; + states = nfaStep.apply(states, input.charAt(pos)); + } + return states.length > 0 && containsAny(states, acceptStateIds); + } + + // package-private for tests + boolean isFrozen() { + return frozen; + } + + private static boolean containsAny(int[] set, int[] targets) { + for (int t : targets) { + for (int s : set) { + if (s == t) return true; + } + } + return false; + } +} diff --git a/reggie-runtime/src/main/java/com/datadoghq/reggie/runtime/NfaStep.java b/reggie-runtime/src/main/java/com/datadoghq/reggie/runtime/NfaStep.java new file mode 100644 index 0000000..2e67051 --- /dev/null +++ b/reggie-runtime/src/main/java/com/datadoghq/reggie/runtime/NfaStep.java @@ -0,0 +1,22 @@ +/* + * Copyright 2026-Present Datadog, Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.datadoghq.reggie.runtime; + +/** One NFA step: given active state IDs and a character, return the next active state IDs. */ +@FunctionalInterface +public interface NfaStep { + int[] apply(int[] currentStates, int c); +} diff --git a/reggie-runtime/src/main/java/com/datadoghq/reggie/runtime/RuntimeCompiler.java b/reggie-runtime/src/main/java/com/datadoghq/reggie/runtime/RuntimeCompiler.java index c714450..97ee7e8 100644 --- a/reggie-runtime/src/main/java/com/datadoghq/reggie/runtime/RuntimeCompiler.java +++ b/reggie-runtime/src/main/java/com/datadoghq/reggie/runtime/RuntimeCompiler.java @@ -49,6 +49,7 @@ import com.datadoghq.reggie.codegen.codegen.FixedSequenceBytecodeGenerator; import com.datadoghq.reggie.codegen.codegen.GreedyBacktrackBytecodeGenerator; import com.datadoghq.reggie.codegen.codegen.GreedyCharClassBytecodeGenerator; +import com.datadoghq.reggie.codegen.codegen.LazyDFABytecodeGenerator; import com.datadoghq.reggie.codegen.codegen.LinearPatternBytecodeGenerator; import com.datadoghq.reggie.codegen.codegen.LiteralAlternationTrieGenerator; import com.datadoghq.reggie.codegen.codegen.MultiGroupGreedyBytecodeGenerator; @@ -453,14 +454,18 @@ private static byte[] generateBytecode( ClassWriter cw = new ClassWriter(ClassWriter.COMPUTE_FRAMES | ClassWriter.COMPUTE_MAXS); - // Class declaration: public final class XXX extends ReggieMatcher + // Class declaration: public final class XXX extends ReggieMatcher [implements NfaStep] + String[] ifaces = + result.strategy == PatternAnalyzer.MatchingStrategy.LAZY_DFA + ? new String[] {"com/datadoghq/reggie/runtime/NfaStep"} + : null; cw.visit( V21, ACC_PUBLIC | ACC_FINAL | ACC_SUPER, "com/datadoghq/reggie/runtime/" + className, null, "com/datadoghq/reggie/runtime/ReggieMatcher", - null); + ifaces); // RECURSIVE_DESCENT strategy doesn't require additional instance fields // Fields are managed within the recursive parser methods @@ -472,7 +477,8 @@ private static byte[] generateBytecode( || result.strategy == PatternAnalyzer.MatchingStrategy.OPTIMIZED_NFA_WITH_LOOKAROUND || result.strategy == PatternAnalyzer.MatchingStrategy.HYBRID_DFA_LOOKAHEAD || result.strategy == PatternAnalyzer.MatchingStrategy.SPECIALIZED_MULTIPLE_LOOKAHEADS - || result.strategy == PatternAnalyzer.MatchingStrategy.SPECIALIZED_LITERAL_LOOKAHEADS; + || result.strategy == PatternAnalyzer.MatchingStrategy.SPECIALIZED_LITERAL_LOOKAHEADS + || result.strategy == PatternAnalyzer.MatchingStrategy.LAZY_DFA; boolean needsRecursiveDescent = result.strategy == PatternAnalyzer.MatchingStrategy.RECURSIVE_DESCENT; generateConstructor(cw, pattern, className, needsNFAState, needsRecursiveDescent, nfa, ast); @@ -824,6 +830,40 @@ private static byte[] generateBytecode( // tracking and would skip backrefCheck states, producing incorrect bounds. break; } + case LAZY_DFA: + { + LazyDFABytecodeGenerator lazyGen = new LazyDFABytecodeGenerator(nfa); + lazyGen.generateStaticFields(cw, "com/datadoghq/reggie/runtime/" + className); + lazyGen.generateNfaStepMethod(cw, "com/datadoghq/reggie/runtime/" + className); + // apply() satisfies NfaStep; matches/match/matchesBounded/matchBounded are compact. + lazyGen.generateApplyMethod(cw, "com/datadoghq/reggie/runtime/" + className); + lazyGen.generateMatchesMethod(cw, "com/datadoghq/reggie/runtime/" + className); + lazyGen.generateMatchMethod(cw, "com/datadoghq/reggie/runtime/" + className); + lazyGen.generateMatchesBoundedMethod(cw, "com/datadoghq/reggie/runtime/" + className); + lazyGen.generateMatchBoundedMethod(cw, "com/datadoghq/reggie/runtime/" + className); + // find* methods use the standard NFA implementation (no capturing groups involved). + NFABytecodeGenerator nfaDelegate = + new NFABytecodeGenerator( + nfa, + null, + null, + result.requiredLiterals, + result.lookaheadGreedyInfo, + result.usePosixLastMatch, + caseInsensitive); + // String overload of matchBounded called internally by findMatchFrom/findMatch. + // Uses a compact stub (no NFA bytecode) to avoid the 64 KB method size limit. + lazyGen.generateMatchBoundedStringMethod(cw, "com/datadoghq/reggie/runtime/" + className); + nfaDelegate.generateFindMethod(cw, "com/datadoghq/reggie/runtime/" + className); + nfaDelegate.generateFindFromMethod(cw, "com/datadoghq/reggie/runtime/" + className); + nfaDelegate.generateFindLongestMatchEndMethod( + cw, "com/datadoghq/reggie/runtime/" + className); + nfaDelegate.generateFindMatchMethod(cw, "com/datadoghq/reggie/runtime/" + className); + nfaDelegate.generateFindMatchFromMethod(cw, "com/datadoghq/reggie/runtime/" + className); + nfaDelegate.generateFindBoundsFromMethod(cw, "com/datadoghq/reggie/runtime/" + className); + break; + } + case OPTIMIZED_NFA: case OPTIMIZED_NFA_WITH_LOOKAROUND: { diff --git a/reggie-runtime/src/main/java/com/datadoghq/reggie/runtime/StateSetKey.java b/reggie-runtime/src/main/java/com/datadoghq/reggie/runtime/StateSetKey.java new file mode 100644 index 0000000..85a1373 --- /dev/null +++ b/reggie-runtime/src/main/java/com/datadoghq/reggie/runtime/StateSetKey.java @@ -0,0 +1,43 @@ +/* + * Copyright 2026-Present Datadog, Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.datadoghq.reggie.runtime; + +import java.util.Arrays; + +final class StateSetKey { + private final int[] states; + private final int hash; + + StateSetKey(int[] sortedStates) { + this.states = sortedStates; + this.hash = Arrays.hashCode(sortedStates); + } + + int[] getStates() { + return states; + } + + @Override + public boolean equals(Object o) { + if (!(o instanceof StateSetKey)) return false; + return Arrays.equals(states, ((StateSetKey) o).states); + } + + @Override + public int hashCode() { + return hash; + } +} diff --git a/reggie-runtime/src/test/java/com/datadoghq/reggie/runtime/LazyDFABytecodeGeneratorTest.java b/reggie-runtime/src/test/java/com/datadoghq/reggie/runtime/LazyDFABytecodeGeneratorTest.java new file mode 100644 index 0000000..fdc9b3b --- /dev/null +++ b/reggie-runtime/src/test/java/com/datadoghq/reggie/runtime/LazyDFABytecodeGeneratorTest.java @@ -0,0 +1,127 @@ +/* + * Copyright 2026-Present Datadog, Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.datadoghq.reggie.runtime; + +import static org.junit.jupiter.api.Assertions.*; + +import java.lang.reflect.Field; +import java.lang.reflect.Method; +import java.util.Random; +import java.util.regex.Pattern; +import org.junit.jupiter.api.Test; + +class LazyDFABytecodeGeneratorTest { + + private static final String LARGE_NFA_PATTERN = "(?:a+b+|b+a+){75}"; + private static final String MATCH_INPUT = "ab".repeat(75); // 150 chars, accepted + + @Test + void testGeneratedClassMatchesNFAForSameInputs() { + ReggieMatcher lazyMatcher = RuntimeCompiler.compile(LARGE_NFA_PATTERN); + Pattern jdk = Pattern.compile(LARGE_NFA_PATTERN); + + // Deterministic positive case — exercises the accept path in the generated class. + String positive = "ab".repeat(75); + assertTrue(jdk.matcher(positive).matches(), "JDK must accept the positive input"); + assertTrue(lazyMatcher.matches(positive), "LAZY_DFA must accept the positive input"); + + // Random corpus — exercises reject paths and correctness across diverse inputs. + Random rng = new Random(42); + String alphabet = "abcdefghijklmnopqrstuvwxyz0123456789"; + for (int i = 0; i < 500; i++) { + int len = rng.nextInt(800); + StringBuilder sb = new StringBuilder(len); + for (int j = 0; j < len; j++) sb.append(alphabet.charAt(rng.nextInt(alphabet.length()))); + String s = sb.toString(); + boolean expected = jdk.matcher(s).matches(); + boolean actual = lazyMatcher.matches(s); + assertEquals(expected, actual, "Mismatch for: " + s.substring(0, Math.min(s.length(), 40))); + } + } + + @Test + void testNfaStepMethodPresent() throws Exception { + ReggieMatcher m = RuntimeCompiler.compile(LARGE_NFA_PATTERN); + Method nfaStep = m.getClass().getDeclaredMethod("nfaStep", int[].class, int.class); + assertNotNull(nfaStep); + } + + @Test + void testCacheIsSharedAcrossInstances() throws Exception { + RuntimeCompiler.clearCache(); + ReggieMatcher m1 = RuntimeCompiler.compile(LARGE_NFA_PATTERN); + // Use a distinct cache key to force a new ReggieMatcher instance while reusing the same + // generated class from the level-2 structural cache, giving two distinct objects. + ReggieMatcher m2 = RuntimeCompiler.cached("alt-key-shared-cache-test", LARGE_NFA_PATTERN); + assertNotSame(m1, m2); // different instances + assertSame(m1.getClass(), m2.getClass()); // same generated class + // Verify the static CACHE field is the same object across both instances. + Field f1 = m1.getClass().getDeclaredField("CACHE"); + Field f2 = m2.getClass().getDeclaredField("CACHE"); + f1.setAccessible(true); + f2.setAccessible(true); + assertSame(f1.get(null), f2.get(null)); + } + + @Test + void testCacheIsNotSharedAcrossPatterns() throws Exception { + RuntimeCompiler.clearCache(); + ReggieMatcher m1 = RuntimeCompiler.compile("(?:a+b+|b+a+){75}"); + ReggieMatcher m2 = RuntimeCompiler.compile("(?:a+b+|b+a+){76}"); + Field f1 = m1.getClass().getDeclaredField("CACHE"); + Field f2 = m2.getClass().getDeclaredField("CACHE"); + f1.setAccessible(true); + f2.setAccessible(true); + assertNotSame(f1.get(null), f2.get(null)); + } + + @Test + void testMatchMethod() { + ReggieMatcher m = RuntimeCompiler.compile(LARGE_NFA_PATTERN); + MatchResult r = m.match(MATCH_INPUT); + assertNotNull(r, "match() must return non-null for a full-input accept"); + assertEquals(0, r.start(0)); + assertEquals(150, r.end(0)); + assertNull(m.match("ab".repeat(74)), "match() must return null for a non-matching input"); + } + + @Test + void testMatchBoundedMethod() { + ReggieMatcher m = RuntimeCompiler.compile(LARGE_NFA_PATTERN); + // input = "xx" + "ab"*75, substring [2, 152) is the ab-repeat portion + String input = "xx" + MATCH_INPUT; + MatchResult r = m.matchBounded(input, 2, 152); + assertNotNull(r, "matchBounded() must return non-null when bounded region matches"); + assertEquals(2, r.start(0)); + assertEquals(152, r.end(0)); + // Region [0, 152) starts with "xx" — does not match the pattern + assertNull( + m.matchBounded(input, 0, 152), + "matchBounded() must return null when region does not match"); + } + + @Test + void testFindMatchFromMethod() { + ReggieMatcher m = RuntimeCompiler.compile(LARGE_NFA_PATTERN); + // embed the match at offset 2 + String input = "xx" + MATCH_INPUT + "yy"; + MatchResult r = m.findMatchFrom(input, 0); + assertNotNull(r, "findMatchFrom() must find the ab-repeat substring"); + assertEquals(2, r.start(0)); + assertEquals(152, r.end(0)); + assertNull(m.findMatchFrom("xxxx", 0), "findMatchFrom() must return null when no match exists"); + } +} diff --git a/reggie-runtime/src/test/java/com/datadoghq/reggie/runtime/LazyDFACacheTest.java b/reggie-runtime/src/test/java/com/datadoghq/reggie/runtime/LazyDFACacheTest.java new file mode 100644 index 0000000..8d7a1d6 --- /dev/null +++ b/reggie-runtime/src/test/java/com/datadoghq/reggie/runtime/LazyDFACacheTest.java @@ -0,0 +1,246 @@ +/* + * Copyright 2026-Present Datadog, Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.datadoghq.reggie.runtime; + +import static org.junit.jupiter.api.Assertions.*; + +import java.lang.reflect.Field; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.atomic.AtomicInteger; +import java.util.concurrent.atomic.AtomicReference; +import org.junit.jupiter.api.Test; + +class LazyDFACacheTest { + + // Minimal NfaStep: state {0} +'a'→ {1}, state {1} +'b'→ {2}, anything else → dead + private static final NfaStep TWO_STEP = + (states, c) -> { + if (states.length == 1 && states[0] == 0 && c == 'a') return new int[] {1}; + if (states.length == 1 && states[0] == 1 && c == 'b') return new int[] {2}; + return new int[0]; + }; + + @Test + void testCacheMissInterns() { + LazyDFACache cache = new LazyDFACache(new int[] {0}, new int[] {2}); + assertTrue(cache.matches("ab", TWO_STEP)); + assertFalse(cache.matches("a", TWO_STEP)); + assertFalse(cache.matches("abc", TWO_STEP)); + } + + @Test + void testCacheHitUsesAsciiTable() { + AtomicInteger callCount = new AtomicInteger(); + NfaStep counting = + (states, c) -> { + callCount.incrementAndGet(); + return TWO_STEP.apply(states, c); + }; + LazyDFACache cache = new LazyDFACache(new int[] {0}, new int[] {2}); + cache.matches("ab", counting); // cold: step called twice + int coldCalls = callCount.getAndSet(0); + assertEquals(2, coldCalls); + + cache.matches("ab", counting); // warm: ASCII hit, step NOT called + assertEquals(0, callCount.get()); + } + + @Test + void testDeadStateEarlyExit() { + AtomicInteger callCount = new AtomicInteger(); + NfaStep dead = + (states, c) -> { + callCount.incrementAndGet(); + return new int[0]; + }; + LazyDFACache cache = new LazyDFACache(new int[] {0}, new int[] {1}); + assertFalse(cache.matches("abc", dead)); + assertEquals(1, callCount.get()); // stops after first dead step + } + + @Test + void testFreezeAtCap() { + int cap = 3; + NfaStep gen = + (states, c) -> { + if (states.length == 1 && c == 'a') return new int[] {states[0] + 1}; + return new int[0]; + }; + LazyDFACache cache = new LazyDFACache(new int[] {0}, new int[] {999}, cap); + assertFalse(cache.matches("aaa", gen)); + assertTrue(cache.isFrozen()); + assertFalse(cache.matches("aaa", gen)); // still works after freeze + } + + @Test + void testFallbackMatchCorrect() { + int cap = 1; // freeze immediately after start state + LazyDFACache cache = new LazyDFACache(new int[] {0}, new int[] {2}, cap); + assertTrue(cache.matches("ab", TWO_STEP)); + assertFalse(cache.matches("a", TWO_STEP)); + assertTrue(cache.isFrozen()); + } + + @Test + void testAcceptStateRecognition() { + LazyDFACache cache = new LazyDFACache(new int[] {0}, new int[] {0}); + assertTrue(cache.matches("", TWO_STEP)); + LazyDFACache cache2 = new LazyDFACache(new int[] {0}, new int[] {99}); + assertFalse(cache2.matches("", TWO_STEP)); + } + + @Test + void testNonAsciiCharFallsBackToNfaStep() { + AtomicInteger callCount = new AtomicInteger(); + NfaStep tracker = + (states, c) -> { + callCount.incrementAndGet(); + return new int[0]; + }; + LazyDFACache cache = new LazyDFACache(new int[] {0}, new int[] {1}); + cache.matches("a", tracker); + callCount.set(0); + cache.matches("Ā", tracker); // c >= 128 + assertEquals(1, callCount.get()); + } + + // ── matchesBounded coverage ──────────────────────────────────────────────── + + @Test + void testMatchesBoundedHappyPath() { + // "xxab" — bounded region [2,4) matches "ab" + LazyDFACache cache = new LazyDFACache(new int[] {0}, new int[] {2}); + String input = "xx" + "ab"; + assertTrue(cache.matchesBounded(input, 2, 4, TWO_STEP)); + // Parity: same result as matches on the substring + assertEquals(cache.matches("ab", TWO_STEP), cache.matchesBounded(input, 2, 4, TWO_STEP)); + } + + @Test + void testMatchesBoundedEmptyRegion() { + // start == end → empty region; accepting only if start state itself accepts + LazyDFACache cache = new LazyDFACache(new int[] {0}, new int[] {2}); + assertFalse(cache.matchesBounded("abc", 1, 1, TWO_STEP)); // start state not accepting + LazyDFACache accepting = new LazyDFACache(new int[] {0}, new int[] {0}); + assertTrue(accepting.matchesBounded("abc", 1, 1, TWO_STEP)); // start state is accepting + } + + @Test + void testMatchesBoundedFrozenFallback() { + // cap=1 freezes immediately; nfaFallbackMatchBounded must respect end boundary + int cap = 1; + LazyDFACache cache = new LazyDFACache(new int[] {0}, new int[] {2}, cap); + String input = "xxab"; + // bounded region [2,4) = "ab" — should match via fallback + assertTrue(cache.matchesBounded(input, 2, 4, TWO_STEP)); + // bounded region [2,3) = "a" only — should not match + assertFalse(cache.matchesBounded(input, 2, 3, TWO_STEP)); + assertTrue(cache.isFrozen()); + } + + @Test + void testMatchesBoundedNonAsciiBoundaryAtCode128() { + // c == 128 (U+0080) must bypass the ASCII table (c >= 128 guard) + AtomicInteger callCount = new AtomicInteger(); + NfaStep tracker = + (states, c) -> { + callCount.incrementAndGet(); + return new int[0]; + }; + LazyDFACache cache = new LazyDFACache(new int[] {0}, new int[] {1}); + String withCode128 = "\u0080"; // code point 128, exactly the boundary + cache.matchesBounded(withCode128, 0, 1, tracker); + // Must have called nfaStep (not the ASCII table) for c == 128 + assertEquals(1, callCount.get()); + } + + @Test + void testCacheEntryNonAsciiBoundaryAtCode128() { + // cacheEntry guard: c >= 128 must skip the ASCII table for c == 128 + AtomicInteger callCount = new AtomicInteger(); + NfaStep tracker = + (states, c) -> { + callCount.incrementAndGet(); + return new int[0]; + }; + LazyDFACache cache = new LazyDFACache(new int[] {0}, new int[] {1}); + String withCode128 = "\u0080"; + cache.matches(withCode128, tracker); // first call — must not cache entry + callCount.set(0); + cache.matches(withCode128, tracker); // second call — must still invoke nfaStep (not cached) + assertEquals(1, callCount.get()); + } + + // ── sentinel constant parity ─────────────────────────────────────────────── + + @Test + void testSentinelConstantsParity() throws Exception { + // UNCACHED, DEAD, FALLBACK must match the values embedded in generated bytecode. + // LazyDFABytecodeGenerator keeps local copies (different module); this test detects drift. + Class generatorClass = + Class.forName("com.datadoghq.reggie.codegen.codegen.LazyDFABytecodeGenerator"); + + Field genUncached = generatorClass.getDeclaredField("UNCACHED"); + Field genDead = generatorClass.getDeclaredField("DEAD"); + Field genFallback = generatorClass.getDeclaredField("FALLBACK"); + genUncached.setAccessible(true); + genDead.setAccessible(true); + genFallback.setAccessible(true); + + assertEquals(LazyDFACache.UNCACHED, genUncached.get(null), "UNCACHED sentinel mismatch"); + assertEquals(LazyDFACache.DEAD, genDead.get(null), "DEAD sentinel mismatch"); + assertEquals(LazyDFACache.FALLBACK, genFallback.get(null), "FALLBACK sentinel mismatch"); + } + + @Test + void testConcurrentInterning() throws Exception { + LazyDFACache cache = new LazyDFACache(new int[] {0}, new int[] {1}); + CountDownLatch ready = new CountDownLatch(2); + CountDownLatch go = new CountDownLatch(1); + AtomicReference r1 = new AtomicReference<>(), r2 = new AtomicReference<>(); + + Thread t1 = + new Thread( + () -> { + ready.countDown(); + try { + go.await(); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + } + r1.set(cache.matches("a", TWO_STEP)); + }); + Thread t2 = + new Thread( + () -> { + ready.countDown(); + try { + go.await(); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + } + r2.set(cache.matches("a", TWO_STEP)); + }); + t1.start(); + t2.start(); + ready.await(); + go.countDown(); + t1.join(); + t2.join(); + assertTrue(r1.get()); + assertTrue(r2.get()); + } +} diff --git a/reggie-runtime/src/test/java/com/datadoghq/reggie/runtime/StateSetKeyTest.java b/reggie-runtime/src/test/java/com/datadoghq/reggie/runtime/StateSetKeyTest.java new file mode 100644 index 0000000..ca96fd2 --- /dev/null +++ b/reggie-runtime/src/test/java/com/datadoghq/reggie/runtime/StateSetKeyTest.java @@ -0,0 +1,60 @@ +/* + * Copyright 2026-Present Datadog, Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.datadoghq.reggie.runtime; + +import static org.junit.jupiter.api.Assertions.*; + +import org.junit.jupiter.api.Test; + +class StateSetKeyTest { + + @Test + void testEqualKeysForSameContents() { + StateSetKey a = new StateSetKey(new int[] {1, 3, 5}); + StateSetKey b = new StateSetKey(new int[] {1, 3, 5}); + assertEquals(a, b); + assertEquals(a.hashCode(), b.hashCode()); + } + + @Test + void testNotEqualForDifferentContents() { + StateSetKey a = new StateSetKey(new int[] {1, 3, 5}); + StateSetKey b = new StateSetKey(new int[] {1, 3, 6}); + assertNotEquals(a, b); + } + + @Test + void testNotEqualForDifferentLength() { + StateSetKey a = new StateSetKey(new int[] {1, 3}); + StateSetKey b = new StateSetKey(new int[] {1, 3, 5}); + assertNotEquals(a, b); + } + + @Test + void testEmptyKey() { + StateSetKey a = new StateSetKey(new int[] {}); + StateSetKey b = new StateSetKey(new int[] {}); + assertEquals(a, b); + assertEquals(a.hashCode(), b.hashCode()); + } + + @Test + void testGetStatesReturnsArray() { + int[] data = {2, 4, 6}; + StateSetKey key = new StateSetKey(data); + assertArrayEquals(data, key.getStates()); + } +}