Release 3.3.0

Summary

Release 3.3.0 focuses on substantially improving the performance, memory use, diagnostics, and benchmark accuracy of the interpreted ALL(*) parser used by trparse.

The release establishes repeatable parser-only performance tests, instruments prediction and runtime stages, reuses parser state across files, replaces allocation-heavy parser data structures, and builds parsing results in compact storage. It also corrects interpreted token reporting and makes OphiRust benchmark timing include its eager lexing work.

ALL(*) parser performance

• Added parser-only DOT benchmarks, including the large grammars-v4 1864.dot input.
• Added five-sample timing reports with mean, standard deviation, and SEM.
• Added --no-output so parsing and tree construction can be measured without serializing output.
• Added --parser-stats diagnostics for prediction calls, DFA activity, lookahead, closure work, prediction contexts, conflicts, full-context fallback, committed ATN walking, parse events, and retained cache storage.
• Shared learned parser DFAs across inputs in one command, with configurable state/memory budgets and an option to disable sharing.
• Added a conservative LL(1) bypass implementation and supporting analysis; the production bypass remains disabled because measurements did not show a reliable improvement and nullable-loop expansion caused correctness regressions.
• Interned prediction contexts in a grammar-scoped arena using compact integer IDs.
• Replaced recursive SortedDictionary context merging with reusable linear-merge workspaces and prediction-local symmetric pair memoization.
• Optimized committed ATN walking with precomputed metadata and reduced repeated state/transition inspection.
• Cached parsed/deserialized parser and lexer .interp runtime data across all files supplied to a single trparse command. This applies to ALL(*), Earley, -x, --xf, and multiple positional inputs.

Compact parsing results

• Added CompactParseTree, which stores node kinds, relationships, token references, and rule metadata in compact indexed arrays.
• Builds interpreted parse trees directly in compact storage instead of constructing the former allocation-heavy DOM representation.
• Preserves the existing parsing-result interfaces and serialized output.
• Added TokenStore, a structure-of-arrays representation for interpreted lexer tokens.
• Changed LexerToken to an indexed value-type view while preserving standalone-token compatibility.
• Keeps token text source-backed and serializes text directly from source spans, avoiding a string allocation per token.
• Updated ALL(*), Earley, and tree construction paths to consume the compact token store without copying token objects.

On the large DOT workload, compact tree construction and token storage produced the largest measured improvement in this release. One measured 1864.dot run improved from 2.656 seconds/312,894 TPS to 2.104 seconds/394,883 TPS, while a subsequent corpus comparison measured AllStar at approximately 308,221 TPS versus approximately 123,000 TPS before compact tree storage.

Correctness and reporting

• Interpreted ALL(*) and Earley runners now report the complete retained token stream, including hidden-channel tokens and EOF, matching generated-target CommonTokenStream.Size reporting.
• Added a DOT cluster.dot regression test verifying the expected 171-token count.
• Parsing-result serialization now supports compact node providers.
• Added coverage for compact tree behavior, token storage, prediction-context canonicalization/merging, LL(1) analysis, runtime reuse, parser diagnostics, and output suppression.

Benchmark fairness

OphiRust CommonTokenStream::new(lexer) eagerly fills the token buffer. Its generated template previously started the parser timer afterward, excluding lexing from PT while targets with lazy token streams included lexing during the start-rule call.

The OphiRust template now measures eager token-buffer construction and adds that duration to PT/TPS. Parser construction and listener setup remain outside the measured interval.

Validation

• Release builds complete with zero warnings and errors.
• The broad applicable AllStar non-performance regression suite passes (282 tests in the final token-count verification run).
• The exact DOT cluster.dot command now reports 171 tokens for ALL(*), matching the generated targets.
• Tag: 3.3.0

Tracking issue: #713.

---

Related: PR #714, issue #713.