Skip to content

Release 4.4.0

Choose a tag to compare

@kaby76 kaby76 released this 17 Oct 10:30

Summary

  • Bump Trash tools and packages to 4.4.0, including release scripts and the local tool manifest.
  • Restore fast DFA-backed parsing for XQuery-hook workloads: parser-only hooks keep eager lexing and the lexer DFA; predicate-independent SLL closures use the parser DFA and fall back to full LL when needed. Parsed hook expressions are cached across files while mutable hook state remains per input.
  • Stream the compressed JDK archive directly into the Java AllStar/XQuery example instead of creating an intermediate tar.
  • Simplify G4X by removing ANTLR-style element and alternative labels, with corresponding parser, interpreter, compatibility-test, and documentation updates.
  • Add an external, ANTLR3-inspired .ast rewrite notation and a post-CST arithmetic prototype, then implement trparse --allstar --ast-map to build a pipeable UnvParseTreeNode AST during parsing rather than constructing a CST first. The normal PAX/tar .pt output remains consumable by Trash tree and XQuery commands.
  • Add a LINVAST-inspired example using action-free C, Java, Kotlin, and Lua grammars with separate mapping files. Expand the mappings to cover representative declarations, functions, classes, expressions, branches, loops, and other language constructs. Extend the mapping language with optional/repeated captures and multi-node text captures, and add checked tree fixtures and tests.

Validation

  • dotnet test -c Release tests/AllStarParserTests/AllStarParserTests.csproj --no-restore: 571 passed.
  • bash examples/ast-during-parse/run-example.sh and bash examples/linvast-ast/run-example.sh passed, including exact AST tree comparisons and XQuery consumption of the resulting parse bundles.
  • On JDK Destinations.java (2,719 tokens), the original parser-only hook path took about 225 s. After the DFA/cache changes, a three-file run took 0.40 s for the first hooked file and 0.089/0.069 s for the next two, near the unhooked 0.21/0.086/0.067 s. Full-LL fallbacks with hooks dropped from 4,309 to 277.

Current scope

The AST mapping is a proof of concept, not a complete LINVAST implementation. --ast-map currently targets ordinary pre-tokenized ALL(*) parsing and does not combine with indirect left recursion, context-aware lexing, or runtime XQuery hooks. The Kotlin demo uses a limited ASCII-only lexer because the full upstream Unicode lexer stalls in the current interpreter; its source files retain the original for reference.