Skip to content

Release 0.6.1#35

Merged
siy merged 14 commits into
mainfrom
release-0.6.1
May 14, 2026
Merged

Release 0.6.1#35
siy merged 14 commits into
mainfrom
release-0.6.1

Conversation

@siy

@siy siy commented May 14, 2026

Copy link
Copy Markdown
Owner

Summary

  • AKIND_DOC_LINE_COMMENT (3) and KIND_DOC_BLOCK_COMMENT (4) trivia kinds; FIRST_USER_KIND shifted to 5; /// and /** post-DFA classification in lexer engine and generator
  • B — Per-rule %recover sync sets: SYNC_<Rule> arrays emitted per rule; lastFailedRuleKind field routes recovery to rule-specific sync set
  • C%checkpoint RuleName grammar directive parsed and stored in Grammar.checkpointRules(); IncrementalParser consumes it; canonical java25.peg declares Stmt, MethodDecl, TypeDecl
  • D — Named captures ($name<expr>) and back-references ($name) runtime via source-span equality; NamedCaptureDetector rejection removed
  • E — MIXED-rule char-level fallback: CharClass and Any in MIXED rules emit token-level proxy via input.charAt(tokens.startAt(pos))
  • FJava25SelfHostDiagTest gate added; shift-in-FieldDecl bug identified and deferred to 0.6.2 (2 assertions @Disabled)
  • GmaxDiagnostics cap honored in generated recovery loop; parse(String, int) wired end-to-end
  • J — README rewritten (0.6.x-only, concrete examples); docs/VISITOR-TUTORIAL.md added (489-line calculator walkthrough)
  • BonusTriviaPostPass first-member trivia loss fixed (0.5.x legacy); lexer empty-match warning softened; canonical java25.peg %whitespace split for per-kind trivia

Test plan

  • mvn install -Djbct.skip=true — 1440 tests, 0 failures
  • Java25ParserGateTest — 20/20 fixtures clean parse
  • FactoryClassGeneratorDiagTest — 0 diagnostics
  • V6FormatterCorpusGateTest — 20/20 round-trip
  • Java25SelfHostDiagTest — dumps diagnostics; 2 assertions disabled pending 0.6.2 shift-in-FieldDecl fix

Known limitations carried to 0.6.2

  • Shift operators in field/local-var init context fail at CompilationUnit level (shift-in-FieldDecl bug)
  • Per-iteration %whitespace tokenization (ZeroOrMore loop emission) deferred

Summary by CodeRabbit

  • New Features

    • Named captures and back-references now supported at runtime.
    • Incremental parsing via %checkpoint directives for selective re-parsing.
    • Per-rule recovery via %recover [chars] RuleName directives.
    • Diagnostic capping via parse(input, maxDiagnostics) overload.
    • Doc comment classification (/// and /** ... */).
  • Bug Fixes

    • Fixed trivia loss for first child after opening delimiters.
  • Documentation

    • Added visitor pattern tutorial for CST transformation.
    • Updated README and grammar DSL reference for v0.6.x.

@coderabbitai

coderabbitai Bot commented May 14, 2026

Copy link
Copy Markdown
📝 Walkthrough

Walkthrough

Peglib v0.6.1 completes runtime support for per-rule %recover and %checkpoint directives, restores named-capture and back-reference semantics with span equality, adds doc-comment trivia classification (doc line/block), implements MIXED-rule char-level fallback with RuleKind propagation through emit, and honors maxDiagnostics caps in Parser.parse. It also fixes first-member trivia loss in TriviaPostPass and introduces a visitor-pattern tutorial with new test coverage and grammar fixture updates.


Changes

Documentation & Release

Layer / File(s) Summary
Release notes and migration guide
CHANGELOG.md, docs/HANDOVER.md
v0.6.1 entry documents all features/fixes/known limitations for this patch release and handover checkpoint updated to reflect session 6 completion.
Architecture and tutorial documentation
README.md, docs/VISITOR-TUTORIAL.md, docs/GRAMMAR-DSL.md, docs/bugs/first-member-trivia-loss-2026-05-12.md
README substantially rewritten for v0.6.x audience with tokens-first CST, visitor transforms, and grammar directives; new visitor-pattern tutorial walks end-to-end calculator example; GRAMMAR-DSL documents %whitespace shape semantics for trivia; bug documentation details first-member trivia-loss investigation and fix strategy.
Version updates across all modules
pom.xml, peglib-*/pom.xml
All Maven module versions bumped from 0.6.0 to 0.6.1.

Checkpoint Rules Grammar & Incremental Integration

Layer / File(s) Summary
Checkpoint rule declaration & parsing
peglib-core/src/main/java/org/pragmatica/peg/grammar/Grammar.java, peglib-core/src/main/java/org/pragmatica/peg/grammar/GrammarParser.java
Grammar record gains checkpointRules: Set<String> component with defensive copy; GrammarParser.parseGrammar() adds %checkpoint RuleName directive parsing to accumulate checkpoint rule names and thread them through Grammar construction.
Checkpoint import composition & incremental preference
peglib-core/src/main/java/org/pragmatica/peg/grammar/GrammarResolver.java, peglib-core/src/main/java/org/pragmatica/peg/v6/incremental/IncrementalParser.java
GrammarResolver unions checkpoint sets from root and imported grammars; IncrementalParser 2-arg constructor now prefers grammar-declared checkpointRules() over hardcoded defaults when non-empty.
Checkpoint directive grammar fixtures & tests
peglib-core/src/test/resources/java25.peg, peglib-core/src/test/java/org/pragmatica/peg/grammar/CheckpointDirectiveTest.java, peglib-incremental/src/test/java/org/pragmatica/peg/v6/incremental/CheckpointDirectiveIncrementalTest.java
java25.peg adds %checkpoint Stmt MethodDecl TypeDecl declarations; comprehensive test suites verify directive parsing, deduplication, coexistence with other directives, and incremental constructor preference logic.

Doc-Comment Trivia Classification (Kind & Lexer)

Layer / File(s) Summary
Doc trivia kind constants & token metadata
peglib-runtime/src/main/java/org/pragmatica/peg/v6/token/TokenArray.java, peglib-core/src/main/java/org/pragmatica/peg/v6/lexer/DfaBuilder.java
New public constants KIND_DOC_LINE_COMMENT (3) and KIND_DOC_BLOCK_COMMENT (4) added; FIRST_USER_KIND shifted from 3 to 5; isTrivia(int) and isTriviaKind(int) updated to classify doc comments as trivia.
Lexer trivia reclassification for doc patterns
peglib-core/src/main/java/org/pragmatica/peg/v6/lexer/LexerEngine.java, peglib-core/src/main/java/org/pragmatica/peg/v6/generator/LexerGenerator.java
LexerEngine.lex Phase A.6 now distinguishes // vs /// (line vs doc-line) and /* vs /** (block vs doc-block) with special handling to exclude /**/ from doc classification; generated lexer mirrors this logic with inline trivia-class detection helpers.
Formatter trivia rendering & test coverage
peglib-formatter/src/main/java/org/pragmatica/peg/formatter/v6/V6TriviaPolicy.java, peglib-core/src/test/java/org/pragmatica/peg/v6/lexer/TriviaClassificationTest.java, peglib-runtime/src/test/java/org/pragmatica/peg/v6/cst/CstArrayTest.java, peglib-runtime/src/test/java/org/pragmatica/peg/v6/token/TokenArrayTest.java
V6TriviaPolicy routing extended for doc comment kinds; TriviaClassificationTest adds fixtures for //////// and /**//***//**/ patterns with isTrivia checks; token array test fixtures updated to reserve and name the new doc kinds.

First-Member Trivia Fix & TriviaPostPass

Layer / File(s) Summary
Trivia scan cursor advancement for opening delimiters
peglib-core/src/main/java/org/pragmatica/peg/tree/TriviaPostPass.java
rebuildNonTerminal now advances the trivia scan cursor past non-trivia-startable characters (like {, (, [) before running whitespace scanning, preventing trivia drop before the first child; added canStartTrivia(char) predicate to identify trivia-startable characters.
Trivia preservation regression tests
peglib-core/src/test/java/org/pragmatica/peg/tree/FirstMemberTriviaTest.java, peglib-core/src/test/java/org/pragmatica/peg/v6/cst/FirstMemberV6TriviaTest.java
FirstMemberTriviaTest validates direct CST leadingTrivia() on first members after delimiters for line/doc-line comments; FirstMemberV6TriviaTest verifies v6 CST generation preserves the same trivia for both // and /// patterns.

Per-Rule Recovery with RuleKind Dispatch

Layer / File(s) Summary
Parser-level recovery state & sync-set dispatch
peglib-core/src/main/java/org/pragmatica/peg/v6/generator/ParserGenerator.java
Generated parser now maintains lastFailedRuleKind and emits per-rule SYNC_<RuleName> arrays for %recover [chars] RuleName directives; fail(expected, ruleKind) method wires ruleKind into recovery routing; syncForRule(int ruleKind) dispatches to the appropriate sync set or DEFAULT_SYNC.
Emit context & expression-level RuleKind propagation
peglib-core/src/main/java/org/pragmatica/peg/v6/generator/ParserGenerator.java
EmitContext gains RuleKind ruleKind field; all expression emission sites (literals, aliases, identifiers, choices, predicates) updated to call fail(..., ruleKindConst(ctx)) for consistent per-rule recovery routing; diagnostic emission guarded by maxDiagnostics cap.
Per-rule recovery directive test & generated code inspection
peglib-core/src/test/java/org/pragmatica/peg/v6/generator/PerRuleRecoverDirectiveTest.java
Validates generated parser source includes per-rule SYNC_<RuleName> arrays and syncForRule dispatch logic; runtime tests confirm statement- and block-level recovery correctness via diagnostics, error presence, and CST node counts; also verifies collapse to DEFAULT_SYNC when no per-rule sets exist.

MIXED-Rule Char-Level Fallback & Expression Emission

Layer / File(s) Summary
CharClass token-level proxy & predicate handling in MIXED rules
peglib-core/src/main/java/org/pragmatica/peg/v6/generator/ParserGenerator.java
emitAndDispatch / emitNotDispatch for CharClass expressions inside MIXED rules now emit token-level byte-peeking proxies rather than character-level matching; predicates (&[...], ![...]) on char-level expressions are kept as parse-time no-ops when appropriate; helper renderCharClassMembership expands ranges and handles case-insensitivity.
MIXED-rule char-level fallback test suite
peglib-core/src/test/java/org/pragmatica/peg/v6/generator/MixedRuleFallbackTest.java
Comprehensive tests cover char-class consumption after lexer reference, mismatch diagnostics, . (Any) token consumption, choice composition with mixed alternatives, negated char classes, and predicate no-op behavior to prevent token rejection.

Diagnostic Cap (maxDiagnostics) Support

Layer / File(s) Summary
Parser API & capped-parse method reflection
peglib-core/src/main/java/org/pragmatica/peg/v6/Parser.java, peglib-core/src/main/java/org/pragmatica/peg/v6/generator/ParserCompiler.java
Parser.parse(String, int maxDiagnostics) overload now forwards cap through lexing to compiled parser; ParserCompiler.CompiledParser gains parseCappedMethod field and new parse(TokenArray, int maxDiagnostics) overload that reflectively invokes generated capped-parse; loader updated to register both parse(TokenArray) and parse(TokenArray, int) methods.
Generated parser diagnostic-capped iteration & recovery
peglib-core/src/main/java/org/pragmatica/peg/v6/generator/ParserGenerator.java
Constructor normalizes maxDiagnostics (negative => uncapped); parse(TokenArray, int maxDiagnostics) public overload added; start-rule iteration checks diagnostics.size() >= maxDiagnostics to break early; emitRecoveryError and emitForcedAdvanceError guard insertion with cap check.
Diagnostic capping test suite
peglib-core/src/test/java/org/pragmatica/peg/v6/generator/MaxDiagnosticsTest.java
Tests verify baseline uncapped behavior, exact cap enforcement (at 1, 5, and 0), and behavior when cap exceeds actual diagnostics; validates that CST builds even when cap is zero.

Named Capture & Back-Reference Runtime Support

Layer / File(s) Summary
Removal of compile-time named-capture detection
peglib-core/src/main/java/org/pragmatica/peg/v6/PegParser.java, peglib-core/src/main/java/org/pragmatica/peg/v6/analyzer/NamedCaptureDetector.java, peglib-core/src/main/java/org/pragmatica/peg/v6/analyzer/NamedCaptureCause.java
PegParser.fromGrammar no longer runs checkNamedCaptures validation; NamedCaptureDetector class and related types (Occurrence, Kind, DetectionResult) removed; NamedCaptureCause record removed; runtime replaces compile-time checking.
Named capture runtime codegen & span semantics
peglib-core/src/main/java/org/pragmatica/peg/v6/generator/ParserGenerator.java
Generated parser gains captures map and captureScopeStack for runtime span/scope management; Capture records source spans; CaptureScope snapshots/restores captures across scope boundaries; BackReference compares captured substring with current input, advancing on match and failing on mismatch; all capture operations routed through fail(expected, ruleKind) for consistent recovery.
Named capture runtime behavior test suite
peglib-core/src/test/java/org/pragmatica/peg/v6/generator/NamedCaptureRuntimeTest.java
Tests validate opening/closing tag matching, empty captures, missing-capture diagnostics, capture-scope isolation/rollback, and backtracking across choice alternatives with captures.

Test Infrastructure & Self-Host Diagnostics

Layer / File(s) Summary
Self-host grammar fixture updates & cache management
peglib-core/src/test/resources/java25.peg, peglib-core/src/main/java/org/pragmatica/peg/v6/PegParser.java
java25.peg %whitespace rule updated to one-or-more form and checkpoint declarations added; PegParser gains public cacheSize() and clearCache() helpers for test cache introspection.
Self-host diagnostic dump & bisect harness
peglib-core/src/test/java/org/pragmatica/peg/v6/diagnostic/Java25SelfHostDiagTest.java
dumpDiagnostics() reads PEG grammar and ~1.85MB Java fixture, parses with max error count, clusters diagnostics by signature, prints hit counts and contextual snippets; two additional disabled tests (selfHostFixtureParsesCleanly, selfHostFixtureProducesShallowCST) gate pending shift-operator bug fix with TODO cross-references.
Grammar fixture constructor updates
peglib-core/src/test/java/org/pragmatica/peg/v6/analyzer/LeftRecursionDetectorTest.java, peglib-core/src/test/java/org/pragmatica/peg/v6/lexer/RuleClassifierTest.java
Test grammar construction helpers updated to pass Set.of() and Map.of() into expanded Grammar constructor to match new checkpointRules/recoverSets signature.

Sequence Diagram(s)

sequenceDiagram
    participant Parser as Parser.parse
    participant Lexer as TokenArray.lex
    participant CompiledParser as Generated Parser
    participant Recovery as Recovery Logic
    participant Diagnostics as Diagnostics

    Parser->>Lexer: input + maxDiagnostics
    Lexer->>Lexer: classify doc comments<br/>in WHITESPACE tokens
    Lexer-->>Parser: TokenArray<br/>(with doc trivia kinds)
    
    Parser->>CompiledParser: parse(tokens, maxDiagnostics)
    CompiledParser->>CompiledParser: initialize<br/>captures, lastFailedRuleKind
    
    loop start-rule attempts
        CompiledParser->>CompiledParser: reset lastFailedRuleKind
        CompiledParser->>CompiledParser: parseStartRule(...)
        
        opt expression fails
            CompiledParser->>CompiledParser: fail(expected, ruleKind)<br/>→ lastFailedRuleKind=ruleKind
            CompiledParser->>Recovery: syncForRule(lastFailedRuleKind)
            Recovery-->>CompiledParser: SYNC_RuleName or DEFAULT_SYNC
            CompiledParser->>CompiledParser: advance to sync token
        end
        
        CompiledParser->>Diagnostics: emit diagnostic<br/>if size < maxDiagnostics
        
        alt diagnostics.size >= maxDiagnostics
            CompiledParser->>CompiledParser: break start-rule loop
        end
    end
    
    CompiledParser->>CompiledParser: build CST<br/>(with named-capture spans)
    CompiledParser-->>Parser: ParseResult<br/>(CST + diagnostics)
    Parser-->>Parser: return with diagnostic cap<br/>honored
Loading
sequenceDiagram
    participant Grammar as Grammar Parser
    participant GrammarResolver as GrammarResolver
    participant IncrementalParser as IncrementalParser
    participant Parser as Compiled Parser
    
    Grammar->>Grammar: parse %checkpoint directives<br/>→ checkpointRules set
    Grammar-->>Parser: Grammar(rules, checkpointRules)
    
    GrammarResolver->>GrammarResolver: load imported grammars
    GrammarResolver->>GrammarResolver: union checkpointRules<br/>from root + imports
    GrammarResolver-->>Parser: Grammar(..., unionedCheckpoints)
    
    Parser->>Parser: compile & cache
    
    IncrementalParser->>Parser: grammar()
    IncrementalParser->>IncrementalParser: checkpointRules<br/>non-empty?
    
    alt grammar declares checkpoints
        IncrementalParser->>IncrementalParser: use grammar.checkpointRules()
    else fallback
        IncrementalParser->>IncrementalParser: use DEFAULT_CHECKPOINT_RULES
    end
    
    IncrementalParser->>IncrementalParser: parse with checkpoint<br/>boundaries
Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~75 minutes

This release spans 8 distinct feature/fix layers with heterogeneous changes across grammar, lexer, parser codegen, runtime behavior, and test infrastructure. The changes include subtle control-flow updates (per-rule recovery dispatch, trivia cursor logic), new codegen patterns (MIXED-rule char-level fallback, named-capture spans), and extensive test coverage. While many changes are internally consistent within each layer, the overall scope and interaction density—especially between recovery routing, diagnostic capping, and expression emission—demand careful sequential review of the parser generator and runtime semantics.


Possibly related PRs

  • siy/java-peglib#27: Implements per-rule %recover directive wiring into generated parsers; main PR completes and extends per-rule recovery with rule-kind–aware dispatch semantics.
  • siy/java-peglib#26: Wires %recover directive interpretation and recovery landing behavior; main PR follows up with per-rule recovery dispatch and routing logic.
  • siy/java-peglib#34: Introduces v6 tokens-first architecture, flat CST, and incremental parsing foundations; main PR extends that baseline with doc-trivia kinds, checkpoint directives, and runtime named-capture support.

Poem

🐰 Fuzzy hops with glee—checkpoints marked, doc comments gleam,
Recovery routes now rule-aware, captures captured in a dream.
First members keep their trivia dear, no more loss in parse,
Diagnostics capped and gentle too; peglib's golden release!
v0.6.1 shines—hop, hop, hurrah!

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch release-0.6.1

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
pom.xml (1)

50-50: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Update pragmatica-lite.version to 0.9.0.

The root pom.xml specifies version 0.24.0, but coding guidelines require version 0.9.0 for the pragmatica-lite:core runtime dependency. Update the property to comply with the requirement for Result/Option/Promise types.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@pom.xml` at line 50, Update the pom property pragmatica-lite.version from
0.24.0 to 0.9.0 so the runtime dependency pragmatica-lite:core uses the required
0.9.0 release; locate the XML property element named <pragmatica-lite.version>
in the root POM and change its value to 0.9.0, then rebuild to ensure the
Result/Option/Promise types are sourced from the correct version.
🧹 Nitpick comments (2)
peglib-core/src/main/java/org/pragmatica/peg/tree/TriviaPostPass.java (1)

454-460: ⚖️ Poor tradeoff

Hardcoded trivia-start predicate may be incomplete.

The canStartTrivia method hardcodes whitespace and comment-start characters (' ', '\t', '\r', '\n', '/', '#'). This may not cover:

  1. Other whitespace: form feed ('\f'), vertical tab ('\u000B')
  2. Grammar-specific comment starters: Some languages use ;, --, or other prefixes
  3. Unicode whitespace: characters like non-breaking space (\u00A0), zero-width space, etc.

While the current set likely covers Java and most common languages, consider either:

  • Consulting the grammar's %whitespace rule to derive valid start characters
  • Or documenting this as a known limitation for exotic grammar patterns
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@peglib-core/src/main/java/org/pragmatica/peg/tree/TriviaPostPass.java` around
lines 454 - 460, The canStartTrivia predicate is too narrow; replace its
hardcoded checks with a Unicode-aware whitespace test and keep the common
comment-start chars—change canStartTrivia to return Character.isWhitespace(c) ||
c == '/' || c == '#', so form feed, vertical tab and other Unicode whitespace
are covered; update any callers (e.g., rebuildNonTerminal) expectations or add a
short comment noting grammar-specific comment markers may still need extending.
peglib-core/src/main/java/org/pragmatica/peg/v6/lexer/DfaBuilder.java (1)

53-60: ⚡ Quick win

Centralize reserved token-kind constants to a single source of truth.

These reserved kind IDs are duplicated here and in TokenArray; future edits can silently desync lexer emission vs runtime classification.

Proposed refactor
+import org.pragmatica.peg.v6.token.TokenArray;
...
-    public static final int KIND_WHITESPACE = 0;
-    public static final int KIND_LINE_COMMENT = 1;
-    public static final int KIND_BLOCK_COMMENT = 2;
-    public static final int KIND_DOC_LINE_COMMENT = 3;
-    public static final int KIND_DOC_BLOCK_COMMENT = 4;
-    public static final int FIRST_USER_KIND = 5;
+    public static final int KIND_WHITESPACE = TokenArray.KIND_WHITESPACE;
+    public static final int KIND_LINE_COMMENT = TokenArray.KIND_LINE_COMMENT;
+    public static final int KIND_BLOCK_COMMENT = TokenArray.KIND_BLOCK_COMMENT;
+    public static final int KIND_DOC_LINE_COMMENT = TokenArray.KIND_DOC_LINE_COMMENT;
+    public static final int KIND_DOC_BLOCK_COMMENT = TokenArray.KIND_DOC_BLOCK_COMMENT;
+    public static final int FIRST_USER_KIND = TokenArray.FIRST_USER_KIND;

Also applies to: 214-219

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@peglib-core/src/main/java/org/pragmatica/peg/v6/lexer/DfaBuilder.java` around
lines 53 - 60, The duplicated reserved token-kind constants (KIND_WHITESPACE,
KIND_LINE_COMMENT, KIND_BLOCK_COMMENT, KIND_DOC_LINE_COMMENT,
KIND_DOC_BLOCK_COMMENT, FIRST_USER_KIND) should be centralized into the single
source of truth in TokenArray and removed from DfaBuilder; replace the local
constant definitions in DfaBuilder with references to TokenArray (e.g. use
TokenArray.KIND_WHITESPACE, TokenArray.FIRST_USER_KIND, etc.), remove the
duplicate block at the second location (lines around the other duplicate
definitions), and update any local references in DfaBuilder to use the
TokenArray symbols so compilation and runtime classification stay in sync.
Ensure the TokenArray constants are public/static and import or fully-qualify
them where needed.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@peglib-core/src/main/java/org/pragmatica/peg/grammar/Grammar.java`:
- Around line 62-72: The canonical compact constructor for Grammar only
defensively copies checkpointRules but not recoverSets, allowing callers to
mutate the outer or inner sets after construction; update the Grammar
constructor to defensively copy recoverSets as well by creating an immutable
outer set (e.g., Set.copyOf) whose elements are immutable copies of each inner
set (e.g., map each inner Set to Set.copyOf(inner) before collecting), so both
recoverSets and its contained sets are unmodifiable at construction time.

In `@peglib-core/src/main/java/org/pragmatica/peg/grammar/GrammarParser.java`:
- Around line 107-117: The logic is incorrectly skipping
parseCheckpointDirective() when the token after a "checkpoint" directive isn't
an Identifier, causing generic parse errors; change the block that checks
"checkpoint".equals(directive.name()) so it always calls advance() and then
invoke parseCheckpointDirective() (use parseCheckpointDirective() result
handling as with Result.Failure<?> f to return f.cause().result()), and on
success add result.unwrap() to checkpointRules instead of falling through to the
later advance() path; adjust control flow around tokens, advance(),
parseCheckpointDirective(), checkpointRules and Result.Failure handling so all
%checkpoint directives go through parseCheckpointDirective().

In `@README.md`:
- Line 225: The fenced code block opening fence (``` ) in the README lacks a
language identifier, triggering markdownlint MD040; update the opening fence to
include a language tag (for example change ``` to ```text or another appropriate
language) so the block becomes a fenced code block with a language identifier.

---

Outside diff comments:
In `@pom.xml`:
- Line 50: Update the pom property pragmatica-lite.version from 0.24.0 to 0.9.0
so the runtime dependency pragmatica-lite:core uses the required 0.9.0 release;
locate the XML property element named <pragmatica-lite.version> in the root POM
and change its value to 0.9.0, then rebuild to ensure the Result/Option/Promise
types are sourced from the correct version.

---

Nitpick comments:
In `@peglib-core/src/main/java/org/pragmatica/peg/tree/TriviaPostPass.java`:
- Around line 454-460: The canStartTrivia predicate is too narrow; replace its
hardcoded checks with a Unicode-aware whitespace test and keep the common
comment-start chars—change canStartTrivia to return Character.isWhitespace(c) ||
c == '/' || c == '#', so form feed, vertical tab and other Unicode whitespace
are covered; update any callers (e.g., rebuildNonTerminal) expectations or add a
short comment noting grammar-specific comment markers may still need extending.

In `@peglib-core/src/main/java/org/pragmatica/peg/v6/lexer/DfaBuilder.java`:
- Around line 53-60: The duplicated reserved token-kind constants
(KIND_WHITESPACE, KIND_LINE_COMMENT, KIND_BLOCK_COMMENT, KIND_DOC_LINE_COMMENT,
KIND_DOC_BLOCK_COMMENT, FIRST_USER_KIND) should be centralized into the single
source of truth in TokenArray and removed from DfaBuilder; replace the local
constant definitions in DfaBuilder with references to TokenArray (e.g. use
TokenArray.KIND_WHITESPACE, TokenArray.FIRST_USER_KIND, etc.), remove the
duplicate block at the second location (lines around the other duplicate
definitions), and update any local references in DfaBuilder to use the
TokenArray symbols so compilation and runtime classification stay in sync.
Ensure the TokenArray constants are public/static and import or fully-qualify
them where needed.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 4a2e8cb4-4da9-439e-8d42-8a8d3e37d1dd

📥 Commits

Reviewing files that changed from the base of the PR and between 918768c and b10ed98.

📒 Files selected for processing (45)
  • CHANGELOG.md
  • README.md
  • docs/GRAMMAR-DSL.md
  • docs/HANDOVER.md
  • docs/VISITOR-TUTORIAL.md
  • docs/bugs/first-member-trivia-loss-2026-05-12.md
  • peglib-core/pom.xml
  • peglib-core/src/main/java/org/pragmatica/peg/grammar/Grammar.java
  • peglib-core/src/main/java/org/pragmatica/peg/grammar/GrammarParser.java
  • peglib-core/src/main/java/org/pragmatica/peg/grammar/GrammarResolver.java
  • peglib-core/src/main/java/org/pragmatica/peg/tree/TriviaPostPass.java
  • peglib-core/src/main/java/org/pragmatica/peg/v6/Parser.java
  • peglib-core/src/main/java/org/pragmatica/peg/v6/PegParser.java
  • peglib-core/src/main/java/org/pragmatica/peg/v6/analyzer/NamedCaptureCause.java
  • peglib-core/src/main/java/org/pragmatica/peg/v6/analyzer/NamedCaptureDetector.java
  • peglib-core/src/main/java/org/pragmatica/peg/v6/generator/LexerGenerator.java
  • peglib-core/src/main/java/org/pragmatica/peg/v6/generator/ParserCompiler.java
  • peglib-core/src/main/java/org/pragmatica/peg/v6/generator/ParserGenerator.java
  • peglib-core/src/main/java/org/pragmatica/peg/v6/incremental/IncrementalParser.java
  • peglib-core/src/main/java/org/pragmatica/peg/v6/lexer/DfaBuilder.java
  • peglib-core/src/main/java/org/pragmatica/peg/v6/lexer/LexerEngine.java
  • peglib-core/src/test/java/org/pragmatica/peg/grammar/CheckpointDirectiveTest.java
  • peglib-core/src/test/java/org/pragmatica/peg/tree/FirstMemberTriviaTest.java
  • peglib-core/src/test/java/org/pragmatica/peg/v6/analyzer/LeftRecursionDetectorTest.java
  • peglib-core/src/test/java/org/pragmatica/peg/v6/cst/FirstMemberV6TriviaTest.java
  • peglib-core/src/test/java/org/pragmatica/peg/v6/diagnostic/Java25SelfHostDiagTest.java
  • peglib-core/src/test/java/org/pragmatica/peg/v6/generator/MaxDiagnosticsTest.java
  • peglib-core/src/test/java/org/pragmatica/peg/v6/generator/MixedRuleFallbackTest.java
  • peglib-core/src/test/java/org/pragmatica/peg/v6/generator/NamedCaptureRuntimeTest.java
  • peglib-core/src/test/java/org/pragmatica/peg/v6/generator/NamedCaptureTest.java
  • peglib-core/src/test/java/org/pragmatica/peg/v6/generator/PerRuleRecoverDirectiveTest.java
  • peglib-core/src/test/java/org/pragmatica/peg/v6/lexer/RuleClassifierTest.java
  • peglib-core/src/test/java/org/pragmatica/peg/v6/lexer/TriviaClassificationTest.java
  • peglib-core/src/test/resources/java25.peg
  • peglib-formatter/pom.xml
  • peglib-formatter/src/main/java/org/pragmatica/peg/formatter/v6/V6TriviaPolicy.java
  • peglib-incremental/pom.xml
  • peglib-incremental/src/test/java/org/pragmatica/peg/v6/incremental/CheckpointDirectiveIncrementalTest.java
  • peglib-maven-plugin/pom.xml
  • peglib-playground/pom.xml
  • peglib-runtime/pom.xml
  • peglib-runtime/src/main/java/org/pragmatica/peg/v6/token/TokenArray.java
  • peglib-runtime/src/test/java/org/pragmatica/peg/v6/cst/CstArrayTest.java
  • peglib-runtime/src/test/java/org/pragmatica/peg/v6/token/TokenArrayTest.java
  • pom.xml
💤 Files with no reviewable changes (4)
  • peglib-core/src/test/java/org/pragmatica/peg/v6/generator/NamedCaptureTest.java
  • peglib-core/src/main/java/org/pragmatica/peg/v6/analyzer/NamedCaptureCause.java
  • peglib-core/src/main/java/org/pragmatica/peg/v6/analyzer/NamedCaptureDetector.java
  • peglib-core/src/main/java/org/pragmatica/peg/v6/PegParser.java

Comment on lines +62 to +72
/**
* Canonical compact constructor. Applies a defensive copy to
* {@code checkpointRules} so the record's exposed set is immutable
* regardless of how callers construct the instance. Matches the style
* used for {@code recoverSets}.
*
* @since 0.6.1
*/
public Grammar {
checkpointRules = Set.copyOf(checkpointRules);
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Defensively copy recoverSets in the canonical constructor as well.

Right now only checkpointRules is copied; recoverSets (and its inner sets) can still be mutated by callers after construction, which can change parser behavior post-validation.

Suggested fix
 public Grammar {
+    recoverSets = recoverSets.entrySet()
+                             .stream()
+                             .collect(Collectors.toUnmodifiableMap(
+                                 Map.Entry::getKey,
+                                 e -> Set.copyOf(e.getValue())));
     checkpointRules = Set.copyOf(checkpointRules);
 }
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
/**
* Canonical compact constructor. Applies a defensive copy to
* {@code checkpointRules} so the record's exposed set is immutable
* regardless of how callers construct the instance. Matches the style
* used for {@code recoverSets}.
*
* @since 0.6.1
*/
public Grammar {
checkpointRules = Set.copyOf(checkpointRules);
}
/**
* Canonical compact constructor. Applies a defensive copy to
* {`@code` checkpointRules} so the record's exposed set is immutable
* regardless of how callers construct the instance. Matches the style
* used for {`@code` recoverSets}.
*
* `@since` 0.6.1
*/
public Grammar {
recoverSets = recoverSets.entrySet()
.stream()
.collect(Collectors.toUnmodifiableMap(
Map.Entry::getKey,
e -> Set.copyOf(e.getValue())));
checkpointRules = Set.copyOf(checkpointRules);
}
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@peglib-core/src/main/java/org/pragmatica/peg/grammar/Grammar.java` around
lines 62 - 72, The canonical compact constructor for Grammar only defensively
copies checkpointRules but not recoverSets, allowing callers to mutate the outer
or inner sets after construction; update the Grammar constructor to defensively
copy recoverSets as well by creating an immutable outer set (e.g., Set.copyOf)
whose elements are immutable copies of each inner set (e.g., map each inner Set
to Set.copyOf(inner) before collecting), so both recoverSets and its contained
sets are unmodifiable at construction time.

Comment on lines +107 to 117
if ("checkpoint".equals(directive.name()) && pos + 1 < tokens.size() && tokens.get(pos + 1) instanceof GrammarToken.Identifier) {
advance();
var result = parseCheckpointDirective();
if (result instanceof Result.Failure< ? > f) {
return f.cause()
.result();
}
checkpointRules.add(result.unwrap());
continue;
}
advance();

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Route all %checkpoint directives through parseCheckpointDirective() for correct errors.

On Line 107, the identifier lookahead prevents malformed %checkpoint directives from using the dedicated error path. They currently fall through and report a generic '<-' expectation instead of “rule name for %checkpoint”.

Proposed fix
-                if ("checkpoint".equals(directive.name()) && pos + 1 < tokens.size() && tokens.get(pos + 1) instanceof GrammarToken.Identifier) {
+                if ("checkpoint".equals(directive.name())) {
                     advance();
                     var result = parseCheckpointDirective();
                     if (result instanceof Result.Failure< ? > f) {
                         return f.cause()
                                 .result();
                     }
                     checkpointRules.add(result.unwrap());
                     continue;
                 }
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@peglib-core/src/main/java/org/pragmatica/peg/grammar/GrammarParser.java`
around lines 107 - 117, The logic is incorrectly skipping
parseCheckpointDirective() when the token after a "checkpoint" directive isn't
an Identifier, causing generic parse errors; change the block that checks
"checkpoint".equals(directive.name()) so it always calls advance() and then
invoke parseCheckpointDirective() (use parseCheckpointDirective() result
handling as with Result.Failure<?> f to return f.cause().result()), and on
success add result.unwrap() to checkpointRules instead of falling through to the
later advance() path; adjust control flow around tokens, advance(),
parseCheckpointDirective(), checkpointRules and Result.Failure handling so all
%checkpoint directives go through parseCheckpointDirective().

Comment thread README.md
Output example:
The `formatRustStyle` output mirrors `cargo check`:

```

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Add a language identifier to the fenced block at Line 225.

This triggers markdownlint MD040. Use a language tag (for example, text) on the opening fence.

🛠️ Proposed fix
-```
+```text
🧰 Tools
🪛 markdownlint-cli2 (0.22.1)

[warning] 225-225: Fenced code blocks should have a language specified

(MD040, fenced-code-language)

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@README.md` at line 225, The fenced code block opening fence (``` ) in the
README lacks a language identifier, triggering markdownlint MD040; update the
opening fence to include a language tag (for example change ``` to ```text or
another appropriate language) so the block becomes a fenced code block with a
language identifier.

@siy siy merged commit 63ea7d1 into main May 14, 2026
2 checks passed
@siy siy deleted the release-0.6.1 branch May 14, 2026 08:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant