Release 0.6.1#35
Conversation
…shift-in-FieldDecl fix (0.6.2)
📝 WalkthroughWalkthroughPeglib v0.6.1 completes runtime support for per-rule ChangesDocumentation & Release
Checkpoint Rules Grammar & Incremental Integration
Doc-Comment Trivia Classification (Kind & Lexer)
First-Member Trivia Fix & TriviaPostPass
Per-Rule Recovery with RuleKind Dispatch
MIXED-Rule Char-Level Fallback & Expression Emission
Diagnostic Cap (
Named Capture & Back-Reference Runtime Support
Test Infrastructure & Self-Host Diagnostics
Sequence Diagram(s)sequenceDiagram
participant Parser as Parser.parse
participant Lexer as TokenArray.lex
participant CompiledParser as Generated Parser
participant Recovery as Recovery Logic
participant Diagnostics as Diagnostics
Parser->>Lexer: input + maxDiagnostics
Lexer->>Lexer: classify doc comments<br/>in WHITESPACE tokens
Lexer-->>Parser: TokenArray<br/>(with doc trivia kinds)
Parser->>CompiledParser: parse(tokens, maxDiagnostics)
CompiledParser->>CompiledParser: initialize<br/>captures, lastFailedRuleKind
loop start-rule attempts
CompiledParser->>CompiledParser: reset lastFailedRuleKind
CompiledParser->>CompiledParser: parseStartRule(...)
opt expression fails
CompiledParser->>CompiledParser: fail(expected, ruleKind)<br/>→ lastFailedRuleKind=ruleKind
CompiledParser->>Recovery: syncForRule(lastFailedRuleKind)
Recovery-->>CompiledParser: SYNC_RuleName or DEFAULT_SYNC
CompiledParser->>CompiledParser: advance to sync token
end
CompiledParser->>Diagnostics: emit diagnostic<br/>if size < maxDiagnostics
alt diagnostics.size >= maxDiagnostics
CompiledParser->>CompiledParser: break start-rule loop
end
end
CompiledParser->>CompiledParser: build CST<br/>(with named-capture spans)
CompiledParser-->>Parser: ParseResult<br/>(CST + diagnostics)
Parser-->>Parser: return with diagnostic cap<br/>honored
sequenceDiagram
participant Grammar as Grammar Parser
participant GrammarResolver as GrammarResolver
participant IncrementalParser as IncrementalParser
participant Parser as Compiled Parser
Grammar->>Grammar: parse %checkpoint directives<br/>→ checkpointRules set
Grammar-->>Parser: Grammar(rules, checkpointRules)
GrammarResolver->>GrammarResolver: load imported grammars
GrammarResolver->>GrammarResolver: union checkpointRules<br/>from root + imports
GrammarResolver-->>Parser: Grammar(..., unionedCheckpoints)
Parser->>Parser: compile & cache
IncrementalParser->>Parser: grammar()
IncrementalParser->>IncrementalParser: checkpointRules<br/>non-empty?
alt grammar declares checkpoints
IncrementalParser->>IncrementalParser: use grammar.checkpointRules()
else fallback
IncrementalParser->>IncrementalParser: use DEFAULT_CHECKPOINT_RULES
end
IncrementalParser->>IncrementalParser: parse with checkpoint<br/>boundaries
Estimated code review effort🎯 4 (Complex) | ⏱️ ~75 minutes This release spans 8 distinct feature/fix layers with heterogeneous changes across grammar, lexer, parser codegen, runtime behavior, and test infrastructure. The changes include subtle control-flow updates (per-rule recovery dispatch, trivia cursor logic), new codegen patterns (MIXED-rule char-level fallback, named-capture spans), and extensive test coverage. While many changes are internally consistent within each layer, the overall scope and interaction density—especially between recovery routing, diagnostic capping, and expression emission—demand careful sequential review of the parser generator and runtime semantics. Possibly related PRs
Poem
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
|
There was a problem hiding this comment.
Actionable comments posted: 3
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
pom.xml (1)
50-50:⚠️ Potential issue | 🟠 Major | ⚡ Quick winUpdate
pragmatica-lite.versionto 0.9.0.The root pom.xml specifies version
0.24.0, but coding guidelines require version0.9.0for thepragmatica-lite:coreruntime dependency. Update the property to comply with the requirement for Result/Option/Promise types.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@pom.xml` at line 50, Update the pom property pragmatica-lite.version from 0.24.0 to 0.9.0 so the runtime dependency pragmatica-lite:core uses the required 0.9.0 release; locate the XML property element named <pragmatica-lite.version> in the root POM and change its value to 0.9.0, then rebuild to ensure the Result/Option/Promise types are sourced from the correct version.
🧹 Nitpick comments (2)
peglib-core/src/main/java/org/pragmatica/peg/tree/TriviaPostPass.java (1)
454-460: ⚖️ Poor tradeoffHardcoded trivia-start predicate may be incomplete.
The
canStartTriviamethod hardcodes whitespace and comment-start characters (' ','\t','\r','\n','/','#'). This may not cover:
- Other whitespace: form feed (
'\f'), vertical tab ('\u000B')- Grammar-specific comment starters: Some languages use
;,--, or other prefixes- Unicode whitespace: characters like non-breaking space (
\u00A0), zero-width space, etc.While the current set likely covers Java and most common languages, consider either:
- Consulting the grammar's
%whitespacerule to derive valid start characters- Or documenting this as a known limitation for exotic grammar patterns
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@peglib-core/src/main/java/org/pragmatica/peg/tree/TriviaPostPass.java` around lines 454 - 460, The canStartTrivia predicate is too narrow; replace its hardcoded checks with a Unicode-aware whitespace test and keep the common comment-start chars—change canStartTrivia to return Character.isWhitespace(c) || c == '/' || c == '#', so form feed, vertical tab and other Unicode whitespace are covered; update any callers (e.g., rebuildNonTerminal) expectations or add a short comment noting grammar-specific comment markers may still need extending.peglib-core/src/main/java/org/pragmatica/peg/v6/lexer/DfaBuilder.java (1)
53-60: ⚡ Quick winCentralize reserved token-kind constants to a single source of truth.
These reserved kind IDs are duplicated here and in
TokenArray; future edits can silently desync lexer emission vs runtime classification.Proposed refactor
+import org.pragmatica.peg.v6.token.TokenArray; ... - public static final int KIND_WHITESPACE = 0; - public static final int KIND_LINE_COMMENT = 1; - public static final int KIND_BLOCK_COMMENT = 2; - public static final int KIND_DOC_LINE_COMMENT = 3; - public static final int KIND_DOC_BLOCK_COMMENT = 4; - public static final int FIRST_USER_KIND = 5; + public static final int KIND_WHITESPACE = TokenArray.KIND_WHITESPACE; + public static final int KIND_LINE_COMMENT = TokenArray.KIND_LINE_COMMENT; + public static final int KIND_BLOCK_COMMENT = TokenArray.KIND_BLOCK_COMMENT; + public static final int KIND_DOC_LINE_COMMENT = TokenArray.KIND_DOC_LINE_COMMENT; + public static final int KIND_DOC_BLOCK_COMMENT = TokenArray.KIND_DOC_BLOCK_COMMENT; + public static final int FIRST_USER_KIND = TokenArray.FIRST_USER_KIND;Also applies to: 214-219
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@peglib-core/src/main/java/org/pragmatica/peg/v6/lexer/DfaBuilder.java` around lines 53 - 60, The duplicated reserved token-kind constants (KIND_WHITESPACE, KIND_LINE_COMMENT, KIND_BLOCK_COMMENT, KIND_DOC_LINE_COMMENT, KIND_DOC_BLOCK_COMMENT, FIRST_USER_KIND) should be centralized into the single source of truth in TokenArray and removed from DfaBuilder; replace the local constant definitions in DfaBuilder with references to TokenArray (e.g. use TokenArray.KIND_WHITESPACE, TokenArray.FIRST_USER_KIND, etc.), remove the duplicate block at the second location (lines around the other duplicate definitions), and update any local references in DfaBuilder to use the TokenArray symbols so compilation and runtime classification stay in sync. Ensure the TokenArray constants are public/static and import or fully-qualify them where needed.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@peglib-core/src/main/java/org/pragmatica/peg/grammar/Grammar.java`:
- Around line 62-72: The canonical compact constructor for Grammar only
defensively copies checkpointRules but not recoverSets, allowing callers to
mutate the outer or inner sets after construction; update the Grammar
constructor to defensively copy recoverSets as well by creating an immutable
outer set (e.g., Set.copyOf) whose elements are immutable copies of each inner
set (e.g., map each inner Set to Set.copyOf(inner) before collecting), so both
recoverSets and its contained sets are unmodifiable at construction time.
In `@peglib-core/src/main/java/org/pragmatica/peg/grammar/GrammarParser.java`:
- Around line 107-117: The logic is incorrectly skipping
parseCheckpointDirective() when the token after a "checkpoint" directive isn't
an Identifier, causing generic parse errors; change the block that checks
"checkpoint".equals(directive.name()) so it always calls advance() and then
invoke parseCheckpointDirective() (use parseCheckpointDirective() result
handling as with Result.Failure<?> f to return f.cause().result()), and on
success add result.unwrap() to checkpointRules instead of falling through to the
later advance() path; adjust control flow around tokens, advance(),
parseCheckpointDirective(), checkpointRules and Result.Failure handling so all
%checkpoint directives go through parseCheckpointDirective().
In `@README.md`:
- Line 225: The fenced code block opening fence (``` ) in the README lacks a
language identifier, triggering markdownlint MD040; update the opening fence to
include a language tag (for example change ``` to ```text or another appropriate
language) so the block becomes a fenced code block with a language identifier.
---
Outside diff comments:
In `@pom.xml`:
- Line 50: Update the pom property pragmatica-lite.version from 0.24.0 to 0.9.0
so the runtime dependency pragmatica-lite:core uses the required 0.9.0 release;
locate the XML property element named <pragmatica-lite.version> in the root POM
and change its value to 0.9.0, then rebuild to ensure the Result/Option/Promise
types are sourced from the correct version.
---
Nitpick comments:
In `@peglib-core/src/main/java/org/pragmatica/peg/tree/TriviaPostPass.java`:
- Around line 454-460: The canStartTrivia predicate is too narrow; replace its
hardcoded checks with a Unicode-aware whitespace test and keep the common
comment-start chars—change canStartTrivia to return Character.isWhitespace(c) ||
c == '/' || c == '#', so form feed, vertical tab and other Unicode whitespace
are covered; update any callers (e.g., rebuildNonTerminal) expectations or add a
short comment noting grammar-specific comment markers may still need extending.
In `@peglib-core/src/main/java/org/pragmatica/peg/v6/lexer/DfaBuilder.java`:
- Around line 53-60: The duplicated reserved token-kind constants
(KIND_WHITESPACE, KIND_LINE_COMMENT, KIND_BLOCK_COMMENT, KIND_DOC_LINE_COMMENT,
KIND_DOC_BLOCK_COMMENT, FIRST_USER_KIND) should be centralized into the single
source of truth in TokenArray and removed from DfaBuilder; replace the local
constant definitions in DfaBuilder with references to TokenArray (e.g. use
TokenArray.KIND_WHITESPACE, TokenArray.FIRST_USER_KIND, etc.), remove the
duplicate block at the second location (lines around the other duplicate
definitions), and update any local references in DfaBuilder to use the
TokenArray symbols so compilation and runtime classification stay in sync.
Ensure the TokenArray constants are public/static and import or fully-qualify
them where needed.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 4a2e8cb4-4da9-439e-8d42-8a8d3e37d1dd
📒 Files selected for processing (45)
CHANGELOG.mdREADME.mddocs/GRAMMAR-DSL.mddocs/HANDOVER.mddocs/VISITOR-TUTORIAL.mddocs/bugs/first-member-trivia-loss-2026-05-12.mdpeglib-core/pom.xmlpeglib-core/src/main/java/org/pragmatica/peg/grammar/Grammar.javapeglib-core/src/main/java/org/pragmatica/peg/grammar/GrammarParser.javapeglib-core/src/main/java/org/pragmatica/peg/grammar/GrammarResolver.javapeglib-core/src/main/java/org/pragmatica/peg/tree/TriviaPostPass.javapeglib-core/src/main/java/org/pragmatica/peg/v6/Parser.javapeglib-core/src/main/java/org/pragmatica/peg/v6/PegParser.javapeglib-core/src/main/java/org/pragmatica/peg/v6/analyzer/NamedCaptureCause.javapeglib-core/src/main/java/org/pragmatica/peg/v6/analyzer/NamedCaptureDetector.javapeglib-core/src/main/java/org/pragmatica/peg/v6/generator/LexerGenerator.javapeglib-core/src/main/java/org/pragmatica/peg/v6/generator/ParserCompiler.javapeglib-core/src/main/java/org/pragmatica/peg/v6/generator/ParserGenerator.javapeglib-core/src/main/java/org/pragmatica/peg/v6/incremental/IncrementalParser.javapeglib-core/src/main/java/org/pragmatica/peg/v6/lexer/DfaBuilder.javapeglib-core/src/main/java/org/pragmatica/peg/v6/lexer/LexerEngine.javapeglib-core/src/test/java/org/pragmatica/peg/grammar/CheckpointDirectiveTest.javapeglib-core/src/test/java/org/pragmatica/peg/tree/FirstMemberTriviaTest.javapeglib-core/src/test/java/org/pragmatica/peg/v6/analyzer/LeftRecursionDetectorTest.javapeglib-core/src/test/java/org/pragmatica/peg/v6/cst/FirstMemberV6TriviaTest.javapeglib-core/src/test/java/org/pragmatica/peg/v6/diagnostic/Java25SelfHostDiagTest.javapeglib-core/src/test/java/org/pragmatica/peg/v6/generator/MaxDiagnosticsTest.javapeglib-core/src/test/java/org/pragmatica/peg/v6/generator/MixedRuleFallbackTest.javapeglib-core/src/test/java/org/pragmatica/peg/v6/generator/NamedCaptureRuntimeTest.javapeglib-core/src/test/java/org/pragmatica/peg/v6/generator/NamedCaptureTest.javapeglib-core/src/test/java/org/pragmatica/peg/v6/generator/PerRuleRecoverDirectiveTest.javapeglib-core/src/test/java/org/pragmatica/peg/v6/lexer/RuleClassifierTest.javapeglib-core/src/test/java/org/pragmatica/peg/v6/lexer/TriviaClassificationTest.javapeglib-core/src/test/resources/java25.pegpeglib-formatter/pom.xmlpeglib-formatter/src/main/java/org/pragmatica/peg/formatter/v6/V6TriviaPolicy.javapeglib-incremental/pom.xmlpeglib-incremental/src/test/java/org/pragmatica/peg/v6/incremental/CheckpointDirectiveIncrementalTest.javapeglib-maven-plugin/pom.xmlpeglib-playground/pom.xmlpeglib-runtime/pom.xmlpeglib-runtime/src/main/java/org/pragmatica/peg/v6/token/TokenArray.javapeglib-runtime/src/test/java/org/pragmatica/peg/v6/cst/CstArrayTest.javapeglib-runtime/src/test/java/org/pragmatica/peg/v6/token/TokenArrayTest.javapom.xml
💤 Files with no reviewable changes (4)
- peglib-core/src/test/java/org/pragmatica/peg/v6/generator/NamedCaptureTest.java
- peglib-core/src/main/java/org/pragmatica/peg/v6/analyzer/NamedCaptureCause.java
- peglib-core/src/main/java/org/pragmatica/peg/v6/analyzer/NamedCaptureDetector.java
- peglib-core/src/main/java/org/pragmatica/peg/v6/PegParser.java
| /** | ||
| * Canonical compact constructor. Applies a defensive copy to | ||
| * {@code checkpointRules} so the record's exposed set is immutable | ||
| * regardless of how callers construct the instance. Matches the style | ||
| * used for {@code recoverSets}. | ||
| * | ||
| * @since 0.6.1 | ||
| */ | ||
| public Grammar { | ||
| checkpointRules = Set.copyOf(checkpointRules); | ||
| } |
There was a problem hiding this comment.
Defensively copy recoverSets in the canonical constructor as well.
Right now only checkpointRules is copied; recoverSets (and its inner sets) can still be mutated by callers after construction, which can change parser behavior post-validation.
Suggested fix
public Grammar {
+ recoverSets = recoverSets.entrySet()
+ .stream()
+ .collect(Collectors.toUnmodifiableMap(
+ Map.Entry::getKey,
+ e -> Set.copyOf(e.getValue())));
checkpointRules = Set.copyOf(checkpointRules);
}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| /** | |
| * Canonical compact constructor. Applies a defensive copy to | |
| * {@code checkpointRules} so the record's exposed set is immutable | |
| * regardless of how callers construct the instance. Matches the style | |
| * used for {@code recoverSets}. | |
| * | |
| * @since 0.6.1 | |
| */ | |
| public Grammar { | |
| checkpointRules = Set.copyOf(checkpointRules); | |
| } | |
| /** | |
| * Canonical compact constructor. Applies a defensive copy to | |
| * {`@code` checkpointRules} so the record's exposed set is immutable | |
| * regardless of how callers construct the instance. Matches the style | |
| * used for {`@code` recoverSets}. | |
| * | |
| * `@since` 0.6.1 | |
| */ | |
| public Grammar { | |
| recoverSets = recoverSets.entrySet() | |
| .stream() | |
| .collect(Collectors.toUnmodifiableMap( | |
| Map.Entry::getKey, | |
| e -> Set.copyOf(e.getValue()))); | |
| checkpointRules = Set.copyOf(checkpointRules); | |
| } |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@peglib-core/src/main/java/org/pragmatica/peg/grammar/Grammar.java` around
lines 62 - 72, The canonical compact constructor for Grammar only defensively
copies checkpointRules but not recoverSets, allowing callers to mutate the outer
or inner sets after construction; update the Grammar constructor to defensively
copy recoverSets as well by creating an immutable outer set (e.g., Set.copyOf)
whose elements are immutable copies of each inner set (e.g., map each inner Set
to Set.copyOf(inner) before collecting), so both recoverSets and its contained
sets are unmodifiable at construction time.
| if ("checkpoint".equals(directive.name()) && pos + 1 < tokens.size() && tokens.get(pos + 1) instanceof GrammarToken.Identifier) { | ||
| advance(); | ||
| var result = parseCheckpointDirective(); | ||
| if (result instanceof Result.Failure< ? > f) { | ||
| return f.cause() | ||
| .result(); | ||
| } | ||
| checkpointRules.add(result.unwrap()); | ||
| continue; | ||
| } | ||
| advance(); |
There was a problem hiding this comment.
Route all %checkpoint directives through parseCheckpointDirective() for correct errors.
On Line 107, the identifier lookahead prevents malformed %checkpoint directives from using the dedicated error path. They currently fall through and report a generic '<-' expectation instead of “rule name for %checkpoint”.
Proposed fix
- if ("checkpoint".equals(directive.name()) && pos + 1 < tokens.size() && tokens.get(pos + 1) instanceof GrammarToken.Identifier) {
+ if ("checkpoint".equals(directive.name())) {
advance();
var result = parseCheckpointDirective();
if (result instanceof Result.Failure< ? > f) {
return f.cause()
.result();
}
checkpointRules.add(result.unwrap());
continue;
}🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@peglib-core/src/main/java/org/pragmatica/peg/grammar/GrammarParser.java`
around lines 107 - 117, The logic is incorrectly skipping
parseCheckpointDirective() when the token after a "checkpoint" directive isn't
an Identifier, causing generic parse errors; change the block that checks
"checkpoint".equals(directive.name()) so it always calls advance() and then
invoke parseCheckpointDirective() (use parseCheckpointDirective() result
handling as with Result.Failure<?> f to return f.cause().result()), and on
success add result.unwrap() to checkpointRules instead of falling through to the
later advance() path; adjust control flow around tokens, advance(),
parseCheckpointDirective(), checkpointRules and Result.Failure handling so all
%checkpoint directives go through parseCheckpointDirective().
| Output example: | ||
| The `formatRustStyle` output mirrors `cargo check`: | ||
|
|
||
| ``` |
There was a problem hiding this comment.
Add a language identifier to the fenced block at Line 225.
This triggers markdownlint MD040. Use a language tag (for example, text) on the opening fence.
🛠️ Proposed fix
-```
+```text🧰 Tools
🪛 markdownlint-cli2 (0.22.1)
[warning] 225-225: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@README.md` at line 225, The fenced code block opening fence (``` ) in the
README lacks a language identifier, triggering markdownlint MD040; update the
opening fence to include a language tag (for example change ``` to ```text or
another appropriate language) so the block becomes a fenced code block with a
language identifier.
Summary
KIND_DOC_LINE_COMMENT(3) andKIND_DOC_BLOCK_COMMENT(4) trivia kinds;FIRST_USER_KINDshifted to 5;///and/**post-DFA classification in lexer engine and generator%recoversync sets:SYNC_<Rule>arrays emitted per rule;lastFailedRuleKindfield routes recovery to rule-specific sync set%checkpoint RuleNamegrammar directive parsed and stored inGrammar.checkpointRules();IncrementalParserconsumes it; canonicaljava25.pegdeclaresStmt,MethodDecl,TypeDecl$name<expr>) and back-references ($name) runtime via source-span equality;NamedCaptureDetectorrejection removedCharClassandAnyin MIXED rules emit token-level proxy viainput.charAt(tokens.startAt(pos))Java25SelfHostDiagTestgate added; shift-in-FieldDecl bug identified and deferred to 0.6.2 (2 assertions@Disabled)maxDiagnosticscap honored in generated recovery loop;parse(String, int)wired end-to-enddocs/VISITOR-TUTORIAL.mdadded (489-line calculator walkthrough)TriviaPostPassfirst-member trivia loss fixed (0.5.x legacy); lexer empty-match warning softened; canonicaljava25.peg%whitespacesplit for per-kind triviaTest plan
mvn install -Djbct.skip=true— 1440 tests, 0 failuresJava25ParserGateTest— 20/20 fixtures clean parseFactoryClassGeneratorDiagTest— 0 diagnosticsV6FormatterCorpusGateTest— 20/20 round-tripJava25SelfHostDiagTest— dumps diagnostics; 2 assertions disabled pending 0.6.2 shift-in-FieldDecl fixKnown limitations carried to 0.6.2
CompilationUnitlevel (shift-in-FieldDecl bug)%whitespacetokenization (ZeroOrMore loop emission) deferredSummary by CodeRabbit
New Features
%checkpointdirectives for selective re-parsing.%recover [chars] RuleNamedirectives.parse(input, maxDiagnostics)overload.///and/** ... */).Bug Fixes
Documentation