Repository navigation
v1.0.7
A parser can be reused after a parse that threw. Before this, the token that caused a failure stayed in the parser's token buffer, so every later parse() call on that parser reported the first failure's error against input that parses cleanly on its own.
Fixed
-
parse()starts from an empty token buffer, so a failed parse no longer poisons the next one (issue 9, PR 10). A parser builds oneTokenBufferinsetTokenizerand keeps it for its lifetime, and nothing emptied it between parses. Anything that parses more than once per process was affected: a build step, a test suite, a server, an editor reparsing on keystroke. The docs playground and the embedded grammar sandboxes were hitting it on every reparse after a typo.The fix covers the LR, LL and GLR parsers, since all three share
parse()and all three had the bug.
Changed
- Requires tlex 1.1.0 or later, for the
TokenBuffer.reset()this calls. The dependency moves from*to^1.1.0, so an install cannot resolve to a tlex without it. - CI runs lint, build and the test suite on pushes to master and on every PR, across node 20 and 22. There was no CI before, so tests only ran through the pre-commit hook.
Behavior worth knowing about
If you pass a Tape rather than a string, consecutive parse() calls sharing one tape no longer inherit the token the previous call had peeked at but not consumed. A tape only moves forward, so a cleared token is dropped rather than re-read. Nothing in this repo or its samples relied on that, and the pattern was already broken for the reason in issue 9, but it is a real difference.
Known gaps
- The GLR parser gets the fix and no test, because
glr.Parser.parseInputreturns an empty forest and has no observable result to assert on (issue 11). - The suites under
samples/still cannot resolvegalore, so CI scopes its test step tosrc/tests(issue 12).
Upgrading
No code changes needed. If you call setTokenizer before every parse as a workaround for issue 9, you can drop it.