Repository navigation
v1.2.0
First tagged release since v1.0.2. It carries two pieces of API news, TokenBuffer.reset() and a behavior change in Tokenizer.addVar, plus a pile of repo hygiene that came with them. Version 1.1.0 was published to npm from this same range and never tagged, so its contents are described here too.
The number is 1.2.0 rather than 1.1.1 for two reasons. TokenBuffer.reset() is new exported API, which is a minor bump on its own, and the addVar fix changes what existing calls do. Neither belongs in a patch release, where downstream code tends to pick the change up without reading anything.
Added
-
TokenBuffer.reset()drops every token read from the tokenizer but not yet consumed, so the nextpeekornextscans afresh (PR 3, published as 1.1.0). Owners that keep one buffer alive across several independent runs need it, because a run that throws part way leaves its offending token at the front of the queue and the following run reads that stale token before it touches its own input. galore hit exactly this, seepanyam/Galore#9.Two things
reset()deliberately leaves alone. The tape is not rewound, so tokens that were peeked at are dropped rather than re-read, and an owner starting over on the same tape has to reposition the tape itself.tokenizerContextis kept, since it belongs to whoever set it. -
First test file for
TokenBuffer, which had none (PR 3).
Fixed
-
Tokenizer.addVarkeeps both definitions when the same variable name is added twice (PR 7, issue 5). It used to build the union of the existing pattern and the new one, then store only the new one, so the first definition was dropped and the union branch never did anything. A name defined more than once now stands for the alternation of everything added under it, in call order, which is also match priority.Surfaced by the first clean lint run after the eslint 9 migration, which flagged the discarded local. Two
%definelines for one name in the playground DSL are the visible case, since%defineis the one path in this repo that reachesaddVar.
Changed
npm run lintandnpm run lintfixwork again (PR 6, issue 4). eslint 9 only reads flat config and the repo still had.eslintrc.json, so both scripts had been exiting before they checked anything, and the pre-commit hook that callslintfixwent along with them.eslint.config.mjsreplaces the old config and follows galore's own eslint 9 migration.- Source formatting across ten files, applied by
eslint --fixagainst the repo's own prettier config (PR 6). Mostly multi-parameter constructors reflowed by prettier 3.7. constdeclarations insidecaseclauses now sit in their own blocks, invm.ts,jsparser.tsandrepr.ts(PR 6). No behavior change, and the suite counts are identical either side.- CI runs lint, build and the test suite on pushes to master and on every PR, across node 20 and 22 (PR 3, PR 6). The only workflow before was the Pages deploy.
addVarandgetVarhave doc comments for the first time, covering what a redefinition means and whatgetVarreturns after several of them (PR 7).
Known issues
- The API reference documents
addVar("DIGIT", /[0-9]/), passing aRegExpwhere the signature wants aRegex. That call throwsRegex Type not yet supported: undefined(issue 8). UseBuilder.exprFromFlexRE("[0-9]"), as the Flex Syntax page shows. - The playground
%definepath indocs/components/playground/dsl.tsis the only caller ofaddVarin this repo and has no tests (issue 9).
Upgrading
reset() is additive, so nothing to do there. If you reuse a TokenBuffer across runs, call reset() between them.
The addVar change is worth a look before upgrading if you build named patterns at runtime. Code that called addVar twice under one name used to end up with the second pattern and now ends up with both, as an alternation. Anything that was relying on the second call to overwrite the first needs to stop reusing the name, since there is no clear-and-replace on the API. Defining each name once behaves exactly as before.