Skip to content

v1.2.0

Latest

Choose a tag to compare

@panyam panyam released this 21 Sep 09:47
· 6 commits to master since this release

v1.2.0

First tagged release since v1.0.2. It carries two pieces of API news, TokenBuffer.reset() and a behavior change in Tokenizer.addVar, plus a pile of repo hygiene that came with them. Version 1.1.0 was published to npm from this same range and never tagged, so its contents are described here too.

The number is 1.2.0 rather than 1.1.1 for two reasons. TokenBuffer.reset() is new exported API, which is a minor bump on its own, and the addVar fix changes what existing calls do. Neither belongs in a patch release, where downstream code tends to pick the change up without reading anything.

Added

  • TokenBuffer.reset() drops every token read from the tokenizer but not yet consumed, so the next peek or next scans afresh (PR 3, published as 1.1.0). Owners that keep one buffer alive across several independent runs need it, because a run that throws part way leaves its offending token at the front of the queue and the following run reads that stale token before it touches its own input. galore hit exactly this, see panyam/Galore#9.

    Two things reset() deliberately leaves alone. The tape is not rewound, so tokens that were peeked at are dropped rather than re-read, and an owner starting over on the same tape has to reposition the tape itself. tokenizerContext is kept, since it belongs to whoever set it.

  • First test file for TokenBuffer, which had none (PR 3).

Fixed

  • Tokenizer.addVar keeps both definitions when the same variable name is added twice (PR 7, issue 5). It used to build the union of the existing pattern and the new one, then store only the new one, so the first definition was dropped and the union branch never did anything. A name defined more than once now stands for the alternation of everything added under it, in call order, which is also match priority.

    Surfaced by the first clean lint run after the eslint 9 migration, which flagged the discarded local. Two %define lines for one name in the playground DSL are the visible case, since %define is the one path in this repo that reaches addVar.

Changed

  • npm run lint and npm run lintfix work again (PR 6, issue 4). eslint 9 only reads flat config and the repo still had .eslintrc.json, so both scripts had been exiting before they checked anything, and the pre-commit hook that calls lintfix went along with them. eslint.config.mjs replaces the old config and follows galore's own eslint 9 migration.
  • Source formatting across ten files, applied by eslint --fix against the repo's own prettier config (PR 6). Mostly multi-parameter constructors reflowed by prettier 3.7.
  • const declarations inside case clauses now sit in their own blocks, in vm.ts, jsparser.ts and repr.ts (PR 6). No behavior change, and the suite counts are identical either side.
  • CI runs lint, build and the test suite on pushes to master and on every PR, across node 20 and 22 (PR 3, PR 6). The only workflow before was the Pages deploy.
  • addVar and getVar have doc comments for the first time, covering what a redefinition means and what getVar returns after several of them (PR 7).

Known issues

  • The API reference documents addVar("DIGIT", /[0-9]/), passing a RegExp where the signature wants a Regex. That call throws Regex Type not yet supported: undefined (issue 8). Use Builder.exprFromFlexRE("[0-9]"), as the Flex Syntax page shows.
  • The playground %define path in docs/components/playground/dsl.ts is the only caller of addVar in this repo and has no tests (issue 9).

Upgrading

reset() is additive, so nothing to do there. If you reuse a TokenBuffer across runs, call reset() between them.

The addVar change is worth a look before upgrading if you build named patterns at runtime. Code that called addVar twice under one name used to end up with the second pattern and now ends up with both, as an alternation. Anything that was relying on the second call to overwrite the first needs to stop reusing the name, since there is no clear-and-replace on the API. Defining each name once behaves exactly as before.