Support for an array of token types in longer_alt property

[msujew]

Currently, longer_alt on token types only accepts a single other token type. However, let's say we have a division token like / in our grammar and single tokens for multiline and singleline comments, starting with /* and // respectively. This is common for every language that derives its syntax from C. Using longer_alt in the / token type currently allows us to select only one of the comment token types.

Of course, we could try to merge their definitions, thereby losing some important information. For example, in Langium, we plan to implement the LSP hover API by looking at the last multiline comment that directly precedes the AST element in question. Doing this generically requires us to rely on the token type name. However, this behavior breaks as soon as we have to merge the token type definitions. That outlines my use case for an array of token types in the longer_alt property.

This improvement should be non-breaking, as the API only has to be extended to look like longer_alt: TokenType | TokenType[]. Are you interested in seeing this change? I would gladly contribute a PR.

[bd82]

Hello @msujew

Are you interested in seeing this change? I would gladly contribute a PR.

Yes, it seems like a good upgrade.

Note the lexer runtime code is super optimized for performance (and thus "ugly").

• https://github.com/Chevrotain/chevrotain/blob/master/packages/chevrotain/src/scan/lexer_public.ts#L566-L605

But I think the performance implications of this feature would not be severe as
the added conditional/loop would only appear inside a pre-existing and uncommon condition.
Meaning that extra "work" that is only performed for "LONGER_ALT" tokens is probably fine.

Comparing dev vs latest release performance can be done using this benchmark:

• https://github.com/Chevrotain/chevrotain/tree/master/packages/chevrotain/benchmark_web

The array of token types capability should also be mentioned in these docs

• https://github.com/Chevrotain/chevrotain/blob/e6c1f2a600ac0a31384b426ea6591c480c4a4b91/packages/website/docs/features/token_alternative_matches.md

[msujew]

Note the lexer runtime code is super optimized for performance (and thus "ugly").

Yeah, I already looked into what needs to be changed previously. I'm used to writing ugly code though ;)

Thanks for setting me up with the required processes (especially the performance benchmark). Expect a PR from me within the next days.

[bd82]

Thanks @msujew I will review it sometime this week.

[bd82]

release in 9.1.0
https://www.npmjs.com/package/chevrotain/v/9.1.0