Is using an empty token pattern to pop_mode for unterminated interpolation supported?

[ap32]

Hi, I have a question about Chevrotain lexer modes.

For string templates like:

"foo {{ bar }} baz"

I use a separate mode for the {{ ... }} section. The issue is malformed input like:

"foo {{ bar baz"

If }} is missing, the lexer may stay in the inner mode and consume too much input.

I found that this seems to work:

createToken({
  name: "BrokenTemplateExit",
  pattern: "",
  pop_mode: true
})

So I am using an empty token as a fallback to pop the mode.

Is this actually supported, or is it accidental / a bug? If I rely on it, what problems could it cause? For example, could it lead to non-advancing lexing, infinite loops, or broken recovery?

Also, what is the recommended way to recover from an unterminated mode like this in Chevrotain?

[bd82]

I don't think I thought of lexer error recovery in combination with token modes.

I naively suspect this empty token could theoretically cause problems.
But logically if it can only appear in "inner" modes and the side effect is to pop mode
Then it should not cause an infinite loop.

In the worst case you can always roll your own lexer which produces Chevrotain like tokens
and add any custom logic.

Another (verbose) options

Duplicate the definitions all the tokens from the "parent" mode
as pop tokens in the inner mode.
So the first token encountered which does not belong in the inner mode will cause the lexer to resume
parsing in the outer mode.

const BazTok = createToken({
  name: "Baz",
  pattern: "baz",
  pop_mode: true
})

createToken({
  name: "BazInner",
  pattern: "baz",
  // pop to exit inner mode
  pop_mode: true
  // category allows BazInner to match in the Parser where `BazTok` would also match
  // https://chevrotain.io/docs/features/token_categories.html#token-categories
  categories: [BazTok],
})

Preprocess: Maybe its not a Chevrotain Lexer problem.

This seems like cleanest approach with the most powerful possibilities.

If your problem is matching brackets, You could run your own separate mini lexer first which
matches brackets with a stack and has custom logic to either:

1. Remove sections of the input where brackets do not match.
2. Add "virtual" brackets if you can safely identify where they are missing.

Note this will cause offsets and line/column position to be invalid
But it seems like a solvable problem.