Unexpected character error does not show correct column

[baodart]

Here's a reproduction example:

import { createToken, Lexer } from 'chevrotain'

const WHITESPACE = createToken({ name: 'WHITESPACE', pattern: /\s+/, group: Lexer.SKIPPED })
const NUMBER = createToken({ name: 'NUMBER', pattern: /(0|[1-9]\d*)(\.\d+)?/ })

const lexer = new Lexer([WHITESPACE, NUMBER], { ensureOptimizations: true })

const { tokens, errors } = lexer.tokenize('-"5"')

console.log('tokens', tokens)
console.log('errors', errors)

which logs

As the lexer can only understand whitespace and number literal, only 5 is captured as tokens here. There are 2 'unexpected character' errors:

• The - with offset 0 and column 1 (correct)
• The (second) " with offset 3 but column 2 (incorrect as 2 is the column of the first ") instead of 4

[bd82]

Thanks for reporting this @baodart and providing a minimal reproduction 👍

There is another issue here, the valid token "5" column is reported as 1 instead of 3
I suspect that the logic to keep track of column numbers in case of lexer error recovery.

• https://github.com/Chevrotain/chevrotain/blob/master/packages/chevrotain/src/scan/lexer_public.ts#L693-L694

[bd82]

fix was released in 11.0.3 version available on npm