JS Lexer exit with error on \uFFFD

[Shnitzelil]

Parsing JS with some script that in some point we have \uFFFD.
The "Next()" function return with the next error:
unexpected � on line 135 and column 5
135: �});
^

Any idea how to handle it better?

This came up after upgrading from 2.4.4 to 2.7.13.

Thanks

[tdewolff]

Hi Kobi, that is because the \uFFFD character cannot be the start of an identifier (or its continuation), which would be the only possibility to have that character present (except from having that character in string or template literals). So unfortunately, the lexer is right to throw an error in this case. What use-case do you have? Could you simply ignore this error?

[Shnitzelil]

Hi Kobi, that is because the \uFFFD character cannot be the start of an identifier (or its continuation), which would be the only possibility to have that character present (except from having that character in string or template literals). So unfortunately, the lexer is right to throw an error in this case. What use-case do you have? Could you simply ignore this error?

This came from a customer that in one of his files he have this character...
Anyway, I can ignore it... is there a way to clear the error from the Lexer once it was "handled"?
thanks

[tdewolff]

Unfortunately, the client's file is invalid JS and will throw an error with any spec compliant parser. I've changed the lexer to clear the error if you decide to continue lexing after an error occurred. that should help you out.

[Shnitzelil]

Thank you.
Will check it once there will be released 👍
Thanks again!

[Shnitzelil]

@tdewolff ,
When do you plan to release new build?

[Shnitzelil]

Unfortunately, the client's file is invalid JS and will throw an error with any spec compliant parser. I've changed the lexer to clear the error if you decide to continue lexing after an error occurred. that should help you out.

How can I know how much to skip? coz when I try I use the Next() it keep trying to tokenize the same character over and over.

[tdewolff]

Should be working now in v2.7.14! Please let me know how it goes.