Regression: "\8", "\9" - Unexpected token ILLEGAL

[zlobz]

2.4.1
See #1106
bf5c615

// Error: Line 1: Unexpected token ILLEGAL
var esprima = require('esprima');
esprima.parse('"\\8"');

Some valid strings that esprima currently does not parse:

"\8"
"\9"

[michaelficarra]

Those are not valid. If you feel otherwise, show me the grammar production that allows them.

[zlobz]

No, it is valid. "\8\9" works in the wild.

> "\8\9"
'89'

https://es5.github.io/#x16
An implementation may extend program syntax and regular expression pattern or flag syntax.

[dmethvin]

Seems like the \ DecimalDigit production would apply? Things are complicated a bit because in non-strict mode there's also LegacyOctalEscapeSequence but none of those productions apply since 8 is not an octal digit. In strict a valid octal sequence generates a syntax error but an invalid one like \8 silently converts to 8.

[zlobz]

Standard ECMA-262 June 1997 http://tecfa.unige.ch/guides/js/e262-pdf.pdf
7.7.4 String Literals

EscapeSequence ::
CharacterEscapeSequence
OctalEscapeSequence
HexEscapeSequence
UnicodeEscapeSequence

CharacterEscapeSequence ::
\ SingleEscapeCharacter
\ NonEscapeCharacter

NonEscapeCharacter::
SourceCharacter but not EscapeCharacter or LineTerminator

EscapeCharacter ::
SingleEscapeCharacter
OctalDigit
x
u

SingleEscapeCharacter :: one of
'"\ b f n r t

[mathiasbynens]

See whatwg/javascript#12 and especially https://bugs.ecmascript.org/show_bug.cgi?id=3477.

[dmethvin]

This is what i saw in ES2015 RC1:

SingleStringCharacter ::
\ EscapeSequence

EscapeSequence ::
CharacterEscapeSequence

CharacterEscapeSequence ::
NonEscapeCharacter

NonEscapeCharacter ::
SourceCharacter but not one of EscapeCharacter or LineTerminator

EscapeCharacter ::
SingleEscapeCharacter
DecimalDigit
x
u

[ariya]

@dmethvin Good analysis! That means \8 does not fulfill any production since 8 is a DecimalDigit and NonEscapeCharacter can't be it.

[zlobz]

@dmethvin

but an invalid one like \8 silently converts to 8.

because it is valid in old spec - ECMA-262 1997. since
8 is SourceCharacter
8 is not EscapeCharacter because it is not OctalDigit

NonEscapeCharacter ::
SourceCharacter but not one of EscapeCharacter or LineTerminator

EscapeCharacter ::
OctalDigit

[dmethvin]

Yeah, I meant \8 was invalid octal.

[ariya]

@xzo That's very old. ES 5.1 already specified EscapeCharacter to be exactly like above (see 7.8.4)

I don't think Esprima needs to support ECMAScript < 5 at all.

[zlobz]

Why parser doesn't need to support it, If modern ECMAScript engines support it? How to parse obfuscated code, for example?

[ariya]

Do we have some information as to how many scripts out there still utilizing such an escape sequence?

In all cases, even if we want to support, at best this is an error that should be tolerated, aka when tolerant is set to true in the parsing options.

[mathiasbynens]

This is on the table for ES7: https://bugs.ecmascript.org/show_bug.cgi?id=3477