Bad LexToken(STRING) or invalid javascript? #39

Open
nuklea opened this Issue Oct 19, 2012 · 4 comments

Comments

Projects
None yet
4 participants

nuklea commented Oct 19, 2012

$ echo "var SohoPriceSlideData = { min_price : '\3,000' };" | slimit

Illegal character "'" at 1:39 after LexToken(COLON,':',1,37)
Illegal character '\\' at 1:40 after LexToken(COLON,':',1,37)
Illegal character "'" at 1:46 after LexToken(NUMBER,'000',1,43)
Traceback (most recent call last):
  File "/home/nuklea/.virtualenvs/project/bin/slimit", line 9, in <module>
    load_entry_point('slimit==0.7.4', 'console_scripts', 'slimit')()
  File "/home/nuklea/.virtualenvs/project/lib/python2.7/site-packages/slimit/minifier.py", line 69, in main
    text, mangle=options.mangle, mangle_toplevel=options.mangle_toplevel)
  File "/home/nuklea/.virtualenvs/project/lib/python2.7/site-packages/slimit/minifier.py", line 38, in minify
    tree = parser.parse(text)
  File "/home/nuklea/.virtualenvs/project/lib/python2.7/site-packages/slimit/parser.py", line 93, in parse
    return self.parser.parse(text, lexer=self.lexer, debug=debug)
  File "/home/nuklea/.virtualenvs/project/lib/python2.7/site-packages/ply/yacc.py", line 265, in parse
    return self.parseopt_notrack(input,lexer,debug,tracking,tokenfunc)
  File "/home/nuklea/.virtualenvs/project/lib/python2.7/site-packages/ply/yacc.py", line 1047, in parseopt_notrack
    tok = self.errorfunc(errtoken)
  File "/home/nuklea/.virtualenvs/project/lib/python2.7/site-packages/slimit/parser.py", line 116, in p_error
    self._raise_syntax_error(token)
  File "/home/nuklea/.virtualenvs/project/lib/python2.7/site-packages/slimit/parser.py", line 89, in _raise_syntax_error
    self.lexer.prev_token, self.lexer.token())
SyntaxError: Unexpected token (SEMI, ';') at 1:48 between LexToken(NUMBER,'000',1,43) and LexToken(RBRACE,'}',1,48)

nuklea commented Oct 22, 2012

@dkrnl ты про javascript или про slimit?

rspivak was assigned Oct 22, 2012

Owner

rspivak commented Oct 22, 2012

Thanks. I'll look into it.

qur2 commented Feb 12, 2013

Just stumbled on the same issue:

echo "a='oui\0'" | slimit

Illegal character "'" at 1:2 after LexToken(EQ,'=',1,1)
Illegal character '\\' at 1:6 after LexToken(ID,'oui',1,3)
Illegal character "'" at 1:8 after LexToken(NUMBER,'0',1,7)
Traceback (most recent call last):
  File "~/.virtualenvs/the-env/bin/slimit", line 8, in <module>
    load_entry_point('slimit==0.7.4', 'console_scripts', 'slimit')()
  File "~/.virtualenvs/the-env/lib/python2.7/site-packages/slimit/minifier.py", line 69, in main
    text, mangle=options.mangle, mangle_toplevel=options.mangle_toplevel)
  File "~/.virtualenvs/the-env/lib/python2.7/site-packages/slimit/minifier.py", line 38, in minify
    tree = parser.parse(text)
  File "~/.virtualenvs/the-env/lib/python2.7/site-packages/slimit/parser.py", line 93, in parse
    return self.parser.parse(text, lexer=self.lexer, debug=debug)
  File "~/.virtualenvs/the-env/lib/python2.7/site-packages/ply/yacc.py", line 265, in parse
    return self.parseopt_notrack(input,lexer,debug,tracking,tokenfunc)
  File "~/.virtualenvs/the-env/lib/python2.7/site-packages/ply/yacc.py", line 1047, in parseopt_notrack
    tok = self.errorfunc(errtoken)
  File "~/.virtualenvs/the-env/lib/python2.7/site-packages/slimit/parser.py", line 116, in p_error
    self._raise_syntax_error(token)
  File "~/.virtualenvs/the-env/lib/python2.7/site-packages/slimit/parser.py", line 89, in _raise_syntax_error
    self.lexer.prev_token, self.lexer.token())
SyntaxError: Unexpected token (NUMBER, '0') at 1:7 between LexToken(ID,'oui',1,3) and None

However, this gives 3 Illegal character warnings but succeeds:

$ echo "a='\0'" | slimit

Illegal character "'" at 1:2 after LexToken(EQ,'=',1,1)
Illegal character '\\' at 1:3 after LexToken(EQ,'=',1,1)
Illegal character "'" at 1:5 after LexToken(NUMBER,'0',1,4)
a=0;

Same here:

$ echo "'\0'" | slimit

Illegal character "'" at 1:0 after None
Illegal character '\\' at 1:1 after None
Illegal character "'" at 1:3 after LexToken(NUMBER,'0',1,2)
0;

With minimal string, I get the warnings and no output:

$ echo "'\'" | slimit

Illegal character "'" at 1:0 after None
Illegal character '\\' at 1:1 after None
Illegal character "'" at 1:2 after None

same issue, testing with 'sting'.replace('some', '\0').split('\0');

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment