Underscores in tokens break parser (but not lexer).

[jimsynz]

I'm working on a ruby runtime for the Handlebars.js templating language. I ran into an issue where the parser erroneously raises a string not in language exception when tokens contain underscores.

Changing the following rule:

rule /{{/, :default do                                                                       
  push_state :expression
  [ :EXPRESSION_START ]
end

to

rule /{{/, :default do                                                                       
  push_state :expression
  [ :EXPRESSIONSTART ]
end

Allows the parser to succeed.

I have prepared examples of working and non-working lexers, parsers and specs for your enjoyment at jamesotron/FlavourSaver on the broken_parser and working_parser branches. In both branches if you run the specs you will find that the lexer output passes but the parser will change behaviour.

Thanks for RLTK, it's an awesome library!

[chriswailes]

Fixed in the latest patch.

[jimsynz]

Great! Thanks Chris!