Ruby 1.9.1 + antlr3 1.7.5 - mismatched character "+"; expecting "+"?

[cadmium-zz]

Hi Kyle,
I'm running the 1.7.5 version of the antlr3 gem, and I'm constantly running into the same kind of problem (it might just be because I'm a noob at antlr!)
I tried running your example code.

In file "AddingMachine.g" I put the following grammar:

grammar AddingMachine;

options {
  language = Ruby;
}

expression returns[ value ]
  : a=NUMBER '+' b=NUMBER { $value = $a.text.to_i + $b.text.to_i }
  | a=NUMBER '-' b=NUMBER { $value = $a.text.to_i - $b.to_i }
  ;

NUMBER: ( '0' .. '9' )+;

SPACE: ' '+ { $channel = HIDDEN };

To test it, I created a file "AddingMachine.rb" which contains:

require "rubygems"
require "AddingMachineLexer"
require "AddingMachineParser"

lexer = AddingMachine::Lexer.new( "1 + 1" )
tokens = ANTLR3::CommonTokenStream.new( lexer )
parser = AddingMachine::Parser.new( tokens )

I ran "antlr4ruby AddingMachine.g" to generate the lexer and parser files. Then when I ran "ruby AddingMachine.rb" however, I got the following error:

line 1:1 mismatched character " "; expecting " "
line 1:2 mismatched character "+"; expecting "+"
line 1:3 mismatched character " "; expecting " "
/home/cadmium/ruby/antlr/AddingMachineLexer.rb:162:in `between?': comparison of Fixnum with String failed (ArgumentError)
    from /home/cadmium/ruby/antlr/AddingMachineLexer.rb:162:in `number!'
    from /home/cadmium/ruby/antlr/AddingMachineLexer.rb:272:in `token!'
    from /var/lib/gems/1.9.1/gems/antlr3-1.7.5/lib/antlr3/recognizers.rb:980:in `block in next_token'
    from /var/lib/gems/1.9.1/gems/antlr3-1.7.5/lib/antlr3/recognizers.rb:971:in `loop'
    from /var/lib/gems/1.9.1/gems/antlr3-1.7.5/lib/antlr3/recognizers.rb:971:in `next_token'
    from /var/lib/gems/1.9.1/gems/antlr3-1.7.5/lib/antlr3/token.rb:291:in `next'
    from /var/lib/gems/1.9.1/gems/antlr3-1.7.5/lib/antlr3/token.rb:306:in `block in each'
    from /var/lib/gems/1.9.1/gems/antlr3-1.7.5/lib/antlr3/token.rb:306:in `loop'
    from /var/lib/gems/1.9.1/gems/antlr3-1.7.5/lib/antlr3/token.rb:306:in `each'
    from /var/lib/gems/1.9.1/gems/antlr3-1.7.5/lib/antlr3/streams.rb:749:in `to_a'
    from /var/lib/gems/1.9.1/gems/antlr3-1.7.5/lib/antlr3/streams.rb:749:in `initialize'
    from AddingMachine.rb:6:in `new'
    from AddingMachine.rb:6:in `'

I'm at a loss to understand what's going on so I'm hoping it makes sense to you! Did I make a mistake in how to use antlr3?

Thanks very much in advance,
Cadmium.

[cadmium-zz]

Also I forgot to mention that I'm using Ubuntu linux (10.4 Lucid Lynx version).

[ohboyohboyohboy]

Hey there Cadmium -- thanks for the bug report and the thorough details you took the time to include. The good news is that your code is correct, and you are using the correct procedure. However, the bad news is that I have not made ANTLR code function correctly in Ruby 1.9 yet. I targeted the code for 1.8.7 and Ruby implementations that are compatible with that version. I discovered later in the design process that the fact that statements like ?c and 'string'[ 3 ] produce string objects in 1.9 violates an assumption made in my state machine code design.

For straightforward ANTLR rules, the code usually works -- strings are still being compared with strings inside those long methods. The problem is that parser rules which require more complicated decisions use a special state machine object (see the ANTLR3::DFA class) to run through all possible matching rule patterns before choosing the correct parser or lexer rule to invoke. These state machines use a range of character code integers to decide the next decision to perform based on the current stream character.

So, inevitably when you run your calculator program, the lexer is calling on a DFA object to choose the best lexer rule the current position in the input text. In Ruby 1.8.7, the input stream is feeding the DFA with integer character values for each character in the string and everything works: integers are being compared against integers. In Ruby 1.9.1, the input stream is feeding the DFA string objects for each character. For some reason, the first few attempts at choosing a token just fail out and those "mismatched character" errors are printed out. Eventually, the DFA tries to check the string against a min and max integer character value by calling the #between? method, and that error is raised.

I know that explanation's a bit haphazard, but I hope it gives you some understanding of the error itself. I do plan on making it work with Ruby 1.9 very soon -- it shouldn't be too difficult, though it will probably require a bit of editing to the code generation templates and the runtime library code. I've just been too busy with work and other adult-like demands like that to get around to repairing that. I'll try to work it out this upcoming weekend.
Meanwhile, if you have 1.8.7 available, try it out with that -- it should work. Though I know if you're trying to use it for any project that requires 1.9, that solution doesn't help you out much. Let me know if you have any thoughts or suggestions, and feel free to send any other questions you have my way.

Best Regards,
Kyle

[cadmium-zz]

Hi Kyle,
Thanks so much for such a prompt and detailed reply. Unfortunately I only have Ruby 1.9.1 installed so I'll wait until the updated version is released (don't feel it's urgent to do though, I know how busy life is!) If I have some free time I might check out the source code you mentioned to see how those characters are being processed.
This error report should prove very informative to others who may be having the same problem (I noticed that no other bug report addressed it). I'll close the issue since it's been answered.
Thanks again,
Cadmium.

[ohboyohboyohboy]

Hey Cadmium,

Good news -- I fixed the Ruby 1.9 incompatibilities, as promised. I published a new gem (version 1.8.0). If you would like, check it out and see if it works for you. If you come across any bugs that weren't caught by the tests (which all passed with 1.9), let me know. I just figured I'd give you a heads up.

Best Regards,
Kyle

[cadmium-zz]

Thanks Kyle, it works great! Hopefully now more people will try antlr on Ruby!
Cheers Cadmium.

[ohboyohboyohboy]

Heya Cadmium,

Thanks -- I'm glad it did the trick for you. Let me know if anything else comes up along the way.

Kyle