Does not work on Rubinius

[NilsHaldenwang]

Hi there,

first of all: Nice work man!

But it seems not to work on Rubinius ( rbx-1.0.1-20100603 ), when trying to run my test-file I get:

An exception occurred running test.rb
Missing or uninitialized constant: ANTLR3::TokenSource::StopIteration (NameError)
Backtrace:
Module#const_missing at kernel/common/module.rb:538
ANTLR3::TokenSource(ERDL::Lexer)#each at /home/nils/.rvm/gems/rbx-1.0.1-      20100603/gems/antlr3-1.8.2/lib/antlr3/token.rb:330
      Enumerable(ERDL::Lexer)#to_a at kernel/common/enumerable.rb:871
ANTLR3::CommonTokenStream#initialize at /home/nils/.rvm/gems/rbx-1.0.1-20100603/gems/antlr3-1.8.2/lib/antlr3/streams.rb:794
ANTLR3::Parser(ERDL::Parser)#cast_input at /home/nils/.rvm/gems/rbx-1.0.1-20100603/gems/antlr3-1.8.2/lib/antlr3/recognizers.rb:1322
ANTLR3::Parser(ERDL::Parser)#initialize at /home/nils/.rvm/gems/rbx-1.0.1-20100603/gems/antlr3-1.8.2/lib/antlr3/recognizers.rb:1277
           ERDL::Parser#initialize at ERDLParser.rb:110
                   main.__script__ at test.rb:22
  Rubinius::CodeLoader#load_script at kernel/delta/codeloader.rb:67
  Rubinius::CodeLoader.load_script at kernel/delta/codeloader.rb:91
           Rubinius::Loader#script at kernel/loader.rb:435
             Rubinius::Loader#main at kernel/loader.rb:527
             Rubinius::Loader.main at kernel/loader.rb:553
                 Object#__script__ at kernel/loader.rb:565

But it works fine on 1.8.7

[NilsHaldenwang]

Okay, installed new Rubinius version, now I get:

An exception occurred running test_adding_machine.rb
TokenData is not a module (TypeError)

Backtrace:
        Rubinius.open_module_under at kernel/delta/rubinius.rb:82
              Rubinius.open_module at kernel/delta/rubinius.rb:95
        AddingMachine.__module_init__ (AddingMachine) at AddingMachineLexer.rb:66
                   main.__script__ at AddingMachineLexer.rb:60
         Rubinius::CodeLoader.require at kernel/common/codeloader.rb:145
            Kernel(Object)#require at kernel/common/kernel.rb:707
                   main.__script__ at test_adding_machine.rb:1
         Rubinius::CodeLoader#load_script at kernel/delta/codeloader.rb:67
         Rubinius::CodeLoader.load_script at kernel/delta/codeloader.rb:91
           Rubinius::Loader#script at kernel/loader.rb:460
             Rubinius::Loader#main at kernel/loader.rb:571
             Rubinius::Loader.main at kernel/loader.rb:609
                 Object#__script__ at kernel/loader.rb:621

[NilsHaldenwang]

Seems like this is rbx related problem, posted an issue at rbx as well.

Btw. seems to work fine at JRuby as well. :-)

[NilsHaldenwang]

dbussink: NilsH: looks like it uses some stuff rbx doesn't have yet
dbussink: NilsH: like StopIteration
NilsH: dbussink, ah, okay, so this means it will not work in near future ?
dbussink: NilsH: well, from what i can see the usage of StopIteration is not really necessary
dbussink: NilsH: it can really simply be refactored out and that even simplifies the code
dbussink: NilsH: Enumerable#next / Continuations aren't currently supported in rbx
dbussink NilsH: some more information on why using #next can be problematic / slow: http://archive.codehaus.org/lists/org.codehaus.jruby.dev/msg/f04d2210909251312q46bd51c0teacc4b0a8c417f0c@mail.gmail.com

[ohboyohboyohboy]

Hello Nils,

Thanks for the compliment -- I appreciate it. Also, thanks for taking the time to research this issue so thoroughly, as you certainly saved me the extra effort involved in researching Rubinius' level of support.

Admittedly, I designed my implementation squarely to be compatible with vanilla Ruby 1.8.7, adding 1.9 compatibility later on. Thus, while I've toyed around with how the library works with Rubinius, I still haven't tested it extensively on that platform.

Thus, I was not aware that Rubinius does not support StopIteration, which, as you probably know, is an exception that simply signals the inner-most loop that's currently executing to stop. Reviewing the code, I see that it is used in the module ANTLR3::TokenSource, which is used to make lexers and certain token streams act Enumerable. While it's mostly there for convenience, it is used by the various TokenStream classes to harvest tokens from lexers.

From what I see in the code so far, it should be a fairly simple fix. I will try and resolve it when I have a little extra time later in the week. If you would like a quick workaround in the meantime, edit the file`lib/antlr3/token.rb'. Change the following code (around line 327) to the code listed below it:

OLD CODE:

    def each
      block_given? or return enum_for( :each )
      loop { yield( self.next ) }
    rescue StopIteration
      return self
    end

NEW CODE:

    def each
      block_given? or return enum_for( :each )
      while token = next_token and not ( token.nil? || token.type == EOF )
        yield( token )
      end
      return self
    end

By the way, the comment made by the Rubinius developer concerning next' methods is generally correct, but actually does not apply in this situation. The defaultnext' method you get when you include Enumerable in a class uses Continuations to magically run the each' method in a step-by-step fashion instead of one single shot through the whole sequence. As a result, they do tend to be obnoxiously slow. However, in this situation, I am explicitly writing a sensiblenext' method specifically to avoid the slow version of the same method you get with Enumerable. Thus, I wouldn't worry about the assessment.

Finally, as I haven't tested the code extensively with jruby yet, I'm glad to hear that it appears to be working on that platform. Thank you for the information. If you encounter any issues with it, feel free to let me know. I don't recall any jruby no-nos off the top of my head, though I wouldn't be surprised if there is somewhere in the code that uses ObjectSpace iteration.

I'll have to run the tests on these platforms over the weekend and see if any issues come up. Again, thanks for the report. I'll let you know when I've uploaded a fix for the issue.

Kyle

[ohboyohboyohboy]

Actually, reading back on my post, I see a bit of redundancy in the code I suggested. I would try replacing it with this instead;

def each
  block_given? or return enum_for( :each )
  while token = next_token and token.type != EOF 
    yield( token )
  end
  return self
end

[NilsHaldenwang]

Hey there,

thx for the extensive answer and quickfix-tips. You dont have to thank me, I try to help good open source projects wherever I can. As I do not have time to dive into all of the codebases, I take it as my duty to fill good bug-reports if I recognize something is going wrong. :-)

Will report any JRuby-related problems coming through my way, as I plan on using your gem in a JRuby application first, and after this for another project on rbx.

But with the TokenData error we have to wait for the rbx-devs to fix the problem, as you can see here: http://github.com/evanphx/rubinius/issues/#issue/512 it seems to be a RBX-related thing. Yesterday dbussink told me in IRC he has an idea where this could be located and it should not be hard to fix.

By the way:
Perhaps we should suggest at the antlr-mailinglist, that they list your work as the recommended ruby-target, as the one they are linking to on the homepage seems to be abandoned,... Will do that this afternoon. :-)

Kind regards,
Nils

[NilsHaldenwang]

Update:
The Rubinius-Crew fixed the issue with the model, and they strongly recommended never to subclass module. I promised I will tell you. :-) They handed in this link as well: http://www.engineyard.com/blog/2010/rubinius-wants-to-help-you-make-ruby-better/

Also evan says he will support the StopIteration thing soon, but he also recommended NOT to use it.

By the way, your quickfix works so far in the cases I tested it. Thx for your work man!

Yeah, and if you wonder why I need to run your gem on rbx:
We would like to have a pure-ruby parser for www.fancy-lang.org , that can run directly at rbx.