Ripper error "lex_p < tokp" caused by multi-byte character #1417

Closed
DavidEGrayson opened this Issue Jan 18, 2014 · 6 comments

Projects

None yet

3 participants

@DavidEGrayson
Contributor

While I was using YARD I found another bug in Ripper. The following code runs fine with ruby 2.0.0p0 in Windows:

# coding: UTF-8
require 'ripper'
Ripper.parse "[:µ]\n\n"

However, if I run it with JRuby 1.7.10 then I get a cryptic error. In the shell output below you can see the exact version of JRuby I am using and the error:

$ jruby -v && jruby jruby_bug.rb
jruby 1.7.10 (1.9.3p392) 2014-01-09 c4ecd6b on Java HotSpot(TM) 64-Bit Server VM 1.7.0_45-b18 [Windows 8-amd64]
RuntimeError: lex_p < tokp
   parse at org/jruby/ext/ripper/RubyRipper.java:306
   parse at c:/jruby-1.7.10/lib/ruby/1.9/ripper/core.rb:23
  (root) at jruby_bug.rb:3

This is probably a bug in Ripper because eval can happily process the same string that Ripper.parse chokes on.

(this was copied from issue #789)

@enebo enebo added a commit that referenced this issue Feb 21, 2014
@enebo enebo Fixes #1417. Ripper error with mbc 669017a
@enebo enebo closed this in af68447 Feb 21, 2014
@enebo enebo added this to the JRuby 1.7.11 milestone Feb 21, 2014
@enebo
Member
enebo commented Feb 21, 2014

@DavidEGrayson could you make sure you don't run into any more errors. I am slowly nailing these errors and this new mechanism for handling multibyte chars should fix a whole class of issues.

@headius
Member
headius commented Feb 22, 2014

I think the fixes for this caused CI regression.

@enebo
Member
enebo commented Feb 22, 2014

yeah I thought I had excluded the single failure (it did not happen for me before I did the commit). I am going to fix it now though (it is non-trivial).

@enebo
Member
enebo commented Feb 22, 2014

Just double-checked. The failures happening now have nothing to do with ripper? (I did exclude the single failure introduced)

@DavidEGrayson
Contributor

Thanks for fixing this @enebo! It's a bit tricky for me to go reproduce the original problem I saw on January 18th while using YARD, but I can confirm that JRuby 1.7.11 does not exhibit the specific, reduced bug I reported above.

@enebo
Member
enebo commented Mar 3, 2014

Cool. I suspect you will eventually run into another multibyte char bug but for strings and regexp literals that last fix addressed the majority of cases.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment