Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ripper error "lex_p < tokp" caused by multi-byte character #1417

Closed
DavidEGrayson opened this Issue Jan 18, 2014 · 6 comments

Comments

Projects
None yet
3 participants
@DavidEGrayson
Copy link
Contributor

DavidEGrayson commented Jan 18, 2014

While I was using YARD I found another bug in Ripper. The following code runs fine with ruby 2.0.0p0 in Windows:

# coding: UTF-8
require 'ripper'
Ripper.parse "[:µ]\n\n"

However, if I run it with JRuby 1.7.10 then I get a cryptic error. In the shell output below you can see the exact version of JRuby I am using and the error:

$ jruby -v && jruby jruby_bug.rb
jruby 1.7.10 (1.9.3p392) 2014-01-09 c4ecd6b on Java HotSpot(TM) 64-Bit Server VM 1.7.0_45-b18 [Windows 8-amd64]
RuntimeError: lex_p < tokp
   parse at org/jruby/ext/ripper/RubyRipper.java:306
   parse at c:/jruby-1.7.10/lib/ruby/1.9/ripper/core.rb:23
  (root) at jruby_bug.rb:3

This is probably a bug in Ripper because eval can happily process the same string that Ripper.parse chokes on.

(this was copied from issue #789)

enebo added a commit that referenced this issue Feb 21, 2014

@enebo enebo closed this in af68447 Feb 21, 2014

@enebo enebo added this to the JRuby 1.7.11 milestone Feb 21, 2014

@enebo

This comment has been minimized.

Copy link
Member

enebo commented Feb 21, 2014

@DavidEGrayson could you make sure you don't run into any more errors. I am slowly nailing these errors and this new mechanism for handling multibyte chars should fix a whole class of issues.

@headius

This comment has been minimized.

Copy link
Member

headius commented Feb 22, 2014

I think the fixes for this caused CI regression.

@enebo

This comment has been minimized.

Copy link
Member

enebo commented Feb 22, 2014

yeah I thought I had excluded the single failure (it did not happen for me before I did the commit). I am going to fix it now though (it is non-trivial).

@enebo

This comment has been minimized.

Copy link
Member

enebo commented Feb 22, 2014

Just double-checked. The failures happening now have nothing to do with ripper? (I did exclude the single failure introduced)

@DavidEGrayson

This comment has been minimized.

Copy link
Contributor Author

DavidEGrayson commented Mar 2, 2014

Thanks for fixing this @enebo! It's a bit tricky for me to go reproduce the original problem I saw on January 18th while using YARD, but I can confirm that JRuby 1.7.11 does not exhibit the specific, reduced bug I reported above.

@enebo

This comment has been minimized.

Copy link
Member

enebo commented Mar 3, 2014

Cool. I suspect you will eventually run into another multibyte char bug but for strings and regexp literals that last fix addressed the majority of cases.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.