Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ripper error "lex_p < tokp" caused by multi-byte character #1417

Closed
DavidEGrayson opened this issue Jan 18, 2014 · 6 comments
Closed

Ripper error "lex_p < tokp" caused by multi-byte character #1417

DavidEGrayson opened this issue Jan 18, 2014 · 6 comments
Milestone

Comments

@DavidEGrayson
Copy link
Contributor

@DavidEGrayson DavidEGrayson commented Jan 18, 2014

While I was using YARD I found another bug in Ripper. The following code runs fine with ruby 2.0.0p0 in Windows:

# coding: UTF-8
require 'ripper'
Ripper.parse "[:µ]\n\n"

However, if I run it with JRuby 1.7.10 then I get a cryptic error. In the shell output below you can see the exact version of JRuby I am using and the error:

$ jruby -v && jruby jruby_bug.rb
jruby 1.7.10 (1.9.3p392) 2014-01-09 c4ecd6b on Java HotSpot(TM) 64-Bit Server VM 1.7.0_45-b18 [Windows 8-amd64]
RuntimeError: lex_p < tokp
   parse at org/jruby/ext/ripper/RubyRipper.java:306
   parse at c:/jruby-1.7.10/lib/ruby/1.9/ripper/core.rb:23
  (root) at jruby_bug.rb:3

This is probably a bug in Ripper because eval can happily process the same string that Ripper.parse chokes on.

(this was copied from issue #789)

enebo added a commit that referenced this issue Feb 21, 2014
@enebo enebo closed this in af68447 Feb 21, 2014
@enebo enebo added this to the JRuby 1.7.11 milestone Feb 21, 2014
@enebo
Copy link
Member

@enebo enebo commented Feb 21, 2014

@DavidEGrayson could you make sure you don't run into any more errors. I am slowly nailing these errors and this new mechanism for handling multibyte chars should fix a whole class of issues.

@headius
Copy link
Member

@headius headius commented Feb 22, 2014

I think the fixes for this caused CI regression.

@enebo
Copy link
Member

@enebo enebo commented Feb 22, 2014

yeah I thought I had excluded the single failure (it did not happen for me before I did the commit). I am going to fix it now though (it is non-trivial).

@enebo
Copy link
Member

@enebo enebo commented Feb 22, 2014

Just double-checked. The failures happening now have nothing to do with ripper? (I did exclude the single failure introduced)

@DavidEGrayson
Copy link
Contributor Author

@DavidEGrayson DavidEGrayson commented Mar 2, 2014

Thanks for fixing this @enebo! It's a bit tricky for me to go reproduce the original problem I saw on January 18th while using YARD, but I can confirm that JRuby 1.7.11 does not exhibit the specific, reduced bug I reported above.

@enebo
Copy link
Member

@enebo enebo commented Mar 3, 2014

Cool. I suspect you will eventually run into another multibyte char bug but for strings and regexp literals that last fix addressed the majority of cases.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants