New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ripper fails to lexical analyse around escaped newline on JRuby #4787

Closed
aycabta opened this Issue Sep 10, 2017 · 2 comments

Comments

Projects
None yet
2 participants
@aycabta

aycabta commented Sep 10, 2017

The code below is valid on JRuby 9.1.13.0, CRuby 2.4.1 and more old days:

1 == 2 ? \
  puts(true) : \
  puts(false)

But Ripper on JRuby fails to lexical analyse source code around escaped newline. It returns strange tokens.

I'm in need when using Ripper for RDoc.

Environment

$ uname -a
Linux x250 4.11.6-3-ARCH #1 SMP PREEMPT Thu Jun 22 12:21:46 CEST 2017 x86_64 GNU/Linux
$ ruby -v
jruby 9.1.13.0 (2.3.3) 2017-09-06 8e1c115 OpenJDK 64-Bit Server VM 25.131-b11 on 1.8.0_131-b11 +jit [linux-x86_64]

Source Code

require 'ripper'
require 'pp'

pp Ripper.lex("1 == 2 ? \\\n  puts(true) : \\\n  puts(false)")

Expected Behavior

This is on CRuby 2.4.1.

[[[1, 0], :on_int, "1"],
 [[1, 1], :on_sp, " "],
 [[1, 2], :on_op, "=="],
 [[1, 4], :on_sp, " "],
 [[1, 5], :on_int, "2"],
 [[1, 6], :on_sp, " "],
 [[1, 7], :on_op, "?"],
 [[1, 8], :on_sp, " "],
 [[1, 9], :on_sp, "\\\n"],
 [[2, 0], :on_sp, "  "],
 [[2, 2], :on_ident, "puts"],
 [[2, 6], :on_lparen, "("],
 [[2, 7], :on_kw, "true"],
 [[2, 11], :on_rparen, ")"],
 [[2, 12], :on_sp, " "],
 [[2, 13], :on_op, ":"],
 [[2, 14], :on_sp, " "],
 [[2, 15], :on_sp, "\\\n"],
 [[3, 0], :on_sp, "  "],
 [[3, 2], :on_ident, "puts"],
 [[3, 6], :on_lparen, "("],
 [[3, 7], :on_kw, "false"],
 [[3, 12], :on_rparen, ")"]]

Actual Behavior

This is on JRuby 9.1.3.0 as described previously.

[[[1, 0], :on_int, "1"],
 [[1, 1], :on_sp, " "],
 [[1, 2], :on_op, "=="],
 [[1, 4], :on_sp, " "],
 [[1, 5], :on_int, "2"],
 [[1, 6], :on_sp, " "],
 [[1, 7], :on_op, "?"],
 [[1, 8], :on_sp, " "],
 [[1, 9], :on_ident, "\\\n"],
 [[2, 0], :on_sp, "  "],
 [[2, 2], :on_lparen, "puts("],
 [[2, 7], :on_kw, "true"],
 [[2, 11], :on_rparen, ")"],
 [[2, 12], :on_sp, " "],
 [[2, 13], :on_op, ":"],
 [[2, 14], :on_ident, "\\\n"],
 [[3, 0], :on_sp, "  "],
 [[3, 2], :on_lparen, "puts("],
 [[3, 7], :on_kw, "false"],
 [[3, 12], :on_rparen, ")"]]

aycabta added a commit to aycabta/rdoc that referenced this issue Sep 10, 2017

Skip a test on JRuby
The test_accept_verbatim_nl_after_backslash is failed on JRuby
because Ripper on JRuby fails to lexical analyse around escaped newline.
I filed a issue for it: jruby/jruby#4787
The "skip" should be removed after the issue is resolved.
@enebo

This comment has been minimized.

Show comment
Hide comment
@enebo

enebo Oct 30, 2017

Member

If I put original heredoc into string like:

contents = <<EOS
1 == 2 ? \
  puts(true) : \
  puts(false)
EOS

then all is happy. when put into a interpolated string with \ then the lexer will actually see the \ character in the stream and it all gets weird. Makes me wonder how well we actually execute that code too...

Member

enebo commented Oct 30, 2017

If I put original heredoc into string like:

contents = <<EOS
1 == 2 ? \
  puts(true) : \
  puts(false)
EOS

then all is happy. when put into a interpolated string with \ then the lexer will actually see the \ character in the stream and it all gets weird. Makes me wonder how well we actually execute that code too...

enebo added a commit that referenced this issue Oct 30, 2017

Fixes #4787 Ripper fails to lexical analyse around escaped newline on…
… JRuby.

Missing a dispatch token (tSP) when we see \\ and then \n.

@enebo enebo added this to the JRuby 9.1.14.0 milestone Oct 30, 2017

@enebo enebo closed this in 3b6f308 Oct 30, 2017

@enebo enebo added the parser label Oct 30, 2017

@aycabta

This comment has been minimized.

Show comment
Hide comment
@aycabta

aycabta Oct 30, 2017

Thank you!

aycabta commented Oct 30, 2017

Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment