Skip to content

Commit

Permalink
Fix token incompatibility for Prism::Translation::Parser::Lexer
Browse files Browse the repository at this point in the history
This PR fixes token incompatibility for `Prism::Translation::Parser::Lexer` when using backquoted heredoc indetiner:

```ruby
<<-`  FOO`
a
b
     FOO
```

## Parser gem (Expected)

Returns `tXSTRING_BEG` as the first token:

```console
$ bundle exec ruby -Ilib -rparser/ruby33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = File.read("example.rb"); p Parser::Ruby33.new.tokenize(buf)'
ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-darwin22]
[s(:xstr,
  s(:str, "a\n"),
  s(:str, "b\n")), [], [[:tXSTRING_BEG, ["<<`", #<Parser::Source::Range example.rb 0...10>]],
[:tSTRING_CONTENT, ["a\n", #<Parser::Source::Range example.rb 11...13>]],
[:tSTRING_CONTENT, ["b\n", #<Parser::Source::Range example.rb 13...15>]],
[:tSTRING_END, ["  FOO", #<Parser::Source::Range example.rb 15...23>]], [:tNL, [nil, #<Parser::Source::Range example.rb 10...11>]]]]
```

## `Prism::Translation::Parser` (Actual)

Previously, the tokens returned by the Parser gem were different. The escaped backslash does not match in the `tSTRING_BEG` token and
value of `tSTRING_END` token.

```console
$ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = File.read("example.rb"); p Prism::Translation::Parser33.new.tokenize(buf)'
ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-darwin22]
[s(:xstr,
  s(:str, "a\n"),
  s(:str, "b\n")), [], [[:tSTRING_BEG, ["<<\"", #<Parser::Source::Range example.rb 0...10>]],
[:tSTRING_CONTENT, ["a\n", #<Parser::Source::Range example.rb 11...13>]],
[:tSTRING_CONTENT, ["b\n", #<Parser::Source::Range example.rb 13...15>]],
[:tSTRING_END, ["`  FOO`", #<Parser::Source::Range example.rb 15...23>]], [:tNL, [nil, #<Parser::Source::Range example.rb 10...11>]]]]
```

After this correction, the AST and tokens returned by the Parser gem are the same:

```console
$ bunlde exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = File.read("example.rb"); p Prism::Translation::Parser33.new.tokenize(buf)'
ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-darwin22]
[s(:xstr,
  s(:str, "a\n"),
  s(:str, "b\n")), [], [[:tXSTRING_BEG, ["<<`", #<Parser::Source::Range example.rb 0...10>]],
[:tSTRING_CONTENT, ["a\n", #<Parser::Source::Range example.rb 11...13>]],
[:tSTRING_CONTENT, ["b\n", #<Parser::Source::Range example.rb 13...15>]],
[:tSTRING_END, ["  FOO", #<Parser::Source::Range example.rb 15...23>]], [:tNL, [nil, #<Parser::Source::Range example.rb 10...11>]]]]
```
  • Loading branch information
koic committed Mar 16, 2024
1 parent 2dfbfba commit 308f8d8
Show file tree
Hide file tree
Showing 3 changed files with 38 additions and 22 deletions.
9 changes: 7 additions & 2 deletions lib/prism/translation/parser/lexer.rb
Original file line number Diff line number Diff line change
Expand Up @@ -278,7 +278,7 @@ def to_a
value = nil
when :tSTRING_BEG
if token.type == :HEREDOC_START
heredoc_identifier_stack.push(value.match(/<<[-~]?["']?(?<heredoc_identifier>.*?)["']?\z/)[:heredoc_identifier])
heredoc_identifier_stack.push(value.match(/<<[-~]?["'`]?(?<heredoc_identifier>.*?)["'`]?\z/)[:heredoc_identifier])
end
if ["\"", "'"].include?(value) && (next_token = lexed[index][0]) && next_token.type == :STRING_END
next_location = token.location.join(next_token.location)
Expand All @@ -294,7 +294,12 @@ def to_a
index += 2
elsif value.start_with?("<<")
quote = value[2] == "-" || value[2] == "~" ? value[3] : value[2]
value = "<<#{quote == "'" || quote == "\"" ? quote : "\""}"
if quote == "`"
type = :tXSTRING_BEG
value = "<<`"
else
value = "<<#{quote == "'" || quote == "\"" ? quote : "\""}"
end
end
when :tSTRING_CONTENT
unless (lines = token.value.lines).one?
Expand Down
5 changes: 5 additions & 0 deletions test/prism/fixtures/heredocs_leading_whitespace.txt
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,11 @@ a
b
FOO

<<-` FOO`
a
b
FOO

<<-' FOO'
a
b
Expand Down
46 changes: 26 additions & 20 deletions test/prism/snapshots/heredocs_leading_whitespace.txt

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

0 comments on commit 308f8d8

Please sign in to comment.