Skip to content

Commit 4ac89dc

Browse files
committed
Fix an AST and token incompatibility for Prism::Translation::Parser
This PR fixes an AST and token incompatibility between Parser gem and `Prism::Translation::Parser` for empty xstring literal. ## Parser gem (Expected) ```console $ bundle exec ruby -Ilib -rparser/ruby33 -ve \ 'buf = Parser::Source::Buffer.new("/tmp/s.rb"); buf.source = "``"; p Parser::Ruby33.new.tokenize(buf)' ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-darwin22] [s(:xstr), [], [[:tXSTRING_BEG, ["`", #<Parser::Source::Range /tmp/s.rb 0...1>]], [:tSTRING_END, ["`", #<Parser::Source::Range /tmp/s.rb 1...2>]]]] ``` ## `Prism::Translation::Parser` (Actual) Previously, the AST and tokens returned by the Parser gem were different: ```console $ bunele exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \ 'buf = Parser::Source::Buffer.new("/tmp/s.rb"); buf.source = "``"; p Prism::Translation::Parser33.new.tokenize(buf)' ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-darwin22] [s(:xstr, s(:str, "")), [], [[:tBACK_REF2, ["`", #<Parser::Source::Range /tmp/s.rb 0...1>]], [:tSTRING_END, ["`", #<Parser::Source::Range /tmp/s.rb 1...2>]]]] ``` After this correction, the AST and tokens returned by the Parser gem are the same: ```console $ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \ 'buf = Parser::Source::Buffer.new("/tmp/s.rb"); buf.source = "``"; p Prism::Translation::Parser33.new.tokenize(buf)' ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-darwin22] [s(:xstr), [], [[:tXSTRING_BEG, ["`", #<Parser::Source::Range /tmp/s.rb 0...1>]], [:tSTRING_END, ["`", #<Parser::Source::Range /tmp/s.rb 1...2>]]]] ```
1 parent 4981233 commit 4ac89dc

File tree

4 files changed

+26
-10
lines changed

4 files changed

+26
-10
lines changed

lib/prism/translation/parser/compiler.rb

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1678,7 +1678,7 @@ def visit_x_string_node(node)
16781678
children, closing = visit_heredoc(node.to_interpolated)
16791679
builder.xstring_compose(token(node.opening_loc), children, closing)
16801680
else
1681-
parts = if node.unescaped.lines.count <= 1
1681+
parts = if node.unescaped.lines.one?
16821682
[builder.string_internal([node.unescaped, srange(node.content_loc)])]
16831683
else
16841684
start_offset = node.content_loc.start_offset

lib/prism/translation/parser/lexer.rb

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -326,7 +326,7 @@ def to_a
326326
type = :tIDENTIFIER
327327
end
328328
when :tXSTRING_BEG
329-
if (next_token = lexed[index][0]) && next_token.type != :STRING_CONTENT
329+
if (next_token = lexed[index][0]) && next_token.type != :STRING_CONTENT && next_token.type != :STRING_END
330330
type = :tBACK_REF2
331331
end
332332
end

test/prism/fixtures/xstring.txt

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,3 +7,7 @@
77
%x{
88
foo
99
}
10+
11+
``
12+
13+
%x{}

test/prism/snapshots/xstring.txt

Lines changed: 20 additions & 8 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

0 commit comments

Comments
 (0)