Skip to content

Commit

Permalink
[ruby/prism] Fix a token incompatibility for `Prism::Translation::Par…
Browse files Browse the repository at this point in the history
…ser::Lexer`

This PR fixes a token incompatibility between Parser gem and `Prism::Translation::Parser` for
beginless range:

## Parser gem (Expected)

Returns `tBDOT2` token:

```console
$ bundle exec ruby -Ilib -rparser/ruby33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "..42"; p Parser::Ruby33.new.tokenize(buf)[2]'
ruby 3.3.0 (2023-12-25 revision ruby/prism@5124f9ac75) [x86_64-darwin22]
[[:tBDOT2, ["..", #<Parser::Source::Range example.rb 0...2>]], [:tINTEGER, [42, #<Parser::Source::Range example.rb 2...4>]]]
```

## `Prism::Translation::Parser` (Actual)

Previously, the parser returned `tDOT2` token when parsing the following:

```console
$ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "..42"; p Prism::Translation::Parser33.new.tokenize(buf)[2]'
ruby 3.3.0 (2023-12-25 revision ruby/prism@5124f9ac75) [x86_64-darwin22]
[[:tDOT2, ["..", #<Parser::Source::Range example.rb 0...2>]], [:tINTEGER, [42, #<Parser::Source::Range example.rb 2...4>]]]
```

After the update, the parser now returns `tBDOT2` token for the same input:

```console
$ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "..42"; p Prism::Translation::Parser33.new.tokenize(buf)[2]'
ruby 3.3.0 (2023-12-25 revision ruby/prism@5124f9ac75) [x86_64-darwin22]
[[:tBDOT2, ["..", #<Parser::Source::Range example.rb 0...2>]], [:tINTEGER, [42, #<Parser::Source::Range example.rb 2...4>]]]
```

This correction enables the restoration of `endless_range_in_conditional.txt` as a test case.

ruby/prism@f624b99ab0
  • Loading branch information
koic authored and matzbot committed Mar 12, 2024
1 parent 1e7ee87 commit 72a5707
Show file tree
Hide file tree
Showing 2 changed files with 1 addition and 2 deletions.
2 changes: 1 addition & 1 deletion lib/prism/translation/parser/lexer.rb
Expand Up @@ -167,7 +167,7 @@ class Lexer
TILDE: :tTILDE,
UAMPERSAND: :tAMPER,
UCOLON_COLON: :tCOLON3,
UDOT_DOT: :tDOT2,
UDOT_DOT: :tBDOT2,
UDOT_DOT_DOT: :tBDOT3,
UMINUS: :tUMINUS,
UMINUS_NUM: :tUNARY_NUM,
Expand Down
1 change: 0 additions & 1 deletion test/prism/parser_test.rb
Expand Up @@ -73,7 +73,6 @@ class ParserTest < TestCase
skip_tokens = [
"comments.txt",
"constants.txt",
"endless_range_in_conditional.txt",
"heredoc_with_comment.txt",
"heredoc_with_escaped_newline_at_start.txt",
"heredocs_leading_whitespace.txt",
Expand Down

0 comments on commit 72a5707

Please sign in to comment.