ParseError with Ruby 3.3.x

[rdimartino]

Noticed that tests are failing with Ruby v3.3.0 and v3.3.1

Here is there error I get

  1) Error:
SyntaxTree::FormattingTest#test_layout:
SyntaxTree::Parser::ParseError: Invalid yield
    /.rbenv/versions/3.3.1/lib/ruby/gems/3.3.0/gems/syntax_tree-6.1.1/lib/syntax_tree/parser.rb:2864:in `on_parse_error'
    /.rbenv/versions/3.3.1/lib/ruby/gems/3.3.0/gems/syntax_tree-6.1.1/lib/syntax_tree.rb:123:in `parse'
    /.rbenv/versions/3.3.1/lib/ruby/gems/3.3.0/gems/syntax_tree-6.1.1/lib/syntax_tree.rb:123:in `parse'
    /syntax_tree-erb/lib/syntax_tree/erb/nodes.rb:467:in `initialize'
    /syntax_tree-erb/lib/syntax_tree/erb/nodes.rb:346:in `new'
    /syntax_tree-erb/lib/syntax_tree/erb/nodes.rb:346:in `rescue in prepare_content'
    /syntax_tree-erb/lib/syntax_tree/erb/nodes.rb:335:in `prepare_content'
    /syntax_tree-erb/lib/syntax_tree/erb/nodes.rb:298:in `initialize'
    /syntax_tree-erb/lib/syntax_tree/erb/parser.rb:589:in `new'
    /syntax_tree-erb/lib/syntax_tree/erb/parser.rb:589:in `parse_erb_tag'
    /syntax_tree-erb/lib/syntax_tree/erb/parser.rb:48:in `block (3 levels) in parse_any_tag'
    /syntax_tree-erb/lib/syntax_tree/erb/parser.rb:327:in `maybe'
    /syntax_tree-erb/lib/syntax_tree/erb/parser.rb:48:in `block (2 levels) in parse_any_tag'
    /syntax_tree-erb/lib/syntax_tree/erb/parser.rb:335:in `atleast'
    /syntax_tree-erb/lib/syntax_tree/erb/parser.rb:46:in `block in parse_any_tag'
    <internal:kernel>:187:in `loop'
    /syntax_tree-erb/lib/syntax_tree/erb/parser.rb:44:in `parse_any_tag'
    /syntax_tree-erb/lib/syntax_tree/erb/parser.rb:423:in `block in parse_html_element'
    /syntax_tree-erb/lib/syntax_tree/erb/parser.rb:347:in `block in many'
    <internal:kernel>:187:in `loop'
    /syntax_tree-erb/lib/syntax_tree/erb/parser.rb:345:in `many'
    /syntax_tree-erb/lib/syntax_tree/erb/parser.rb:423:in `parse_html_element'
    /syntax_tree-erb/lib/syntax_tree/erb/parser.rb:49:in `block (3 levels) in parse_any_tag'
    /syntax_tree-erb/lib/syntax_tree/erb/parser.rb:327:in `maybe'
    /syntax_tree-erb/lib/syntax_tree/erb/parser.rb:49:in `block (2 levels) in parse_any_tag'
    /syntax_tree-erb/lib/syntax_tree/erb/parser.rb:335:in `atleast'
    /syntax_tree-erb/lib/syntax_tree/erb/parser.rb:46:in `block in parse_any_tag'
    <internal:kernel>:187:in `loop'
    /syntax_tree-erb/lib/syntax_tree/erb/parser.rb:44:in `parse_any_tag'
    /syntax_tree-erb/lib/syntax_tree/erb/parser.rb:423:in `block in parse_html_element'
    /syntax_tree-erb/lib/syntax_tree/erb/parser.rb:347:in `block in many'
    <internal:kernel>:187:in `loop'
    /syntax_tree-erb/lib/syntax_tree/erb/parser.rb:345:in `many'
    /syntax_tree-erb/lib/syntax_tree/erb/parser.rb:423:in `parse_html_element'
    /syntax_tree-erb/lib/syntax_tree/erb/parser.rb:49:in `block (3 levels) in parse_any_tag'
    /syntax_tree-erb/lib/syntax_tree/erb/parser.rb:327:in `maybe'
    /syntax_tree-erb/lib/syntax_tree/erb/parser.rb:49:in `block (2 levels) in parse_any_tag'
    /syntax_tree-erb/lib/syntax_tree/erb/parser.rb:335:in `atleast'
    /syntax_tree-erb/lib/syntax_tree/erb/parser.rb:46:in `block in parse_any_tag'
    <internal:kernel>:187:in `loop'
    /syntax_tree-erb/lib/syntax_tree/erb/parser.rb:44:in `parse_any_tag'
    /syntax_tree-erb/lib/syntax_tree/erb/parser.rb:423:in `block in parse_html_element'
    /syntax_tree-erb/lib/syntax_tree/erb/parser.rb:347:in `block in many'
    <internal:kernel>:187:in `loop'
    /syntax_tree-erb/lib/syntax_tree/erb/parser.rb:345:in `many'
    /syntax_tree-erb/lib/syntax_tree/erb/parser.rb:423:in `parse_html_element'
    /syntax_tree-erb/lib/syntax_tree/erb/parser.rb:49:in `block (3 levels) in parse_any_tag'
    /syntax_tree-erb/lib/syntax_tree/erb/parser.rb:327:in `maybe'
    /syntax_tree-erb/lib/syntax_tree/erb/parser.rb:49:in `block (2 levels) in parse_any_tag'
    /syntax_tree-erb/lib/syntax_tree/erb/parser.rb:335:in `atleast'
    /syntax_tree-erb/lib/syntax_tree/erb/parser.rb:46:in `block in parse_any_tag'
    <internal:kernel>:187:in `loop'
    /syntax_tree-erb/lib/syntax_tree/erb/parser.rb:44:in `parse_any_tag'
    /syntax_tree-erb/lib/syntax_tree/erb/parser.rb:27:in `block in parse'
    /syntax_tree-erb/lib/syntax_tree/erb/parser.rb:347:in `block in many'
    <internal:kernel>:187:in `loop'
    /syntax_tree-erb/lib/syntax_tree/erb/parser.rb:345:in `many'
    /syntax_tree-erb/lib/syntax_tree/erb/parser.rb:27:in `parse'
    /syntax_tree-erb/lib/syntax_tree/erb.rb:21:in `parse'
    /syntax_tree-erb/lib/syntax_tree/erb.rb:17:in `block in format'
    /.rbenv/versions/3.3.1/lib/ruby/gems/3.3.0/gems/prettier_print-1.2.1/lib/prettier_print.rb:385:in `format'
    /syntax_tree-erb/lib/syntax_tree/erb.rb:17:in `format'
    /syntax_tree-erb/test/formatting_test.rb:44:in `assert_formatting'
    /syntax_tree-erb/test/formatting_test.rb:32:in `test_layout'

I did a little digging and syntax_tree-erb v0.10.2 doesn't have the ParseError with Ruby 3.3.1 (but the layout test still fails for a difference in formatting). v0.10.3 through v0.11.0 all have the ParseError

[rdimartino]

Actually seems to be an upstream issue with Ruby 3.3.x and syntax_tree.

❯ rbenv shell 3.2.4
❯ stree ast -e 'yield'
(program (statements ((yield nil))))
❯ rbenv shell 3.3.0
❯ stree ast -e 'yield'
Error: Invalid yield
> 1 | yield
    | ^

[rdimartino]

Hmmm. Seems like it's actually upstream of syntax_tree too. Appears to be from Ripper, but now I'm out of my depth

❯ rbenv shell 3.2.4    
❯ ruby -e 'require "ripper"; puts Ripper.sexp("yield", raise_errors: true)'
program
yield0

❯ rbenv shell 3.3.0                                                        
❯ ruby -e 'require "ripper"; puts Ripper.sexp("yield", raise_errors: true)'
/Users/robertdimartino/.rbenv/versions/3.3.0/lib/ruby/3.3.0/ripper/sexp.rb:40:in `sexp': Invalid yield (SyntaxError)
	from -e:1:in `<main>'

[rdimartino]

Aha! It's a known bug https://bugs.ruby-lang.org/issues/20186

See also ruby-formatter/rufo#319

[davidwessman]

Just realized I had notifications off, I just merge a version were I put a placeholder in place when parsing + formatting.

Also discussed it with Jean Boussier from Shopify and created an issue with syntax_tree:
ruby-syntax-tree/syntax_tree#457