Soft deprecation of the project?

[iliabylich]

So, Prism is out and it's:

1. correct
2. comes out of the box with MRI (and IIRC with other Ruby implementations)
3. faster (I think on TruffleRuby/JRuby it's absurdly faster because it doesn't run in interpreted mode like it does with parser)
4. always up-to-date
5. written in a language with perfect interop capabilities

On the other side backporting changes to parser becomes more and more difficult. MRI doesn't use yacc/bison anymore, instead there's Lrama that is also an LALR(1) parser generator but it has more "syntax sugar" that Racc doesn't have, and so the grammar will diverge more and more over time. And if (or once) parse.y is deprecated it will be even more complex to "translate" grammar changes.

I personally believe that parser should be soft-deprecated. What do you guys think? @kddnewton @bbatsov @koic @palkan @marcandre @mbj

And if you agree do you see anything we could do to parser to make transition easier?

[mbj]

The real use case of this gem left for me is multi version capability. Prism is not shipped with all the ruby versions my use cases still have to support, basically all non EOL (MRI) rubies.

If this gem where to stop supporting future ruby versions I'd be fine. Cannot speak for the rest of the ecosystem OFC.

[kddnewton]

In general I think this is the right long-term direction, but I'm uncomfortable soft deprecating until we mirror the tree rewriting functionality, since so many consumers use that. I think it could be added to the Ruby API of Prism relatively easily, but I haven't had time to look into this yet.

I'm not sure if there are any other APIs that I'm not thinking about, but that's the main one I know we are missing.

[bbatsov]

Indeed. I don't see how we can deprecated parser without the rewriting functionality. RuboCop also relies on the multi-version capability quite a lot, although I guess we can live without it if push comes to shove.

One other question - if parser gets soft deprecated will it still be updated to track newer Ruby releases? I think that a realistic time-frame for full migration to Prism is probably 2-3 years (for the projects that are still active, that is), so parser will definitely require some maintenance is the mean time.

[iliabylich]

One other question - if parser gets soft deprecated will it still be updated to track newer Ruby releases?

My initial thought was to stop doing it. At this point parser looks to me a lot like a (bad) polyfill and extending it more to be compliant with new grammar features feels like a disservice to the community.

[mbj]

I think we need to put evolutionary pressure on and NOT release compatibility with new versions.

[bbatsov]

I'm not super fond of measures that will be quite disruptive to the end users - to me it seems just wild to stop supporting something without providing a complete alternative for it. E.g. what happens if the rewriting is still not supported by Prism when Ruby 3.5 comes out?

At this point parser looks to me a lot like a (bad) polyfill and extending it more to be compliant with new grammar features feels like a disservice to the community.

Breaking existing apps is not exactly in the service of the community either. ;-)

I'm not arguing that parser should go away - I'm just concerned about abandoning the project prematurely and causing pain for the users of tool depending on it.

[mbj]

To play a bit of devils advocate here: How is not supporting a new future ruby release "breaking existing apps"?

[bbatsov]

Okay, "causing problems for the existing apps". I'm fairly certain RuboCop's users won't be exactly happy if we tell them we don't really support newer Ruby releases for whatever reasons. :-) And I'm not sure that simply stopping work on parser will motivate everyone to implement whatever is missing, etc. Also - how is the translation layer (Prism -> Parser) going to work when Parser stops tracking upstream changes and the available nodes in both libraries are out of sync?

Moving fully to Prism's native AST is going to be quite the time investment for many projects. I'd like for us to get there, but ideally I don't want us to be pressured into a timeline for this given that everyone working on RuboCop does this in their spare time.

[mbj]

Is there a translation layer from prosm to this parsers AST somewhere?

[palkan]

Is there a translation layer from prism to this parsers AST somewhere?

Prism supports AST translation for popular parsers: https://github.com/ruby/prism/tree/main/lib/prism/translation

And if (or once) parse.y is deprecated it will be even more complex to "translate" grammar changes.

I think, deprecating Parser as soon as parse.y is deprecated makes sense. Otherwise the effort required to port Ruby changes to Parser would be enormous.

how is the translation layer (Prism -> Parser) going to work when Parser stops tracking upstream changes and the available nodes in both libraries are out of sync?

From my experience, adding new nodes could be done relatively easy (compared to upgrading the grammar and lexer). I see, how, in theory, we can use Prism for newer Ruby versions as a parser and a Parser-compatible AST builder.

So, if we can keep Parser Ruby internals and APIs work in new Ruby versions (without providing the corresponding syntax support), that would smooth the transition.

To sum up, I would treat soft-deprecation as follows: stop adding new rubyXY.y files, but maintain compatibility with potential Ruby changes (I mean something breaking, like kwargs separation, etc.).

[iliabylich]

To sum up, I would treat soft-deprecation as follows: stop adding new rubyXY.y files, but maintain compatibility with potential Ruby changes (I mean something breaking, like kwargs separation, etc.).

Yes, that's more or less what I proposed (probably the word "deprecate" was a bit misleading). I'm not proposing to yank parser from rubygems.org, only to stop backporting new grammar changes.

I personally never touched rewriting API, shouldn't it "just work" with an AST constructed by Prism's compatibility layer?

Breaking existing apps is not exactly in the service of the community either. ;-)

My understanding is that it shouldn't break anything. If you are on Ruby 3.3+ use Prism (probably with a compatibility layer but it will be a part of the tooling, fully transparent for end users), otherwise use legacy parser gem. Or is it going to be a nightmare in terms of configuration and testing?

[palkan]

shouldn't it "just work" with an AST constructed by Prism's compatibility layer?

Yeah, it works right now. Well, at least, for Ruby Next.

I think, what could be done in terms of ensuring this compatibility is to make it possible to run rewriter tests via Prism and the translation layer. I can take a look at this (since I've already done the same for Ruby Next).

[mbj]

Is there an unparser equivalent yet?

[kddnewton]

Truthfully I don't know if parse.y is going to be deprecated, or if that will happen any time soon at all. I imagine it could stick around for quite a long time. That being said, it is getting further away from strict bison/racc syntax, so I know it's quite a maintenance burden to hold on to.

My understanding is that it shouldn't break anything. If you are on Ruby 3.3+ use Prism (probably with a compatibility layer but it will be a part of the tooling, fully transparent for end users), otherwise use legacy parser gem. Or is it going to be a nightmare in terms of configuration and testing?

I think this would work, but we will want to test the translation layer even more. I think there are issues with string escapes at the moment, but I'm not sure.

I think, what could be done in terms of ensuring this compatibility is to make it possible to run rewriter tests via Prism and the translation layer. I can take a look at this (since I've already done the same for Ruby Next).

That would be really great.

Is there an unparser equivalent yet?

I haven't managed to get it over the finish line yet, but yes https://github.com/ruby-syntax-tree/syntax_tree-prism is that work. It's almost done.

[iliabylich]

Is there an unparser equivalent yet?

Probably no, but I've found "untokenizer" made by @koic 😄

[bbatsov]

My understanding is that it shouldn't break anything. If you are on Ruby 3.3+ use Prism (probably with a compatibility layer but it will be a part of the tooling, fully transparent for end users), otherwise use legacy parser gem. Or is it going to be a nightmare in terms of configuration and testing?

To me it seems that'd be quite painful, especially if Prism doesn't support specifying target Ruby versions. And by "break" I mostly refer to "break the way" things are happening right now, not so much actually breaking existing apps. (and force clients to take some steps to ensure their tools will work with future Rubies that are not supported by parser) Anyways, I guess me and the rest of our team will have some thinking to do about the future and the next steps on our end.

To summarize, for us the biggest problems right now are:

• RuboCop allows users to specify which Ruby version they are targeting, regardless of the Ruby runtime and that's not currently supported by Prism (this was actually one of the things about parser I liked the most when I migrated RuboCop from ripper to parser in the early days of the project)
• We rely heavily on the re-writing and ideally this should be added to Prism. Otherwise we might consider pulling it into rubocop-ast or something along those lines

I'm guessing that @koic and @marcandre will have more to add on the subject.

[mbj]

Probably no, but I've found "untokenizer" made by @koic 😄

unfortunately not enough for my use case.

@bbatsov Okay so I saw a world where parser would stop to add new ruby version support, and everyone who needs newer ruby version support would than conditionally use Prism via a compatibiltiy layer. But your use case of running one ruby version and targeting another breaks this option.

All my use cases would be covered but not yours, curious if there are plans for a version target @kddnewton?

[marcandre]

I think the rewriter could definitely be included in rubocop-ast, or even better be a separate gem altogether.

[koic]

Much of what @bbatsov has already said aligns with my own thoughts, and I completely agree with his perspective. Specifically, I also share concerns about the impact on RuboCop users who rely on the Parser gem.

Direct Support for Prism Is Not Straightforward

First, I think the path to directly supporting Prism without going through Prism::Translation::Parser is quite challenging for RuboCop.

For instance, node pattern would require new support to describe the Prism AST, which differs from that of the Parser gem AST. Moreover, ensuring a seamless transition for users likely means supporting existing implementations written using the traditional Parser gem AST in parallel. So, It's important to note that changes in AST programming will affect not only RuboCop core developers but also its gem users.

And, node pattern is just one example, and there are some other aspects of design and implementation to consider.

The Importance of Prism::Translation::Parser

In reality, the transition will likely involve maintaining a Parser gem-compatible API via Prism::Translation::Parser for a while, gradually moving forward.

If the goal is to slowly wind down development of the Parser gem, then enhancing maintainability through Prism::Translation::Parser rather than starting with vanilla Prism would be a more practical approach, especially considering tools heavily dependent on the Parser gem.

It might be possible to design a transition where older Ruby versions continue to use the Parser gem, while newer Ruby versions rely on Prism::Translation::Parser, minimizing the impact on users.

So, if Prism::Translation::Parser becomes stable, even if the Parser gem stops tracking Ruby's parse.y in the future, Ruby syntax changes could still be handled through Prism::Translation::Parser.

Summary

Either way, since the future remains uncertain, I think compatibility maintaining Parser gem and Prism::Translation::Parser should be a priority for now.

[kddnewton]

All my use cases would be covered but not yours, curious if there are plans for a version target @kddnewton?

We have version targeting built into Prism right now (you can pass version: into the parse* family of functions).

[iliabylich]

I'm personally not going to backport changes anymore. If you want to keep this project up-to-date feel free to do so. I can still bump versions for existing ..3.3.X versions and fix potential deprecation warnings (for a few years or so, I understand that there are people that are stuck with older Rubies).

[bbatsov]

@iliabylich Fair enough. I definitely wouldn't expect you to be doing work you don't consider meaningful. I'm not sure who from our team has access to the repo, but it might be a good idea to make some of us co-maintainers, so we can keep supporting it as long as necessary for the needs of RuboCop.