Needed: Guide to migrate ripper-dependent libraries to use parser

[bf4]

Libraries that use 'ripper' are cruby-only. Switching them to use parser would make them compatible with other rubies. We should put together a guide for how to achieve this migration, and any costs/benefits of doing so.

[whitequark]

Interesting. I can easily see value in such a guide, but the approaches used by ripper and parser are radically different; I'm pretty certain that such libraries will have to be ported on a case-by-case basis. However I'd be interested to hear your thoughts.

Benjamin Fleischer notifications@github.com wrote:

Libraries that use 'ripper' are cruby-only. Switching them to use parser would make them compatible with other rubies. We should put together a guide for how to achieve this migration, and any costs/benefits of doing so.

—
Reply to this email directly or view it on GitHub.

[bf4]

Well, I maintain metric_fu and some of the libraries I depend on use ripper, which limits my using them to mri, which is annoying. They are rails_best_practices, flog, and cane

[yorickpeterse]

I could take a look at it since I'm currently undergoing the same process.

[bf4]

@yorickpeterse I'm guessing you're specifically thinking of yorickpeterse/ruby-lint@ab442c5

[yorickpeterse]

Yes, a work in progress of this can be seen on this branch: https://github.com/YorickPeterse/ruby-lint/commits/refactor-definitions-builder

Keep in mind that with ruby-lint's case I already had an AST quite similar to the one produced by parser. Although I still have to change a lot this at least makes it easier.

[yorickpeterse]

A sidenote: now that we're actually starting to document parts of parser it might be a good idea to get a standard YARD structure going instead of just lumping everything into a single directory. ruby-lint has (in my opinion) a decent setup (including a stylesheet that fixes some issues that I have with YARD) but I'm open to everything.

[whitequark]

I've never intended to dump all docs into a single directory; the documentation for the code itself should go alongside the code, and documentation for the development process and practices is what's in the doc/. (LLVM uses a similar structure, and it also has a separate guide category.)

While I don't have much time on my hands for documenting parser, I'll review all PRs ASAP.

[bbatsov]

@bf4 Btw, next the JRuby major release will feature Ripper, which should alleviate the problem a bit. When I started RuboCop I didn't exactly enjoy the the thought of locking things down to MRI, but Ripper was the only parser/lexer that was really viable. That said I'd really like to us to move to Parser at some point down the road. @whitequark Maybe should mention somewhere in README how far along the way Parser is to be considered 'production-ready', since many people (me included) are wary of using young projects that have not yet been extensively deployed somewhere, especially when they are not intimately familiar with them.

[whitequark]

@bbatsov I did mention in README the 100% grammar coverage, which pretty much sums it up. It parses more syntax than ruby_parser and should parse all existing syntax, what else do you need?

[bbatsov]

Well, the exact wording is "100% test coverage for Bison grammars (except error recovery)." I cannot be certain how accurate the grammars actually are. :-) The README says "The Bison parser rules are derived from Ruby MRI parse.y", which can be interpreted in a number of ways. I'm not criticising you, I'm just saying that the current wording leaves some room for interpretation.

Anyways, can I interpret your answer as "Parser at this point can be used as a replacement for Ripper."?

[mbj]

@bbatsov As a user some ruby parsers in the wild: ripper, melbourne (rbx), ruby_parser and finally parser. My personal opinion is: parser is the best by design. Sure it is a young project. But it delivers more value than the others. Especially for my focus: Writing tools.

The pros of parser are listed briefly in the README. And another opinion: It can be used as a replacement for ripper.

[yorickpeterse]

@mbj You can't use it as a drop-in replacement of Ripper though (this is a feature I suppose since Ripper is retarded in many ways).

[mbj]

@yorickpeterse I dint say something about drop in replacement ;)

[bbatsov]

@mbj @yorickpeterse I hate Ripper so much, that I'd never wish for a drop-in replacement. Apart from Ripper's awful sexp format, another major issue there is the lack of source code locations for some elements (like keywords) in the sexps. Often in RuboCop we have to combine the results from Ripper.sexp and Ripper.lex to implement some checks.

I've been playing a bit with Parser today and I've been impressed with both the sane and consisten sexp format and the source_map functionality.

[mbj]

@bbatsov I evaluated lots of parsers/asts before building my mutant tool. If parser would have been there at the time I'd have used it. It is IMHO the best ruby parsing / manipulation solution.

I watched development closely and I'd be really surprised you find major issues ;)

[whitequark]

@bbatsov Yes, parser can be used as a replacement for Ripper. I'll add that to README when I get back to my notebook.

[bbatsov]

@whitequark I pretty much know that by now :-) I've started porting RuboCop to Parser last evening and I'm amazed how much simpler the Parser based code is compared to the Ripper version. A truly amazing project, thanks for creating it!

[bf4]

This would be great to convert, as well https://github.com/flyerhzm/code_analyzer

[bbatsov]

@bf4 Little value in porting this lib. It doesn't offer much more than you'd get from Parser's built-in AST Processor.

At any rate - porting such small libs is trivial.

[whitequark]

I assume no one's going to write this. If not, comment and I'll reopen.