GritQL integration progress (April 3)

[arendjr]

It's been almost two months since I originally proposed integrating GritQL into Biome. Since then a lot has happened. And a lot more remains to be done. With this post I hope to give a little bit of an update of where we are at, and also indicate what the next steps will be. Brace yourselves, we're getting technical :)

Recap

First a little overview of what happened over the past two months:

• Less than two weeks after my original RFC, Grit open-sourced the grammar for GritQL.
• This allowed me to implement Biome's GritQL parser, which has since been merged into our main branch.
• A few weeks ago, Grit open-sourced their GritQL engine, which I have also been contributing to since.

Current State

So where are we right now? With some of the pieces starting to fall into place, there's still a major technical challenge to overcome...

The GritQL engine -- codenamed Marzano, you'll see that name pop up here and there -- makes extensive use of TreeSitter for parsing languages. As a result, TreeSitter data types are extensively used within Marzano. But Biome has its own parsers and would ideally avoid pulling in alternative parsers for the languages it already supports.

How to resolve this? Well, I've been contributing to the GritQL open-source repository trying to tackle this exact issue. Within the repository there's the marzano-core crate which contains the core data structures and logic for Grit's pattern matching and rewriting engine. Those are still currently tied to TreeSitter's data structures, but I've been working piece-wise to loosen those dependencies and introduce traits as abstractions. It's hard to estimate how long exactly this will take, but the end goal is for the TreeSitter dependencies to become fully optional. At that point I intend to extract another crate, which I'll probably call grit-core-patterns. That crate would contain the data structures for Grit's patterns and as much of the logic as I can while keeping the crate free of TreeSitter dependencies.

Looking Forward

Once we have the grit-core-patterns crate, we can incorporate it into Biome, at which point there will be at least a few more steps to integrate it:

• We need to construct Grit "programs" from the CST we can parse. The parser is already there, but the conversion into programs is yet to be written.
• Grit patterns can contains code snippets which also need parsing. Grit uses the marzano-language crate for this, but it's again TreeSitter-specific. We'll need to create our own implementation that uses Biome's parsers.
• We'll need to reimplement all the abstract traits from grit-core-patterns using Biome-specific logic.

Once those steps are complete, we should be pretty much able to execute Grit programs in order to search for code patterns, and rewriting patterns should be a relatively quick next step after that.

Once all that's in place, we'll also need some infrastructure to be able to configure Grit snippets as plugins. That's where we'll try to solve actual use cases for our linter, transformer, and maybe even formatter.

Next Steps

1. For now I'll keep hacking at marzano-core to try to make all its TreeSitter dependencies optional.
2. Simultaneously I'm working towards implementing a biome search command that would serve as a proof-of-concept for using GritQL within Biome that would be usable even before we can use them as plugins.
   • This command will be disabled for users for now, but during the experimentation phase I may try to get this command working with the current marzano-core including all its TreeSitter dependencies. It would be a bit of a "Parsenstein" solution, but it could be used to show we are on the right track.

What about JS plugins?

Besides GritQL plugins, the original RFC also mentioned JS/TS plugins. I did a little bit of investigation into Boa, which seems like a promising solution for getting JS/TS plugins into Biome. But for now, all my attention is on getting GritQL to work first. So if somebody wants to volunteer, we'd surely welcome the help! :)

Wrap-up

That's all for now. If you have any questions or suggestions, feel free to ask!

[Sec-ant]

@arendjr This is amazing and you have all my respect for what you're doing! I wonder if there will be some repetitive but necessary works in this area can be labeled as good first issues, so we can also get our feet wet to push this forward. 😄

[arendjr]

Thanks! I will definitely give this some thought.

Surely the biome search command could be designed further up to the point where all that’s missing is invoking the Grit engine. For instance, displaying the search results is a nice task on its own.

[arendjr]

Latest progress update can be found here: #2585