Using ast-grep as a library

[charliermarsh]

Awesome project! I'm interested in potentially using ast-grep as the foundation for Ruff's plugin system (some discussion in astral-sh/ruff#283).

What I'm envisioning: users express custom lint rules via the ast-grep YAML syntax; Ruff picks up those rules, parses + executes them via ast-grep, and includes the surfaced messages + fixes in its own output abstractions.

Before I dig in, I wanted to see if (1) this kind of use-case is within-scope for ast-grep, (2) if it's possible now or depends on future work, and (3) if you had any pointers on how to get started (I know the API docs are a work-in-progress so any tips would be appreciated).

[HerringtonDarkholme]

Hi @charliermarsh ! Thanks for your kind interest in ast-grep!

I also know Ruff by @morgante on twitter. Thanks for building such a great library for Python.

Back to Ruff and ast-grep. I believe their underlying parser are different. ast-grep relies heavily on tree-sitter as parser for now. But Ruff uses rustypython as the parsing library. Integrating ast-grep will introduce two sets of different parsing that will probably introduce confusion to users. (e.g. different parsing result and linting results for the same code running the same Ruff.)

Also, let me answer your question more directly.

1. Using ast-grep as library is definitely one of my goal! But it requires the lib user embracing tree-sitter.
2. It is possible to use ast-grep for now! It is composed of two crates: ast-grep-core for matching and ast-grep-config for YAML rule parsing. (but I expect there will be API change)
3. An example can be found here https://github.com/ast-grep/ast-grep/blob/main/crates/cli/src/scan.rs

[charliermarsh]

Oh, thank you! Flattered that you're familiar with Ruff.

FWIW I've considered migrating Ruff to tree-sitter (astral-sh/ruff#295, astral-sh/ruff#286). I think there would be a variety of benefits, but of course there would be a bunch of tradeoffs so I haven't committed to anything yet (just exploring the options).

But even if Ruff itself uses the RustPython parser internally, I could still be interested in letting users write Ruff plugins via the ast-grep syntax and powering those plugins via the ast-grep engine (which would of course use tree-sitter, not RustPython, under-the-hood). I want to add some kind of plugin system to Ruff that doesn't require users to write Rust code, and something based on tree-sitter and/or ast-grep seems like a good foundation (even if the Rust-based checks are using RustPython -- in other words, I'm open to using different parsers for different subtasks).

This is plenty of info for me to get started on exploring that route. Thank you for writing this up! Feel free to close out the issue if you'd like :)

[charliermarsh]

I'l track on the Ruff side here: astral-sh/ruff#516. I have some other stuff to get to first, so won't happen immediately, but I am definitely interested in trying this out.

[HerringtonDarkholme]

Hey @charliermarsh! ast-grep now supports Python API. https://ast-grep.github.io/guide/api-usage/py-api.html

That said, it does not support AST other than tree-sitter. I think it is possible to abstract AST formats but let's track it in another issue!

Thanks!