This repo contains cargo-generate templates intended to simplify developing procedural macros in the style described by the ferrous systems blog post on testing proc macros.
To quickly summarize the approach recommended there and which is adhered to by these templates, procedural macro expansion is broken down into four distinct stages:
- parse
- Parse the contents of the input [TokenStream](https://doc.rust-lang.org/proc_macro/struct.TokenStream.html) to produce an Abstract Syntax Tree (AST) in Rust that represents the structure of the macro. The AST is likely to consist of structs and enums whose contents are [syn](https://docs.rs/syn) token types (at least, that's the assumption made by the templates in this repo).
- analyze
- Transform the AST of the macro into a Rust model abstracting over the domain of the macro.
- lower
- Transform the model into an "intermediate representation" (IR) of the output code generated by the macro. This IR, similar to the AST, should probably consist mostly if not entirely of [syn](https://docs.rs/syn) tokens, which simplifies the process of producing a [TokenStream](https://doc.rust-lang.org/proc_macro/struct.TokenStream.html) in the final stage, `codegen`
- codegen
- Transform the IR into a [TokenStream](https://doc.rust-lang.org/proc_macro/struct.TokenStream.html) using the [quote!](https://docs.rs/quote/latest/quote/macro.quote.html) macro.
If you're wondering "why does it need to be broken down like this?", then you should probably read the abovementioned blog post about testing procedural macros. But the answer to your question is, "it doesn't".
So far in my time learning about procedural macros, I have come across a small ecosystem of crates all written by the same author:
- proc_macro2
- proc_macro_error
- syn
- quote
There are probably more, but I want to point them out here because I understand it can be mildly confusing and to get started writing procedural macros and I'd like to help whatever poor soul finds their self resting their weary eyes on these bedraggled words.
proc_macro2 is an
abstraction over the upstream Rust
proc_macro library that, for the
purpose of these templates, serves to enable unit testing of procedural macro
helper code. This is not possible with the upstream proc_macro types because
they can only ever be used directly inside a procedural macro.
It's worth noting that proc_macro2 and proc_macro both export a type called
TokenStream. proc_macro2::TokenStream, in my very limited understanding, is
necessary since the upstream proc_macro::TokenStream type can only be used in
the definition of actual procedural macro functions (ie not helper functions or
unit tests).
proc_macro_error
claims to make procedural macro error reporting simple and easy to use. I did
have some difficulty with it myself initially when I was working on my first
macro parser, but I've also never tried any alternative so I take the author at
their word. I can definitely attest that the error messages I've seen while
using it were mildly useful, though I did also find myself making liberal use
of eprintln! since the syntax I was parsing is very custom and DSL-like.
syn is arguably the powerhouse of this collection of procedural macro crates and the one you will undoubtedly spend the most time with if the syntax you are parsing or code you are generating involves any non-trivial amount of complexity. It contains various types representing valid Rust tokens and parsing functions and traits that simplify your life as a macro writer. It's definitely worth reading the code to understand:
- the
Parsertrait - the various
parsefunctions - and other stuff I'm probably forgetting right now
quote is a crate that exports a macro of
the same name that will expand the "example" rust syntax passed to it using
in-scope variables to produce a proc_macro2::TokenStream instance that can be
converted into proc_macro::TokenStream and returned as the output of the
procedural macro.