Next steps in macro support

[matklad]

The basic infra for macros is in place: hir uses HirFileId which can refer to both usual files and macro expansions.

Now we need to learn how to actually expand macros. We need to handle two kinds of macros: macro by example (macro_rules) and procedural macros. Crates like https://github.com/nrc/proc-macro-rules (thanks @nrc!) should allow us to handle only the second case. Basically, the pipeline should look like

ra::SyntaxNode 
-> proc_macro2::TokenStream 
-> existing macro ecosystem 
-> proc_macro2::TokenStream
-> ra::SyntaxNode

We need to implement only first and last arrow (which might be not easy as it seems because we need hygiene info somewhere).

The next big step after understanding macro_rules would be plugging them into the resolver in a proper recursive way.

[matklad]

Hm, I am having second thoughts regarding the usage of proc_macro2 here:

• I'd love to use TokenStream as keys/values in database, and that requires Hash & PartialEq
• I'd love to construct TokenStream from SyntaxNodes and account for hygene, and that seems complicated with proc_macro2
• proc_macro2 uses some thead_local state for CODE_MAP, and that seems unfortunate.

Hopefully designing our own token tree type wouldn't be too difficult!

We also could try a better API

[kazimuth]

question: is it possible to expand proc_macros from a crate like this on stable rust? Afaict the only caller allowed to invoke proc_macros is rustc.

[edwin0cheng]

@kazimuth , It is possible to do that with dynamic loading: https://github.com/fedochet/rust-proc-macro-expander

That is using nightly rustc api, But by dispatching the macro parsing to separate process, it is allowed RA on build in stable. On the other hand, if rust ABI is not concerned, it is possible to change that to use only stable rust api too.

[pksunkara]

I have a custom derive which takes in structs and builds more structs and their implementations. The generated structs and methods don't seem to be available for resolving types and autocompletion and highlighting. I think this issue solves it, doesn't it? Or am I wrong?

[pksunkara]

Also, from a discussion in zulip

when declaring macro_rules (not talking about proc_macros), you are generally given token hints like ident, expr etc.. Since we read the macro_rules def, can we use those hints to resolve the tokens inside macro call?

More generally, we should make compleiton and goto definition work for $meta_vars, then highlighting

Should I create a different issue for this or does this issue cover it?

[bjorn3]

Did you enable proc-macro support with the following config?

{
    "rust-analyzer.cargo.loadOutDirsFromCheck": true,
    "rust-analyzer.procMacro.enable": true,
}

[pksunkara]

Yeah, I even restarted a lot of things too. The types are definitely not being resolved.

[matklad]

Our macro engine has been pretty capable for a long time now, so closing in favor of more specific issues.

[zeenix]

@matklad Hi. I've had the settings mentioned above, enabled in my config for a very long time but completion for proc macro generated code doesn't work for zbus at least. Is there any other configurations I need to enable or does zbus macros need to do something?

[flodiebold]

Attribute proc macros aren't supported yet, just derives and function-like ones. See e.g. #6029.