RFC: Procedural macros 1.1

[alexcrichton]

Extract a very small sliver of today's procedural macro system in the compiler,
just enough to get basic features like custom derive working, to have an
eventually stable API. Ensure that these features will not pose a maintenance
burden on the compiler but also don't try to provide enough features for the
"perfect macro system" at the same time. Overall, this should be considered an
incremental step towards an official "plugins 2.0".

Rendered

[aturon]

Suggestion: this might want to be titled Plugins 1.1, as it doesn't really impact the "macro" system (if you think of that as macro_rules, which I think many do).

[aturon]

cc @jimmycuadra

[eddyb]

@aturon I believe this is because @nrc wanted to avoid "plugins" for "procedural macros".

[alexcrichton]

@aturon yeah it's true that this only really affects what we call plugins today, but I believe that @nrc wanted to start avoiding the term "plugin" as everything will eventually be a macro in the "macros 2.0" world.

[eddyb]

This name is unfortunate, couldn't we use libmacro?

[alexcrichton]

This is currently done because macro is a keyword, and this doesn't propose changing that just yet. I was thinking that we could perhaps wholesale rename the crate one day as well if macro becomes an available name.

[MovingtoMars]

How about libmacros? This follows the approach that Go takes, ie the strings package.

[sfackler]

This feels a bit of an abuse of "link" if we switch plugins to IPC executables - maybe a separate flag would be a good idea?

[alexcrichton]

I believe the intention is that --extern works to load macros (it's what Cargo will pass), so this was just a natural extension of that. We could perhaps add something like -L macro=..., but at this point it doesn't seem strictly necessary as the compiler can always distinguish whether an artifact is a macro or an rlib.

[eddyb]

I believe -L for rustc means "library search path" not "link search path".

[alexcrichton]

@nrc could you comment on the Context argument here? I believe both @eddyb and @cgswords think it may not be necessary, in which case the pointer could be dropped everywhere and these methods could just be fmt::Display and FromStr (standard traits)

[nrc]

I agree it may not be necessary. To be clear you must have some handle to the compiler, let's call that MacroContext, then either you can pass that around in the various functions, or you can leave it in TLS. The former is less ergonomic, but the later might have problems with multi-threaded macros and is arguably not as good engineering practice. I'm undecided as to which way we should swing in terms of the cost/benefit, but it would make a big difference to any libmacro APIs (methods vs free functions + the argument itself).

[eddyb]

Keep in mind that if you do interning with integer indices you can't even implement fmt::Debug on Token without putting something in TLS, and there is no meaningful difference between passing &mut MacroContext around and having it in TLS except for needing to set up TLS in each thread, as opposed to passing the context to one child thread, if needed.

If any TLS is used (as is now needed to impl Debug for Token), then the threading problems are unavoidable and having the context usable cross-thread would let users write broken code.

[cgswords]

In the discussion about parallel expansion, there needs to be a principled way to combine contexts / TLS: if a multi-threaded macro manages to manipualte its MacroContext in each of the threads (eg by interning things), combining these contexts in a principled way (eg unifying the interners and re-numbering some of the macro output) is basically the only way to place nice, anyway. To this end, it's TLS (or, more accurately, atomically-accessed globals) will end up the more ergonomic option.

[eddyb]

Cargo's usage of --extern would also work.

[aturon]

Thanks @alexcrichton for the awesome writeup! We've been working for a while on some way enable usage of Serde and similar libraries on stable in the near term, without massive stabilization. For example, there's a code generation proposal (which sits entirely on the side from the macro/plugin system), as well as attempts to find a very small increment toward "macros 2.0". But this is the first proposal that can be implemented immediately, has a clear path toward speedy stabilization, and imposes virtually no maintenance burden. I'm incredibly excited to be so close to a solution for the biggest cause of nightly usage today.

One thing that wasn't totally obvious to me, given that the macros 2.0 plan is a bit out of cache: what, if any, parts of what you propose to stabilize here would end up being deprecated once macros 2.0 has fully landed?

(To be clear, I have zero problem stabilizing the small slice of functionality you've outlined here in the near term even if in the long term we plan to deprecate it.)

[eddyb]

I would've expected at least #[macro_use] to be necessary.

[alexcrichton]

cc @nrc, I think that importing custom derive annotations was a somewhat unspecified extension of "macros 2.0" right now, so to me it seems somewhat up in the air about whether we'd require that or not.

I don't mind one way or another, but I'm curious what @nrc thinks as well.

Eventually of course custom-derive will be imported through the normal module system I believe, so this is purely just a temporary stopgap.

[eddyb]

The problem is that if you do anything by default now, you'll need to opt out to be able to use the normal module system.

[alexcrichton]

Aha, an excellent point! You've convinced me, I'll add it.

[eddyb]

Couldn't we add parenthesis around all such fragments at stringification time to avoid the misinterpretation?
Plugin authors can't do anything about this if all they have is conversion to String.

[alexcrichton]

Sure! This is just an implementation detail for the compiler though, right?

[eddyb]

It's visible to the plugins (and stable code can observe it with stringify! right now), so adding extra parens should be specified somewhere.

[alexcrichton]

@aturon

what, if any, parts of what you propose to stabilize here would end up being deprecated once macros 2.0 has fully landed?

This is a good question! Of the pieces here, what I would imagine is:

• If we decide to rename anything, we'd accumulate deprecations for old names.
• The behavior of "go to a string, parse, then go back to a token stream" will likely be deprecated in favor of directly manipulating the token stream.
• The definition form of custom #[derive] will likely change and the proposed form here would be deprecated. We likely eventually want one that's imported via use (somehow).

Other than that though I imagine these pieces to survive stabilization:

• The concept of a "macro" crate type
• The concept of a "macro crate" provided in the standard distribution, providing all details necessary to talk to the compiler.
• The ability to inform Cargo of crates which are macro crates
• Cargo's behavior of compiling macro crates for the host and passing them to the compiler for the next unit of compilation.
• Transformers like #[derive] operating over token streams and allowing customization via attributes that are stripped during the expansion process

[ubsan]

First, I don't think this RFC has gone through enough design, and fast-tracking it just to get people off of nightly is not a good idea, IMO. However, I am in favor of eventually (soon-ish) getting this sliver.

Some of my specific issues:

First: librustc_macro is a terrible stable name! My vote's in for libmacro. It's not a rustc-specific thing, it's an all-of-Rust thing!

Second: Context seems like an unnecessary implementation detail, and, if they're used the same way as in the original proposal "Procedural macros should report errors, warnings, etc. via the MacroContext [Context]."... why. We already have an existing error reporting mechanism, with Results. This new way seems very un-Rustic.

Third: I don't like how macros are defined. I'd rather they be defined with something like macro instead, especially as we eventually are going to allow proc macros and normal stuff in the same library.

[eddyb]

@ubsan I don't like Context either, but Result won't cut it, macros 2.0 will eventually want a full-blown diagnostics API, where you can emit multiple warnings and non-fatal errors.
If the necessary information is placed in TLS, the API would work without a Context type.

[sfackler]

@ubsan Result is an all-or-nothing thing. In the compilation process, it is very common to be creating many diagnostics during one compilation pass. A compilation error is not necessarily fatal - you (within reason) want to continue as long as possible to avoid forcing the user to go through a recompilation cycle for every error when many could all be reported at once. In addition, warnings, notes, etc are certainly not fatal.

[arielb1]

I think librust_macro could be a fine name.

On the integration level: plugins imply some form of host ABI. Stable plugins mean a stable HOSTRUSTC. I think we can just leave HOSTRUSTC as an implementation-defined thing, but we must be careful not to change its definition on existing platforms.

[ubsan]

@sfackler makes sense. I still don't think Context is the best, though.

[sfackler]

Do you have any suggestions for a better replacement?

[ubsan]

@sfackler No, but I think @eddyb does :P

[arielb1]

On naming: I see librust_macro as an ordinary plugin library, and rustc_macro_derive as an exposed proc macro that (unstable) compiler plugins should eventually be able to reproduce (BTW, what is the nomenclature wrt. proc macro vs. plugin vs. compiler extension?)

rustc_macro_derive is a bad name, but we still don't have modular macros. I think pseudo-modular #[WHATEVER_CRATE_NAME_WE_PICK_derive] would work.

As for the API: maybe have #[CRATE_derive_v0] for the Context-free version?

[sfackler]

How does shoving Context into TLS hide implementation details?

[eddyb]

@sfackler The TLS slot wouldn't be exposed, the APIs would just not take any context.

[sfackler]

@ubsan's concern with Context seemed to be that it was an implementation detail, and I'm confused as to how moving all of the methods on Context to free functions hides any details of the implementation. It seems totally isomorphic in that regard.

[arielb1]

Yeah. A Context parameter is easy to ignore and hard to add later.

[nikomatsakis]

Huzzah! The @rust-lang/lang team has decided to accept this RFC. We decided to ask @alexcrichton to move some of the detailed matters (e.g., prefixes, etc) into unresolved questions to be addressed during stabilization. Thanks everyone!

[nikomatsakis]

Tracking issue: rust-lang/rust#35900

If you'd like to keep following the development of this feature, please subscribe to that issue, thanks! :)

[llogiq]

I still miss my comment about the inferior traceability of code through macros 1.1 as opposed to the current systems. Could we at least add it to the drawbacks section?

[nikomatsakis]

@llogiq I pushed some text about spans

[llogiq]

Ok then. I'll keep an eye on the implementation. 😉