rustc: Stabilize the `proc_macro` feature

[alexcrichton]

This commit stabilizes some of the proc_macro language feature as well as a
number of APIs in the proc_macro crate as previously discussed. This
means that on stable Rust you can now define custom procedural macros which
operate as attributes attached to items or macro_rules!-like bang-style
invocations. This extends the suite of currently stable procedural macros,
custom derives, with custom attributes and custom bang macros.

Note though that despite the stabilization in this commit procedural macros are
still not usable on stable Rust. To stabilize that we'll need to stabilize at
least part of the use_extern_macros feature. Currently you can define a
procedural macro attribute but you can't import it to call it!

A summary of the changes made in this PR (as well as the various consequences)
is:

• The proc_macro language and library features are now stable.
• Other APIs not stabilized in the proc_macro crate are now named under a
  different feature, such as proc_macro_diagnostic or proc_macro_span.
• A few checks in resolution for proc_macro being enabled have switched over
  to use_extern_macros being enabled. This means that code using
  #![feature(proc_macro)] today will likely need to move to
  #![feature(use_extern_macros)].

It's intended that this PR, once landed, will be followed up with an attempt to
stabilize a small slice of use_extern_macros just for procedural macros to
make this feature 100% usable on stable.

[rust-highfive]

r? @aturon

(rust_highfive has picked a reviewer for you, use r? to override)

[rust-highfive]

Warning

• These commits modify submodules.

[alexcrichton]

r? @nikomatsakis

cc @petrochenkov, @dtolnay, @nrc

[petrochenkov]

What's the point in renaming unstable library features?
This is only breaking all the code using them.

[alexcrichton]

The proc_macro feature at this point has been used-and-abused I think a bit too much. This PR is, afaik, the main thrust of proc_macro for stabilization. The renamings here aren't intended to purely break everyone else's code, but rather clearly singify different groupings of features instead of leaving them all in a perpetual singular proc_macro bucket

[petrochenkov]

Are you going to use proc_macro_lib3/4/5/... for the next portions of stabilized APIs? :)
I used proc_macro_12 in my PR (#50473 (comment)), because it's Macros 1.2.
(OK, it doesn't matter anyway.)

[alexcrichton]

As I'm sure you know, the stable feature name doesn't actually matter. I initially changed these while the other unstable features were still proc_macro and then ended up later changing the other proc_macro features to different names. As a result this stuck. It's an arbitrary name.

[petrochenkov]

The custom_attribute feature no longer works as unknown attibutes are looked up as macros.

Oh man, this is going to be painful.

[petrochenkov]

I don't see much use in landing this PR before stabilizing use_extern_macros and completing #51952, but it's good to have a PR open for some publicity.

I plan to fix tool attributes in the next few weeks, so poor users of #[rustfmt_skip] could migrate to #[rustfmt::skip] if custom attributes are not fixed either.

I think custom attributes could be restored in some form as well, e.g. if some attribute is "definitely unresolved" as a macro, then it can be kept in the source as a custom attribute (still under a feature gate) instead of producing a dummy expansion, like it's done now.

[alexcrichton]

While true that this PR isn't too useful landing by itself it does need to land eventually for proc-macro stabilization, and this was larger and more thorny than I thought it would be so I wanted to put it up for review and get it in soon to iron out all the CI issues.

[SimonSapin]

The custom_attribute feature no longer works as unknown attibutes are looked up as macros.

Oh man, this is going to be painful.

Does this also break rustc_plugin::Registry::register_attribute in #[plugin_registrar]? Servo still has a lint plugin with (unfortunately) no clear upgrade path.

[eddyb]

FWIW, I have mentioned before that if the unrooted_must_root_lint analysis is "close enough" to being sound, it should naturally be replaceable with a trait-based system that's partially implemented inside the compiler itself, with most of the logic being in impls/derives in Servo crates.

[SimonSapin]

@eddyb Yes, it might be "just" a matter of someone familiar enough with what the lint does and the soundness of GC rooting to come up with a new design and implementing it. In the meantime, a lint is what we have.

[alexcrichton]

@SimonSapin yes it'd break that because those attributes aren't hooked into the normal resolver. You can rename them with a rustc_ prefix, though, and it should continue working. All attributes with a rustc_ prefix are gated and skip the resolver.

[alexcrichton]

I think @nikomatsakis is out this week, so...

r? @nrc

[alercah]

I have a major reservation about stabilizing that I brought up in the tracking issue.

[petrochenkov]

@alexcrichton
The attribute resolution changes (tool attributes, custom attributes) are really a part of the macro modularization feature (#35896), gating them on proc_macro was a mistake.

I think we can land the proc macro library API part right now (it adds nothing fundamentally new compared to TokenStream::parse) and the part of language proc_macro feature that allows to define #[proc_macro] and #[proc_macro_attribute] items, but not use them.
Everything else should be effectively moved under the use_extern_macros gate and has some blocker issues to fix before stabilization.

Could you leave this stabilization work to me?
I have more time now than in the previous ~year and have a plan with regards to what needs to be done and in what order.

[bors]

⌛️ Testing commit 65f3007 with merge 2750fc6...

[kennytm]

@bors retry

GitHub outage causing the CIs not seeing the auto branch got updated.

[bors]

⌛️ Testing commit 65f3007 with merge 1ecf692...

[bors]

☀️ Test successful - status-appveyor, status-travis
Approved by: petrochenkov
Pushing 1ecf692 to master...

[rust-highfive]

📣 Toolstate changed by #52081!

Tested on commit 1ecf692.
Direct link to PR: #52081

💔 clippy-driver on windows: test-pass → test-fail (cc @Manishearth @llogiq @mcarton @oli-obk, @rust-lang/infra).
💔 clippy-driver on linux: test-pass → test-fail (cc @Manishearth @llogiq @mcarton @oli-obk, @rust-lang/infra).