Decouple proc_macro from the rest of the compiler.

[eddyb]

This PR removes all dependencies of proc_macro on compiler crates and allows multiple copies of proc_macro, built even by different compilers (but from the same source), to interoperate.

Practically, it allows:

• running proc macro tests at stage1 (I moved most from -fulldeps to the regular suites)
• using proc macros in the compiler itself (may require some rustbuild trickery)

On the server (i.e. compiler front-end) side:

• server::* traits are implemented to provide the concrete types and methods
  • the concrete types are completely separated from the proc_macro public API
  • the only use of the type implementing Server is to be passed to Client::run

On the client (i.e. proc macro) side (potentially using a different proc_macro instance!):

• client::Client wraps around client-side (expansion) function pointers
  • it encapsulates the proc_macro instance used by the client
  • its run method can be called by a server, to execute the client-side function
    • the client instance is bridged to the provided server, while it runs
    • currently a thread is spawned, could use process isolation in the future
      (not the case anymore, see #56058)
• proc macro crates get a generated static holding a &[ProcMacro]
  • this describes all derives/attr/bang proc macros, replacing the "registrar" function
  • each variant of ProcMacro contains an appropriately typed Client<fn(...) -> ...>

proc_macro public APIs call into the server via an internal "bridge":

• only a currently running proc macro Client can interact with those APIs
  • server code might not be able to (if it uses a different proc_macro instance)
    • however, it can always create and run its own Client, but that may be inefficient
• the bridge uses serialization, C ABI and integer handles to avoid Rust ABI instability
• each invocation of a proc macro results in disjoint integers in its proc_macro handles
  • this prevents using values of those types across invocations (if they even can be kept)

r? @alexcrichton cc @jseyfried @nikomatsakis @Zoxc @thepowersgang

[eddyb]

@bors try

[bors]

🔒 Merge conflict

[eddyb]

@bors try

[bors]

⌛️ Trying commit 6e63e8d with merge c4a77e1...

[eddyb]

cc @rust-lang/infra We shouldn't merge this without a crater run, although you might want to wait on a review as well (then again, the results could inform the review, so whichever comes first).

[bors]

☀️ Test successful - status-travis
State: approved= try=True

[Zoxc]

Do we want to commit to using interning for literals? I'm not sure interning it terrible useful for that.

[eddyb]

Wait, I can make these fields private again. Definitely not considering stabilizing any of this.

[Zoxc]

This should probably be a rustbuild change.

[bjorn3]

Could you use something like bincode for data bridging, instead of 800 lines of own code?

[eddyb]

@bjorn3 I'm not serializing the data (but rather using handles), and bincode (like any other serialization) is annoying to use without derived impls (which you can't have because this is proc_macro itself - or at least I don't want to try to make that work, for now).

I am going to look into simplifying the bridge code, removing most unsafe, and making it more amenable to message-passing, however.

[bjorn3]

I get it.

[alexcrichton]

Some awesome progress! The "dream" of deleting libserialize is coming ever closer...

I'm personally having a lot of trouble following what's going on here though. Would it be possible to add a summary comment to the bridge.rs module explaining what's going on in some more depth? The PR description currently references a lot of new types/traits that I'm not sure what's going on after reading the first pass.

Perhaps you could detail an example of what happens when a small/example procedural macro is invoked?

I'm also a little worried about all the FIXME or todo annotations about ABI problems. For example it looks like this will require the ABI of PathBuf to never change, right? (the risk being that if we change the ABI we can't bootstrap the compiler?) There were other things like trait objects which I was also fairly suspicious about. (as well as fmt::Formatter)

One perhaps larger-than-intended change here as well is dealing with the standard library itself. Today because proc_macro links to libsyntax it dynamically links to the standard library, which means that crates linking to the proc_macro crate automatically link dynamically (for better or worse) to the standard library as well. Running around with two copies of the standard library (one in the plugin that we load and one in the compiler itself) can wreak havoc with panic hooks and the like. (or any other form of global state). Do we know how this'll get mitigated?

[eddyb]

the risk being that if we change the ABI we can't bootstrap the compiler

I tried to have a reassuring comment in the PR description - we can always bootstrap the compiler, it'd just require fixing the ABI incompatibility within the in-tree libproc_macro.
That is, we don't have to wait 6 weeks to get such a fix in.

I will try to address most if not all such issues before merging the PR - see the TODO bit at the end of the description, I've been convinced, since opening the PR, that I should redo some parts of it.

Perhaps you could detail an example of what happens when a small/example procedural macro is invoked?

Heh, I tried this and ended up with the weird lists in the PR description, I'll try again once I've mad the code changes I plan to (for now I've also added [WIP] to the PR title).

Running around with two copies of the standard library (one in the plugin that we load and one in the compiler itself) can wreak havoc with panic hooks and the like. (or any other form of global state). Do we know how this'll get mitigated?

Did you see the commit where I moved the panic silencing hook from the compiler to proc_macro?
Even if we're still single-process for now, we could say that the proc macro ("the client") might run in a different process (thus use a different libstd) than the compiler frontend ("the server").
So cross-process solutions (such as making proc_macro state self-contained and serializing to plain bytes, or using the C ABI as-if it were serializing) should also deal with multiple libstd in-process.

[Mark-Simulacrum]

Not sure if you wanted perf, or whether this is even helpful, but http://perf.rust-lang.org/compare.html?start=75af15ee6ca0c12b699a17984b033363cd25e9c3&end=c4a77e121949c30306dbbf48259a49c3aeb55028&stat=instructions%3Au

[eddyb]

@Mark-Simulacrum It doesn't look bad, that's for sure, thanks! If we switch to spawning a thread (or even a process) per invocation it'd warrant another perf run.

[aidanhs]

Crater run started

[aidanhs]

Hi @eddyb (crater requester), @Zoxc (PR reviewer)! Crater results are at: http://cargobomb-reports.s3.amazonaws.com/pr-49219/index.html. 'Blacklisted' crates (spurious failures etc) can be found here. If you see any spurious failures not on the list, please make a PR against that file.

(interested observers: Crater is a tool for testing the impact of changes on the crates.io ecosystem. You can find out more at the repo if you're curious)

[eddyb]

@bors r=alexcrichton

[bors]

📌 Commit 3a04d44 has been approved by alexcrichton

[bors]

⌛️ Testing commit 3a04d44 with merge d48ab69...

[bors]

☀️ Test successful - status-appveyor, status-travis
Approved by: alexcrichton
Pushing d48ab69 to master...

[petrochenkov]

I can't believe you've done this!