Make the rustc driver and interface demand driven

[Zoxc]

This introduces a new crate rustc_interface which is the canonical interface for creating and using the compiler. It allows you to access a Compiler type in a closure and that types have methods to run passes on demand. The interesting parts are found here (defining the queries) and here (methods to create a Compiler).

cc @rust-lang/compiler @rust-lang/dev-tools @rust-lang/rustdoc

[rust-highfive]

r? @estebank

(rust_highfive has picked a reviewer for you, use r? to override)

[oli-obk]

Since this will impact rls, clippy and miri quite strongly, could you open PRs against them showing how those changes will impact them?

I would assume rls and miri to be easy (since they don't change compilation and just need access to the TyCtxt to start their business), but clippy inserts new lints, which I'm not sure how to do with the new interface (looks to me like rustdoc has the same problem, but the FIXME shows that you are aware of this already).

[rust-highfive]

📣 Toolstate changed by #56732!

Tested on commit 913ad6d.
Direct link to PR: #56732

💔 clippy-driver on windows: test-pass → build-fail (cc @Manishearth @llogiq @mcarton @oli-obk @phansch, @rust-lang/infra).
💔 clippy-driver on linux: test-pass → build-fail (cc @Manishearth @llogiq @mcarton @oli-obk @phansch, @rust-lang/infra).
💔 miri on windows: test-pass → build-fail (cc @oli-obk @RalfJung @eddyb, @rust-lang/infra).
💔 miri on linux: test-pass → build-fail (cc @oli-obk @RalfJung @eddyb, @rust-lang/infra).
💔 rls on windows: test-pass → build-fail (cc @nrc @Xanewok, @rust-lang/infra).
💔 rls on linux: test-pass → build-fail (cc @nrc @Xanewok, @rust-lang/infra).

[Manishearth]

I'm not entirely clear on how this changes the driver -- mind patching clippy's driver to work again? Use rustup-toolchain-install-master -n master -f to get a master toolchain so that you can build clippy.

I think what we need to do is invoke parsing, attach the lints, and then continue with compilation. But I'm not sure.

Overall it seems like this makes the driver API more brittle since driver consumers need to know what steps to invoke (as opposed to running the driver with some callbacks, where any change in the compilation process can still be picked up by clippy). There aren't many consumers of the driver interface, but this was supposed to be the path forward for things like clippy. Perhaps an interface which is "does the rustc compilation steps, but with callbacks" can still be exposed?

[Zoxc]

@Manishearth See rust-lang/rust-clippy#3733

[Manishearth]

Oh, I didn't notice that, thanks so much!

While trying to implement this I didn't realize Callbacks existed now, I retract my objection about the driver api being more brittle 😄