Auto merge of rust-lang#3411 - RalfJung:sysroot, r=RalfJung

Handle Miri sysroot entirely outside the Miri driver

(Extracted from rust-lang/miri#3409)

This entirely moves the responsibility of setting miri-sysroot to whatever *invokes* the Miri driver.  cargo-miri knows whether it is inside rustdoc or not and can adjust accordingly. I previously avoided doing that because there are a bunch of places that are invoking the driver (cargo-miri, the ui test suite, `./miri run`, `./x.py run miri`) and they all need to be adjusted now. But it is also somewhat less fragile as we usually have more information there -- and we can just decide that `./miri run file.rs --sysroot path` is not supported. The advantage of this is that the driver is reasonably clean and doesn't need magic environment variables like MIRI_SYSROOT, and we don't have to fight rustc_driver to use a different default sysroot. Everything is done in cargo-miri (and the other much simpler driver wrappers) where it can hopefully be debugged much better.

src/tools/miri/README.md

@@ -451,36 +451,32 @@ Some native rustc `-Z` flags are also very relevant for Miri:
 * `-Zmir-emit-retag` controls whether `Retag` statements are emitted. Miri
   enables this per default because it is needed for [Stacked Borrows] and [Tree Borrows].
 
-Moreover, Miri recognizes some environment variables:
+Moreover, Miri recognizes some environment variables (unless noted otherwise, these are supported
+by all intended entry points, i.e. `cargo miri` and `./miri {test,run}`):
 
 * `MIRI_AUTO_OPS` indicates whether the automatic execution of rustfmt, clippy and toolchain setup
   should be skipped. If it is set to `no`, they are skipped. This is used to allow automated IDE
   actions to avoid the auto ops.
 * `MIRI_LOG`, `MIRI_BACKTRACE` control logging and backtrace printing during
   Miri executions, also [see "Testing the Miri driver" in `CONTRIBUTING.md`][testing-miri].
-* `MIRIFLAGS` (recognized by `cargo miri` and the test suite) defines extra
-  flags to be passed to Miri.
-* `MIRI_LIB_SRC` defines the directory where Miri expects the sources of the
-  standard library that it will build and use for interpretation. This directory
-  must point to the `library` subdirectory of a `rust-lang/rust` repository
-  checkout.
-* `MIRI_SYSROOT` (recognized by `cargo miri` and the Miri driver) indicates the sysroot to use. When
-  using `cargo miri`, this skips the automatic setup -- only set this if you do not want to use the
-  automatically created sysroot. For directly invoking the Miri driver, this variable (or a
-  `--sysroot` flag) is mandatory. When invoking `cargo miri setup`, this indicates where the sysroot
-  will be put.
-* `MIRI_TEST_TARGET` (recognized by the test suite and the `./miri` script) indicates which target
+* `MIRIFLAGS` defines extra flags to be passed to Miri.
+* `MIRI_LIB_SRC` defines the directory where Miri expects the sources of the standard library that
+  it will build and use for interpretation. This directory must point to the `library` subdirectory
+  of a `rust-lang/rust` repository checkout.
+* `MIRI_SYSROOT` indicates the sysroot to use. When using `cargo miri`, this skips the automatic
+  setup -- only set this if you do not want to use the automatically created sysroot. When invoking
+  `cargo miri setup`, this indicates where the sysroot will be put.
+* `MIRI_TEST_TARGET` (recognized by `./miri {test,run}`) indicates which target
   architecture to test against.  `miri` and `cargo miri` accept the `--target` flag for the same
   purpose.
-* `MIRI_TEST_THREADS` (recognized by the test suite): set the number of threads to use for running tests.
-  By default the number of cores is used.
-* `MIRI_NO_STD` (recognized by `cargo miri`) makes sure that the target's sysroot is built without
-  libstd. This allows testing and running no_std programs.
-  (Miri has a heuristic to detect no-std targets based on the target name; this environment variable
-  is only needed when that heuristic fails.)
-* `RUSTC_BLESS` (recognized by the test suite and `cargo-miri-test/run-test.py`): overwrite all
+* `MIRI_TEST_THREADS` (recognized by `./miri test`): set the number of threads to use for running tests.
+  By default, the number of cores is used.
+* `MIRI_NO_STD` makes sure that the target's sysroot is built without libstd. This allows testing
+  and running no_std programs. (Miri has a heuristic to detect no-std targets based on the target
+  name; this environment variable is only needed when that heuristic fails.)
+* `RUSTC_BLESS` (recognized by `./miri test` and `cargo-miri-test/run-test.py`): overwrite all
   `stderr` and `stdout` files instead of checking whether the output matches.
-* `MIRI_SKIP_UI_CHECKS` (recognized by the test suite): don't check whether the
+* `MIRI_SKIP_UI_CHECKS` (recognized by `./miri test`): don't check whether the
   `stderr` or `stdout` files match the actual output.
 
 The following environment variables are *internal* and must not be used by

src/tools/miri/cargo-miri/src/phases.rs

@@ -172,8 +172,6 @@ pub fn phase_cargo_miri(mut args: impl Iterator<Item = String>) {
     // Forward all further arguments (not consumed by `ArgSplitFlagValue`) to cargo.
     cmd.args(args);
 
-    // Let it know where the Miri sysroot lives.
-    cmd.env("MIRI_SYSROOT", miri_sysroot);
     // Set `RUSTC_WRAPPER` to ourselves.  Cargo will prepend that binary to its usual invocation,
     // i.e., the first argument is `rustc` -- which is what we use in `main` to distinguish
     // the two codepaths. (That extra argument is why we prefer this over setting `RUSTC`.)
@@ -200,10 +198,7 @@ pub fn phase_cargo_miri(mut args: impl Iterator<Item = String>) {
     // always applied. However, buggy build scripts (https://github.com/eyre-rs/eyre/issues/84) and
     // also cargo (https://github.com/rust-lang/cargo/issues/10885) will invoke `rustc` even when
     // `RUSTC_WRAPPER` is set, bypassing the wrapper. To make sure everything is coherent, we want
-    // that to be the Miri driver, but acting as rustc, on the target level. (Target, rather than
-    // host, is needed for cross-interpretation situations.) This is not a perfect emulation of real
-    // rustc (it might be unable to produce binaries since the sysroot is check-only), but it's as
-    // close as we can get, and it's good enough for autocfg.
+    // that to be the Miri driver, but acting as rustc, in host mode.
     //
     // In `main`, we need the value of `RUSTC` to distinguish RUSTC_WRAPPER invocations from rustdoc
     // or TARGET_RUNNER invocations, so we canonicalize it here to make it exceedingly unlikely that
@@ -212,14 +207,19 @@ pub fn phase_cargo_miri(mut args: impl Iterator<Item = String>) {
     // bootstrap `rustc` thing in our way! Instead, we have MIRI_HOST_SYSROOT to use for host
     // builds.
     cmd.env("RUSTC", fs::canonicalize(find_miri()).unwrap());
-    cmd.env("MIRI_BE_RUSTC", "target"); // we better remember to *unset* this in the other phases!
+    // In case we get invoked as RUSTC without the wrapper, let's be a host rustc. This makes no
+    // sense for cross-interpretation situations, but without the wrapper, this will use the host
+    // sysroot, so asking it to behave like a target build makes even less sense.
+    cmd.env("MIRI_BE_RUSTC", "host"); // we better remember to *unset* this in the other phases!
 
     // Set rustdoc to us as well, so we can run doctests.
     if let Some(orig_rustdoc) = env::var_os("RUSTDOC") {
         cmd.env("MIRI_ORIG_RUSTDOC", orig_rustdoc);
     }
     cmd.env("RUSTDOC", &cargo_miri_path);
 
+    // Forward some crucial information to our own re-invocations.
+    cmd.env("MIRI_SYSROOT", miri_sysroot);
     cmd.env("MIRI_LOCAL_CRATES", local_crates(&metadata));
     if verbose > 0 {
         cmd.env("MIRI_VERBOSE", verbose.to_string()); // This makes the other phases verbose.
@@ -412,6 +412,9 @@ pub fn phase_rustc(mut args: impl Iterator<Item = String>, phase: RustcPhase) {
     // Arguments are treated very differently depending on whether this crate is
     // for interpretation by Miri, or for use by a build script / proc macro.
     if target_crate {
+        // Set the sysroot.
+        cmd.arg("--sysroot").arg(env::var_os("MIRI_SYSROOT").unwrap());
+
         // Forward arguments, but patched.
         let emit_flag = "--emit";
         // This hack helps bootstrap run standard library tests in Miri. The issue is as follows:
@@ -574,6 +577,12 @@ pub fn phase_runner(mut binary_args: impl Iterator<Item = String>, phase: Runner
         cmd.env(name, val);
     }
 
+    if phase != RunnerPhase::Rustdoc {
+        // Set the sysroot. Not necessary in rustdoc, where we already set the sysroot when invoking
+        // rustdoc itself, which will forward that flag when invoking rustc (i.e., us), so the flag
+        // is present in `info.args`.
+        cmd.arg("--sysroot").arg(env::var_os("MIRI_SYSROOT").unwrap());
+    }
     // Forward rustc arguments.
     // We need to patch "--extern" filenames because we forced a check-only
     // build without cargo knowing about that: replace `.rlib` suffix by

src/tools/miri/miri-script/src/commands.rs

@@ -2,6 +2,7 @@ use std::env;
 use std::ffi::OsString;
 use std::io::Write;
 use std::ops::Not;
+use std::path::PathBuf;
 use std::process;
 use std::thread;
 use std::time;
@@ -20,10 +21,11 @@ const JOSH_FILTER: &str =
 const JOSH_PORT: &str = "42042";
 
 impl MiriEnv {
-    fn build_miri_sysroot(&mut self, quiet: bool) -> Result<()> {
-        if self.sh.var("MIRI_SYSROOT").is_ok() {
+    /// Returns the location of the sysroot.
+    fn build_miri_sysroot(&mut self, quiet: bool) -> Result<PathBuf> {
+        if let Some(miri_sysroot) = self.sh.var_os("MIRI_SYSROOT") {
             // Sysroot already set, use that.
-            return Ok(());
+            return Ok(miri_sysroot.into());
         }
         let manifest_path = path!(self.miri_dir / "cargo-miri" / "Cargo.toml");
         let Self { toolchain, cargo_extra_flags, .. } = &self;
@@ -57,8 +59,8 @@ impl MiriEnv {
             .with_context(|| "`cargo miri setup` failed")?;
             panic!("`cargo miri setup` didn't fail again the 2nd time?");
         };
-        self.sh.set_var("MIRI_SYSROOT", output);
-        Ok(())
+        self.sh.set_var("MIRI_SYSROOT", &output);
+        Ok(output.into())
     }
 }
 
@@ -505,8 +507,10 @@ impl Command {
             flags.push("--edition=2021".into()); // keep in sync with `tests/ui.rs`.`
         }
 
-        // Prepare a sysroot.
-        e.build_miri_sysroot(/* quiet */ true)?;
+        // Prepare a sysroot, and add it to the flags.
+        let miri_sysroot = e.build_miri_sysroot(/* quiet */ true)?;
+        flags.push("--sysroot".into());
+        flags.push(miri_sysroot.into());
 
         // Then run the actual command. Also add MIRIFLAGS.
         let miri_manifest = path!(e.miri_dir / "Cargo.toml");

src/tools/miri/src/bin/miri.rs

@@ -271,25 +271,6 @@ fn run_compiler(
     callbacks: &mut (dyn rustc_driver::Callbacks + Send),
     using_internal_features: std::sync::Arc<std::sync::atomic::AtomicBool>,
 ) -> ! {
-    if target_crate {
-        // Miri needs a custom sysroot for target crates.
-        // If no `--sysroot` is given, the `MIRI_SYSROOT` env var is consulted to find where
-        // that sysroot lives, and that is passed to rustc.
-        let sysroot_flag = "--sysroot";
-        if !args.iter().any(|e| e.starts_with(sysroot_flag)) {
-            // Using the built-in default here would be plain wrong, so we *require*
-            // the env var to make sure things make sense.
-            let miri_sysroot = env::var("MIRI_SYSROOT").unwrap_or_else(|_| {
-                show_error!(
-                    "Miri was invoked in 'target' mode without `MIRI_SYSROOT` or `--sysroot` being set"
-                    )
-            });
-
-            args.push(sysroot_flag.to_owned());
-            args.push(miri_sysroot);
-        }
-    }
-
     // Don't insert `MIRI_DEFAULT_ARGS`, in particular, `--cfg=miri`, if we are building
     // a "host" crate. That may cause procedural macros (and probably build scripts) to
     // depend on Miri-only symbols, such as `miri_resolve_frame`:

src/tools/miri/tests/ui.rs

@@ -112,6 +112,13 @@ fn run_tests(
     config.program.envs.push(("RUST_BACKTRACE".into(), Some("1".into())));
 
     // Add some flags we always want.
+    config.program.args.push(
+        format!(
+            "--sysroot={}",
+            env::var("MIRI_SYSROOT").expect("MIRI_SYSROOT must be set to run the ui test suite")
+        )
+        .into(),
+    );
     config.program.args.push("-Dwarnings".into());
     config.program.args.push("-Dunused".into());
     config.program.args.push("-Ainternal_features".into());