cross-compile broken on 1.25.0

[infinity0]

Used to work on previous versions, now fails for any foreign architecture - I tested both ppc64el and mips, ppc64el trace below:

RUST_BACKTRACE=1 ./x.py build --config debian/config.toml -v --on-fail env
running: /usr/bin/cargo build --manifest-path /<<BUILDDIR>>/rustc-1.25.0+dfsg1/src/bootstrap/Cargo.toml --verbose --verbose
[..]
Copying stage2 std from stage1 (x86_64-unknown-linux-gnu -> powerpc64le-unknown-linux-gnu / powerpc64le-unknown-linux-gnu)
    c Libdir { compiler: Compiler { stage: 2, host: "powerpc64le-unknown-linux-gnu" }, target: "powerpc64le-unknown-linux-gnu" }
    > CleanTools { compiler: Compiler { stage: 2, host: "powerpc64le-unknown-linux-gnu" }, target: "powerpc64le-unknown-linux-gnu", mode: Libstd }
      c Assemble { target_compiler: Compiler { stage: 1, host: "powerpc64le-unknown-linux-gnu" } }
    < CleanTools { compiler: Compiler { stage: 2, host: "powerpc64le-unknown-linux-gnu" }, target: "powerpc64le-unknown-linux-gnu", mode: Libstd }
  < StdLink { compiler: Compiler { stage: 1, host: "x86_64-unknown-linux-gnu" }, target_compiler: Compiler { stage: 2, host: "powerpc64le-unknown-linux-gnu" }, target: "powerpc64le-unknown-linux-gnu" }
< Std { target: "powerpc64le-unknown-linux-gnu", compiler: Compiler { stage: 2, host: "powerpc64le-unknown-linux-gnu" } }
c Assemble { target_compiler: Compiler { stage: 2, host: "x86_64-unknown-linux-gnu" } }
> Test { compiler: Compiler { stage: 2, host: "x86_64-unknown-linux-gnu" }, target: "x86_64-unknown-linux-gnu" }
  c Std { target: "x86_64-unknown-linux-gnu", compiler: Compiler { stage: 2, host: "x86_64-unknown-linux-gnu" } }
  c Assemble { target_compiler: Compiler { stage: 1, host: "x86_64-unknown-linux-gnu" } }
  c Test { compiler: Compiler { stage: 1, host: "x86_64-unknown-linux-gnu" }, target: "x86_64-unknown-linux-gnu" }
Uplifting stage1 test (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
  c Assemble { target_compiler: Compiler { stage: 1, host: "x86_64-unknown-linux-gnu" } }
  > TestLink { compiler: Compiler { stage: 1, host: "x86_64-unknown-linux-gnu" }, target_compiler: Compiler { stage: 2, host: "x86_64-unknown-linux-gnu" }, target: "x86_64-unknown-linux-gnu" }
Copying stage2 test from stage1 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu)
    c Libdir { compiler: Compiler { stage: 2, host: "x86_64-unknown-linux-gnu" }, target: "x86_64-unknown-linux-gnu" }
thread 'main' panicked at 'File::open(stamp) failed with No such file or directory (os error 2)', bootstrap/util.rs:71:8
stack backtrace:
   0: std::sys::unix::backtrace::tracing::imp::unwind_backtrace
   1: std::sys_common::backtrace::print
   2: std::panicking::default_hook::{{closure}}
   3: std::panicking::default_hook
   4: std::panicking::rust_panic_with_hook
   5: std::panicking::begin_panic
   6: std::panicking::begin_panic_fmt
   7: bootstrap::util::read_stamp_file
             at bootstrap/util.rs:71
   8: bootstrap::compile::add_to_sysroot
             at bootstrap/compile.rs:936
   9: <bootstrap::compile::TestLink as bootstrap::builder::Step>::run
             at bootstrap/compile.rs:417
  10: bootstrap::builder::Builder::ensure
             at bootstrap/builder.rs:821
  11: <bootstrap::compile::Test as bootstrap::builder::Step>::run
             at bootstrap/compile.rs:351
  12: bootstrap::builder::Builder::ensure
             at bootstrap/builder.rs:821
  13: <bootstrap::compile::Test as bootstrap::builder::Step>::make_run
             at bootstrap/compile.rs:327
  14: bootstrap::builder::StepDescription::maybe_run
             at bootstrap/builder.rs:185
  15: bootstrap::builder::StepDescription::run
             at bootstrap/builder.rs:205
  16: bootstrap::builder::Builder::run
             at bootstrap/builder.rs:408
  17: bootstrap::Build::build
             at bootstrap/lib.rs:390
  18: bootstrap::main
             at bootstrap/bin/main.rs:29
  19: std::rt::lang_start::{{closure}}
             at src/libstd/rt.rs:74
  20: std::panicking::try::do_call
  21: __rust_maybe_catch_panic
  22: std::rt::lang_start_internal
  23: std::rt::lang_start
             at src/libstd/rt.rs:74
  24: main
  25: __libc_start_main
  26: _start
Traceback (most recent call last):
  File "./x.py", line 20, in <module>
    bootstrap.main()
  File "/<<BUILDDIR>>/rustc-1.25.0+dfsg1/src/bootstrap/bootstrap.py", line 760, in main
    bootstrap()
  File "/<<BUILDDIR>>/rustc-1.25.0+dfsg1/src/bootstrap/bootstrap.py", line 751, in bootstrap
    run(args, env=env, verbose=build.verbose)
  File "/<<BUILDDIR>>/rustc-1.25.0+dfsg1/src/bootstrap/bootstrap.py", line 149, in run
    raise RuntimeError(err)
RuntimeError: failed to run: /<<BUILDDIR>>/rustc-1.25.0+dfsg1/build/bootstrap/debug/bootstrap build --config debian/config.toml -v --on-fail env

config.toml:

[build]
submodules = false
vendor = true
locked-deps = false
verbose = 2

rustc = "/usr/bin/rustc"
cargo = "/usr/bin/cargo"

build = "x86_64-unknown-linux-gnu"
host = ["powerpc64le-unknown-linux-gnu"]
target = ["powerpc64le-unknown-linux-gnu"]

docs = true

[install]
prefix = "/usr"

[target.x86_64-unknown-linux-gnu]
llvm-config = "/usr/lib/llvm-6.0/bin/llvm-config"

[target.powerpc64le-unknown-linux-gnu]
llvm-config = "/usr/lib/llvm-6.0/bin/llvm-config"

[llvm]
link-shared = true

[rust]
use-jemalloc = false
optimize = true
dist-src = false

channel = "stable"

debuginfo = true
debuginfo-lines = true
debuginfo-only-std = false
rpath = false

[nikomatsakis]

@infinity0 any chance you could use https://github.com/rust-lang-nursery/cargo-bisect-rustc to narrow down the PR (or nightly) where this became a problem? Should be quite easy to do, I would think!

[nikomatsakis]

cc @Mark-Simulacrum -- any thoughts on changes to x.py that might cause this problem?

I take back what I said about cargo-bisect-rustc, I didn't realize these errors were coming from x.py. It'd still be nice to narrow down when it started to fail.

[nikomatsakis]

triage: P-high

Probably not T-compiler so much as build time. But I'm calling P-high so we can keep a bit of an eye on it.

[Mark-Simulacrum]

@infinity0 Is there a chance you could test master as well and confirm if it fails there? It'd be nice to have a full log as well if possible. It looks like we're expecting the libtest stamp to exist (which seems reasonable) but it doesn't so we're either not building libtest or it's directory is being cleared out for some reason...

[infinity0]

@Mark-Simulacrum with current master eeea94c I am getting cargo:warning=cc: error: ../../libcompiler_builtins/compiler-rt/lib/builtins/absvdi2.c: No such file or directory which seems to be a different issue, I can retry once that is fixed though.

[nikomatsakis]

Question: @Mark-Simulacrum do you see this as an "infra" issue or something else?

[pnkfelix]

going to attempt to locally reproduce

[Mark-Simulacrum]

I think it's an infra issue; I need to investigate why this is broken and why it doesn't break our CI since we have similar builders IIRC.

[pnkfelix]

@Mark-Simulacrum I'm going to add you to the assignee list as I may not be able to have time to reproduce this in the near future.

[jimblandy]

@Mark-Simulacrum For the missing absvdi2.c, might you have an empty src/libcompiler_builtins/compiler-rt directory? If that's the case, then this may help:

$ git submodule update --init --recursive

I don't understand quite what's going on here, but there's been a lot of churn in bootstrap.py related to submodules lately, so maybe something is not working properly there.

[nikomatsakis]

Visiting for compiler team triage. As this is a @rust-lang/infra issue, should we remove T-compiler tag? @rust-lang/infra, do you all think you have this in hand?

[Mark-Simulacrum]

Either I or @alexcrichton probably have to diagnose this. I don't think it's a compiler issue at this point though; we can remove T-compiler.

[Mark-Simulacrum]

@infinity0 Can you check if ./x.py dist works? ./x.py build fails for me as well, but our Docker image only runs dist. I'm guessing here, but one problem might be that build is missing a dependency on the build target from cross targets when only powerpc is present in hosts and targets; it might be enough to add x86_64-unknown-linux-gnu to those in config.toml.

So to summarize:

• Can you verify if ./x.py dist works for you with that toml? Primarily asking because we seem to test a similar configuration for the dist builder for powerpc.
• Can you check if adding the build triple (x86_64-unknown-linux-gnu) to host and target arrays in config.toml also fixes the build?

[infinity0]

Original issue seems to be fixed with 1.26.0, didn't yet check with master.

[Mark-Simulacrum]

If you could check master and beta (with perhaps higher priority on beta) to verify that this works there we can then close.