Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Utilize -Zbinary-dep-depinfo in rustbuild #63470

Merged
merged 1 commit into from Aug 16, 2019

Conversation

@Mark-Simulacrum
Copy link
Member

commented Aug 11, 2019

We no longer utilize stamp-file mtimes at all inside rustbuild, and a future PR may be able to entirely eliminate them by eagerly copying to the appropriate sysroot. The only mtime-based dependency tracking left is for documentation because we lie to Cargo about the rustdoc binary, so Cargo does not track changes to the real binary, and codegen-backends because binary-dep-depinfo does not emit that information into the depfiles.

Both of these are fixable in the longer term but this existing patch gives us the following benefits:

  • We no longer delete Cargo target directories manually within a stage. Cross-stage, changes to codegen backends will still clear out target directories. This means that incremental state persists across individual steps (e.g., rebuilding libstd does not clear out librustc incremental state). Fixes #54712.
  • Dependency tracking across steps within a given stage is now fully precise. We will not clear out all codegen backend dependencies due to changes in librustc_driver, for example, only deleting the final librustc_codegen_llvm crate. Fixes #54008, fixes #50481.
  • We properly track codegen backends as a dependency (equivalent to rustc) across changes. Fixes #53284, and fixes #52719.
  • Cross-stage dependency tracking of crates is also much more accurate and reliable. Most likely fixes #49979 (but no reproduction steps in that issue). Fixes #59105.

cc #63012

@Mark-Simulacrum

This comment has been minimized.

Copy link
Member Author

commented Aug 11, 2019

To be clear this is not intended to land as-is since it won't actually work as-is since the rustc needs to be overridden -- the currently nightly can't bootstrap successfully I believe due to MaybeUninit changes -- I'm using 2b78e10 as the base commit via rustup-toolchain-install-master locally and we shouldn't couple the depdep changes to cfg(bootstrap).

@rust-highfive

This comment was marked as off-topic.

Copy link
Collaborator

commented Aug 11, 2019

The job mingw-check of your PR failed (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
2019-08-11T19:05:39.3128354Z ##[command]git remote add origin https://github.com/rust-lang/rust
2019-08-11T19:05:39.3313858Z ##[command]git config gc.auto 0
2019-08-11T19:05:39.3394797Z ##[command]git config --get-all http.https://github.com/rust-lang/rust.extraheader
2019-08-11T19:05:39.3447350Z ##[command]git config --get-all http.proxy
2019-08-11T19:05:39.3584964Z ##[command]git -c http.extraheader="AUTHORIZATION: basic ***" fetch --force --tags --prune --progress --no-recurse-submodules --depth=2 origin +refs/heads/*:refs/remotes/origin/* +refs/pull/63470/merge:refs/remotes/pull/63470/merge
---
2019-08-11T19:06:12.9069125Z do so (now or later) by using -b with the checkout command again. Example:
2019-08-11T19:06:12.9069330Z 
2019-08-11T19:06:12.9069530Z   git checkout -b <new-branch-name>
2019-08-11T19:06:12.9069558Z 
2019-08-11T19:06:12.9069601Z HEAD is now at 8844b12a1 Merge 0afbfe248460b7fccf00849de9ac672862124a67 into 8a068699a24de306334a1f66b9a83552766d853c
2019-08-11T19:06:12.9212756Z ##[section]Starting: Collect CPU-usage statistics in the background
2019-08-11T19:06:12.9215188Z ==============================================================================
2019-08-11T19:06:12.9215259Z Task         : Bash
2019-08-11T19:06:12.9215300Z Description  : Run a Bash script on macOS, Linux, or Windows
---
2019-08-11T19:11:09.1846753Z Checking rustdoc artifacts (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
2019-08-11T19:11:09.1868402Z Checking std artifacts (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
2019-08-11T19:11:09.5853459Z    Compiling cc v1.0.35
2019-08-11T19:11:09.5853787Z     Checking core v0.0.0 (/checkout/src/libcore)
2019-08-11T19:11:15.0181689Z error[E0093]: unrecognized intrinsic function: `uninit`
2019-08-11T19:11:15.0183952Z     |
2019-08-11T19:11:15.0183952Z     |
2019-08-11T19:11:15.0184432Z 728 |     pub fn uninit<T>() -> T;
2019-08-11T19:11:15.0184755Z     |     ^^^^^^^^^^^^^^^^^^^^^^^^ unrecognized intrinsic
2019-08-11T19:11:18.7086112Z    Compiling libc v0.2.60
2019-08-11T19:11:19.6179404Z    Compiling unwind v0.0.0 (/checkout/src/libunwind)
2019-08-11T19:11:20.1175990Z    Compiling build_helper v0.1.0 (/checkout/src/build_helper)
2019-08-11T19:11:20.3094492Z error: aborting due to previous error
2019-08-11T19:11:20.3094492Z error: aborting due to previous error
2019-08-11T19:11:20.3094599Z 
2019-08-11T19:11:20.3097860Z For more information about this error, try `rustc --explain E0093`.
2019-08-11T19:11:20.4221159Z error: Could not compile `core`.
2019-08-11T19:11:20.4221981Z warning: build failed, waiting for other jobs to finish...
2019-08-11T19:11:21.1342126Z error: build failed
2019-08-11T19:11:21.1375353Z command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "check" "--target" "x86_64-unknown-linux-gnu" "-j" "2" "--release" "--color" "always" "--features" "panic-unwind backtrace compiler-builtins-c" "--manifest-path" "/checkout/src/libstd/Cargo.toml" "--message-format" "json"
2019-08-11T19:11:21.1386698Z failed to run: /checkout/obj/build/bootstrap/debug/bootstrap check
2019-08-11T19:11:21.1387050Z Build completed unsuccessfully in 0:02:28
2019-08-11T19:11:21.1387050Z Build completed unsuccessfully in 0:02:28
2019-08-11T19:11:29.9243239Z ##[error]Bash exited with code '1'.
2019-08-11T19:11:29.9278545Z ##[section]Starting: Checkout
2019-08-11T19:11:29.9280284Z ==============================================================================
2019-08-11T19:11:29.9280353Z Task         : Get sources
2019-08-11T19:11:29.9280396Z Description  : Get sources from a repository. Supports Git, TfsVC, and SVN repositories.

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

@alexcrichton
Copy link
Member

left a comment

That's an impressive list of fixed issues!

This may also want to apply https://gist.github.com/alexcrichton/e0840447982f93cb22bbd98878d2c205 to get colors back in error messages and fix the progress bar with pipelining

Show resolved Hide resolved src/bootstrap/bin/rustc.rs Outdated
Show resolved Hide resolved src/bootstrap/compile.rs

@Mark-Simulacrum Mark-Simulacrum force-pushed the Mark-Simulacrum:rustc-depdep branch 2 times, most recently from e4f7726 to 0be63bd Aug 12, 2019

@rust-highfive

This comment has been minimized.

Copy link
Collaborator

commented Aug 12, 2019

The job x86_64-gnu-llvm-6.0 of your PR failed (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
2019-08-12T18:58:37.8552072Z ##[command]git remote add origin https://github.com/rust-lang/rust
2019-08-12T18:58:37.8730136Z ##[command]git config gc.auto 0
2019-08-12T18:58:37.8807199Z ##[command]git config --get-all http.https://github.com/rust-lang/rust.extraheader
2019-08-12T18:58:37.8857854Z ##[command]git config --get-all http.proxy
2019-08-12T18:58:37.8988671Z ##[command]git -c http.extraheader="AUTHORIZATION: basic ***" fetch --force --tags --prune --progress --no-recurse-submodules --depth=2 origin +refs/heads/*:refs/remotes/origin/* +refs/pull/63470/merge:refs/remotes/pull/63470/merge
---
2019-08-12T18:59:13.5557268Z do so (now or later) by using -b with the checkout command again. Example:
2019-08-12T18:59:13.5557320Z 
2019-08-12T18:59:13.5557745Z   git checkout -b <new-branch-name>
2019-08-12T18:59:13.5557775Z 
2019-08-12T18:59:13.5557825Z HEAD is now at d29203c7d Merge 0be63bd405b1a78f07e528b60972a0e05ee9d1f4 into 60960a260f7b5c695fd0717311d72ce62dd4eb43
2019-08-12T18:59:13.5718160Z ##[section]Starting: Collect CPU-usage statistics in the background
2019-08-12T18:59:13.5721160Z ==============================================================================
2019-08-12T18:59:13.5721206Z Task         : Bash
2019-08-12T18:59:13.5721240Z Description  : Run a Bash script on macOS, Linux, or Windows
---
2019-08-12T19:57:52.7030279Z .................................................................................................... 1300/8875
2019-08-12T19:57:59.2473769Z .................................................................................................... 1400/8875
2019-08-12T19:58:05.4539230Z .................................................................................................... 1500/8875
2019-08-12T19:58:16.0749222Z ....................................................................................i............... 1600/8875
2019-08-12T19:58:23.5643833Z i................................................................................................... 1700/8875
2019-08-12T19:58:30.0542706Z ...........................................................................iiiii.................... 1800/8875
2019-08-12T19:58:51.5059152Z .................................................................................................... 2000/8875
2019-08-12T19:58:53.8218187Z .................................................................................................... 2100/8875
2019-08-12T19:58:56.3277882Z .................................................................................................... 2200/8875
2019-08-12T19:59:03.6203121Z .................................................................................................... 2300/8875
---
2019-08-12T20:02:49.0796314Z ......................................................F............................................. 5300/8875
2019-08-12T20:02:56.0246796Z ........i........................................................................................... 5400/8875
2019-08-12T20:03:01.1637736Z .................................................................................................... 5500/8875
2019-08-12T20:03:12.8930664Z .................................................................................................... 5600/8875
2019-08-12T20:03:26.6177343Z ...ii...i..ii...........i........................................................................... 5700/8875
2019-08-12T20:03:41.1958596Z .................................................................................................... 5900/8875
2019-08-12T20:03:45.7230242Z .................................................................................................... 6000/8875
2019-08-12T20:03:45.7230242Z .................................................................................................... 6000/8875
2019-08-12T20:03:59.3523629Z ....i..ii........................................................................................... 6100/8875
2019-08-12T20:04:17.3355668Z ...............................................i.................................................... 6300/8875
2019-08-12T20:04:19.3514211Z .................................................................................................... 6400/8875
2019-08-12T20:04:21.6889439Z ...................i................................................................................ 6500/8875
2019-08-12T20:04:25.9978187Z .................................................................................................... 6600/8875
---
2019-08-12T20:08:16.9345838Z failures:
2019-08-12T20:08:16.9367396Z 
2019-08-12T20:08:16.9368023Z ---- [ui] ui/macros/builtin-std-paths.rs stdout ----
2019-08-12T20:08:16.9368691Z 
2019-08-12T20:08:16.9369270Z error: test compilation failed although it shouldn't!
2019-08-12T20:08:16.9369785Z status: exit code: 1
2019-08-12T20:08:16.9370864Z command: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "/checkout/src/test/ui/macros/builtin-std-paths.rs" "-Zthreads=1" "--target=x86_64-unknown-linux-gnu" "--error-format" "json" "-Zui-testing" "--emit" "metadata" "-C" "prefer-dynamic" "--out-dir" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/macros/builtin-std-paths" "-Crpath" "-O" "-Cdebuginfo=0" "-Zunstable-options" "-Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/macros/builtin-std-paths/auxiliary" "-A" "unused"
2019-08-12T20:08:16.9372149Z ------------------------------------------
2019-08-12T20:08:16.9372400Z 
2019-08-12T20:08:16.9372779Z ------------------------------------------
2019-08-12T20:08:16.9373001Z stderr:
2019-08-12T20:08:16.9373001Z stderr:
2019-08-12T20:08:16.9373718Z ------------------------------------------
2019-08-12T20:08:16.9373985Z error[E0433]: failed to resolve: could not find `Hash` in `hash`
2019-08-12T20:08:16.9374669Z    |
2019-08-12T20:08:16.9374834Z LL |     core::hash::Hash,
2019-08-12T20:08:16.9374834Z LL |     core::hash::Hash,
2019-08-12T20:08:16.9375019Z    |                 ^^^^ could not find `Hash` in `hash`
2019-08-12T20:08:16.9375164Z 
2019-08-12T20:08:16.9375328Z error[E0433]: failed to resolve: could not find `Hash` in `hash`
2019-08-12T20:08:16.9375929Z    |
2019-08-12T20:08:16.9376501Z LL |     std::hash::Hash,
2019-08-12T20:08:16.9376501Z LL |     std::hash::Hash,
2019-08-12T20:08:16.9376681Z    |                ^^^^ could not find `Hash` in `hash`
2019-08-12T20:08:16.9377175Z error: aborting due to 2 previous errors
2019-08-12T20:08:16.9377299Z 
2019-08-12T20:08:16.9377716Z For more information about this error, try `rustc --explain E0433`.
2019-08-12T20:08:16.9377849Z 
---
2019-08-12T20:08:16.9400139Z thread 'main' panicked at 'Some tests failed', src/tools/compiletest/src/main.rs:536:22
2019-08-12T20:08:16.9400428Z note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace.
2019-08-12T20:08:16.9416666Z 
2019-08-12T20:08:16.9418035Z 
2019-08-12T20:08:16.9420739Z command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools-bin/compiletest" "--compile-lib-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib" "--run-lib-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/lib" "--rustc-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "--src-base" "/checkout/src/test/ui" "--build-base" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui" "--stage-id" "stage2-x86_64-unknown-linux-gnu" "--mode" "ui" "--target" "x86_64-unknown-linux-gnu" "--host" "x86_64-unknown-linux-gnu" "--llvm-filecheck" "/usr/lib/llvm-6.0/bin/FileCheck" "--host-rustcflags" "-Crpath -O -Cdebuginfo=0 -Zunstable-options  -Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "--target-rustcflags" "-Crpath -O -Cdebuginfo=0 -Zunstable-options  -Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "--docck-python" "/usr/bin/python2.7" "--lldb-python" "/usr/bin/python2.7" "--gdb" "/usr/bin/gdb" "--quiet" "--llvm-version" "6.0.0\n" "--system-llvm" "--cc" "" "--cxx" "" "--cflags" "" "--llvm-components" "" "--llvm-cxxflags" "" "--adb-path" "adb" "--adb-test-dir" "/data/tmp/work" "--android-cross-path" "" "--color" "always"
2019-08-12T20:08:16.9421235Z 
2019-08-12T20:08:16.9421287Z 
2019-08-12T20:08:16.9430791Z failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test
2019-08-12T20:08:16.9430932Z Build completed unsuccessfully in 1:02:46
2019-08-12T20:08:16.9430932Z Build completed unsuccessfully in 1:02:46
2019-08-12T20:08:18.1170799Z ##[error]Bash exited with code '1'.
2019-08-12T20:08:18.1212444Z ##[section]Starting: Checkout
2019-08-12T20:08:18.1214185Z ==============================================================================
2019-08-12T20:08:18.1214230Z Task         : Get sources
2019-08-12T20:08:18.1214284Z Description  : Get sources from a repository. Supports Git, TfsVC, and SVN repositories.

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

@Mark-Simulacrum Mark-Simulacrum force-pushed the Mark-Simulacrum:rustc-depdep branch 2 times, most recently from c034578 to cecdcdb Aug 12, 2019

@Mark-Simulacrum

This comment has been minimized.

Copy link
Member Author

commented Aug 12, 2019

Just force pushed what should hopefully be a fix for the cfg(bootstrap) bit; with stage0.txt modified to the beta scheduled to be released Tuesday we can likely land this PR in one go to avoid wasting two cycles.

@Mark-Simulacrum Mark-Simulacrum force-pushed the Mark-Simulacrum:rustc-depdep branch 2 times, most recently from 5739811 to beab7fb Aug 12, 2019

@rust-highfive

This comment has been minimized.

Copy link
Collaborator

commented Aug 13, 2019

The job mingw-check of your PR failed (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
2019-08-12T23:53:16.4280624Z ##[command]git remote add origin https://github.com/rust-lang/rust
2019-08-12T23:53:16.4479176Z ##[command]git config gc.auto 0
2019-08-12T23:53:16.4540035Z ##[command]git config --get-all http.https://github.com/rust-lang/rust.extraheader
2019-08-12T23:53:16.4590193Z ##[command]git config --get-all http.proxy
2019-08-12T23:53:16.4723820Z ##[command]git -c http.extraheader="AUTHORIZATION: basic ***" fetch --force --tags --prune --progress --no-recurse-submodules --depth=2 origin +refs/heads/*:refs/remotes/origin/* +refs/pull/63470/merge:refs/remotes/pull/63470/merge
---
2019-08-12T23:53:53.4139249Z do so (now or later) by using -b with the checkout command again. Example:
2019-08-12T23:53:53.4139296Z 
2019-08-12T23:53:53.4139496Z   git checkout -b <new-branch-name>
2019-08-12T23:53:53.4139524Z 
2019-08-12T23:53:53.4139569Z HEAD is now at 1a9aa9925 Merge beab7fb5aff44c0d2f79407c3efd489fec64a802 into 60960a260f7b5c695fd0717311d72ce62dd4eb43
2019-08-12T23:53:53.4294082Z ##[section]Starting: Collect CPU-usage statistics in the background
2019-08-12T23:53:53.4296936Z ==============================================================================
2019-08-12T23:53:53.4296995Z Task         : Bash
2019-08-12T23:53:53.4297044Z Description  : Run a Bash script on macOS, Linux, or Windows
---
2019-08-12T23:59:24.4589361Z     Checking backtrace v0.3.34
2019-08-12T23:59:26.4508250Z     Checking rustc-std-workspace-alloc v1.0.0 (/checkout/src/tools/rustc-std-workspace-alloc)
2019-08-12T23:59:26.4512106Z     Checking panic_unwind v0.0.0 (/checkout/src/libpanic_unwind)
2019-08-12T23:59:26.6072567Z     Checking hashbrown v0.4.0
2019-08-12T23:59:28.4944861Z error: use of item 'core::try' that will be deprecated in future version 1.39.0: use the `?` operator instead
2019-08-12T23:59:28.4946216Z     |
2019-08-12T23:59:28.4946216Z     |
2019-08-12T23:59:28.4946854Z 524 |     r#try,
2019-08-12T23:59:28.4947732Z     |
2019-08-12T23:59:28.4948199Z     = note: `-D deprecated-in-future` implied by `-D warnings`
2019-08-12T23:59:28.4948367Z 
2019-08-12T23:59:30.3433869Z error: aborting due to previous error
2019-08-12T23:59:30.3433869Z error: aborting due to previous error
2019-08-12T23:59:30.3434560Z 
2019-08-12T23:59:30.3771646Z error: Could not compile `std`.
2019-08-12T23:59:30.3771725Z 
2019-08-12T23:59:30.3772235Z To learn more, run the command again with --verbose.
2019-08-12T23:59:30.3794089Z command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "check" "--target" "x86_64-unknown-linux-gnu" "-Zbinary-dep-depinfo" "-j" "2" "--release" "--color" "always" "--features" "panic-unwind backtrace compiler-builtins-c" "--manifest-path" "/checkout/src/libstd/Cargo.toml" "--message-format" "json-render-diagnostics"
2019-08-12T23:59:30.3800655Z failed to run: /checkout/obj/build/bootstrap/debug/bootstrap check
2019-08-12T23:59:30.3800939Z Build completed unsuccessfully in 0:02:52
2019-08-12T23:59:30.3800939Z Build completed unsuccessfully in 0:02:52
2019-08-12T23:59:36.9017102Z ##[error]Bash exited with code '1'.
2019-08-12T23:59:36.9058225Z ##[section]Starting: Checkout
2019-08-12T23:59:36.9060323Z ==============================================================================
2019-08-12T23:59:36.9060402Z Task         : Get sources
2019-08-12T23:59:36.9060452Z Description  : Get sources from a repository. Supports Git, TfsVC, and SVN repositories.

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

@Mark-Simulacrum Mark-Simulacrum force-pushed the Mark-Simulacrum:rustc-depdep branch from beab7fb to cc0bdab Aug 13, 2019

@rust-highfive

This comment has been minimized.

Copy link
Collaborator

commented Aug 13, 2019

The job x86_64-gnu-llvm-6.0 of your PR failed (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
2019-08-13T00:49:15.8870884Z ##[command]git remote add origin https://github.com/rust-lang/rust
2019-08-13T00:49:15.9049632Z ##[command]git config gc.auto 0
2019-08-13T00:49:15.9122297Z ##[command]git config --get-all http.https://github.com/rust-lang/rust.extraheader
2019-08-13T00:49:15.9171555Z ##[command]git config --get-all http.proxy
2019-08-13T00:49:15.9308213Z ##[command]git -c http.extraheader="AUTHORIZATION: basic ***" fetch --force --tags --prune --progress --no-recurse-submodules --depth=2 origin +refs/heads/*:refs/remotes/origin/* +refs/pull/63470/merge:refs/remotes/pull/63470/merge
---
2019-08-13T00:49:51.4556209Z do so (now or later) by using -b with the checkout command again. Example:
2019-08-13T00:49:51.4556234Z 
2019-08-13T00:49:51.4556405Z   git checkout -b <new-branch-name>
2019-08-13T00:49:51.4556446Z 
2019-08-13T00:49:51.4556486Z HEAD is now at 0b563536d Merge cc0bdabef81d6a5f66caeefb3744145f22f7fa7f into 60960a260f7b5c695fd0717311d72ce62dd4eb43
2019-08-13T00:49:51.4729816Z ##[section]Starting: Collect CPU-usage statistics in the background
2019-08-13T00:49:51.4733004Z ==============================================================================
2019-08-13T00:49:51.4733071Z Task         : Bash
2019-08-13T00:49:51.4733109Z Description  : Run a Bash script on macOS, Linux, or Windows
---
2019-08-13T01:46:59.6090570Z .................................................................................................... 1300/8875
2019-08-13T01:47:06.2606012Z .................................................................................................... 1400/8875
2019-08-13T01:47:11.8981421Z .................................................................................................... 1500/8875
2019-08-13T01:47:21.9724751Z ....................................................................................i............... 1600/8875
2019-08-13T01:47:29.3827132Z i................................................................................................... 1700/8875
2019-08-13T01:47:35.8508370Z ...........................................................................iiiii.................... 1800/8875
2019-08-13T01:47:57.0335565Z .................................................................................................... 2000/8875
2019-08-13T01:47:59.4164377Z .................................................................................................... 2100/8875
2019-08-13T01:48:02.0424979Z .................................................................................................... 2200/8875
2019-08-13T01:48:09.4670696Z .................................................................................................... 2300/8875
---
2019-08-13T01:51:58.4298800Z .................................................................................................... 5300/8875
2019-08-13T01:52:05.5776011Z ........i........................................................................................... 5400/8875
2019-08-13T01:52:10.8147842Z .................................................................................................... 5500/8875
2019-08-13T01:52:22.8010324Z .................................................................................................... 5600/8875
2019-08-13T01:52:36.4495964Z ...ii...i..ii...........i........................................................................... 5700/8875
2019-08-13T01:52:51.0155298Z .................................................................................................... 5900/8875
2019-08-13T01:52:55.6030007Z .................................................................................................... 6000/8875
2019-08-13T01:52:55.6030007Z .................................................................................................... 6000/8875
2019-08-13T01:53:09.1527375Z ....i..ii........................................................................................... 6100/8875
2019-08-13T01:53:27.2263613Z ...............................................i.................................................... 6300/8875
2019-08-13T01:53:29.3326601Z .................................................................................................... 6400/8875
2019-08-13T01:53:31.7576693Z ...................i................................................................................ 6500/8875
2019-08-13T01:53:36.1665260Z .................................................................................................... 6600/8875
---
2019-08-13T01:57:25.5834481Z 
2019-08-13T01:57:25.5835094Z ---- [ui] ui/lint/lint-qualification.rs stdout ----
2019-08-13T01:57:25.5835363Z diff of stderr:
2019-08-13T01:57:25.5835483Z 
2019-08-13T01:57:25.5835812Z + warning: use of deprecated item 'try': use the `?` operator instead
2019-08-13T01:57:25.5836164Z +   --> $DIR/lint-qualification.rs:13:36
2019-08-13T01:57:25.5836316Z +    |
2019-08-13T01:57:25.5836652Z + LL |     let _ = || -> Result<(), ()> { try!(Ok(())); Ok(()) }; // issue #37345
2019-08-13T01:57:25.5836926Z +    |
2019-08-13T01:57:25.5837059Z +    = note: `#[warn(deprecated)]` on by default
2019-08-13T01:57:25.5837178Z + 
2019-08-13T01:57:25.5837289Z 1 error: unnecessary qualification
2019-08-13T01:57:25.5837289Z 1 error: unnecessary qualification
2019-08-13T01:57:25.5837592Z 2   --> $DIR/lint-qualification.rs:10:5
2019-08-13T01:57:25.5837755Z 3    |
2019-08-13T01:57:25.5837856Z 
2019-08-13T01:57:25.5837970Z 
2019-08-13T01:57:25.5838092Z The actual stderr differed from the expected stderr.
2019-08-13T01:57:25.5838454Z Actual stderr saved to /checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/lint/lint-qualification/lint-qualification.stderr
2019-08-13T01:57:25.5839779Z To update references, rerun the tests and pass the `--bless` flag
2019-08-13T01:57:25.5840264Z To only update this specific test, also pass `--test-args lint/lint-qualification.rs`
2019-08-13T01:57:25.5840600Z error: 1 errors occurred comparing output.
2019-08-13T01:57:25.5840748Z status: exit code: 1
2019-08-13T01:57:25.5840748Z status: exit code: 1
2019-08-13T01:57:25.5841826Z command: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "/checkout/src/test/ui/lint/lint-qualification.rs" "-Zthreads=1" "--target=x86_64-unknown-linux-gnu" "--error-format" "json" "-Zui-testing" "-C" "prefer-dynamic" "--out-dir" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/lint/lint-qualification" "-Crpath" "-O" "-Cdebuginfo=0" "-Zunstable-options" "-Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/lint/lint-qualification/auxiliary" "-A" "unused"
2019-08-13T01:57:25.5842783Z ------------------------------------------
2019-08-13T01:57:25.5842934Z 
2019-08-13T01:57:25.5843234Z ------------------------------------------
2019-08-13T01:57:25.5843405Z stderr:
2019-08-13T01:57:25.5843405Z stderr:
2019-08-13T01:57:25.5843693Z ------------------------------------------
2019-08-13T01:57:25.5844021Z warning: use of deprecated item 'try': use the `?` operator instead
2019-08-13T01:57:25.5844707Z    |
2019-08-13T01:57:25.5844707Z    |
2019-08-13T01:57:25.5845016Z LL |     let _ = || -> Result<(), ()> { try!(Ok(())); Ok(()) }; // issue #37345
2019-08-13T01:57:25.5845326Z    |
2019-08-13T01:57:25.5845456Z    = note: `#[warn(deprecated)]` on by default
2019-08-13T01:57:25.5845563Z 
2019-08-13T01:57:25.5845678Z error: unnecessary qualification
2019-08-13T01:57:25.5845678Z error: unnecessary qualification
2019-08-13T01:57:25.5845988Z   --> /checkout/src/test/ui/lint/lint-qualification.rs:10:5
2019-08-13T01:57:25.5846164Z    |
2019-08-13T01:57:25.5846286Z LL |     foo::bar(); //~ ERROR: unnecessary qualification
2019-08-13T01:57:25.5846534Z    |
2019-08-13T01:57:25.5846667Z note: lint level defined here
2019-08-13T01:57:25.5846988Z   --> /checkout/src/test/ui/lint/lint-qualification.rs:1:9
2019-08-13T01:57:25.5847140Z    |
---
2019-08-13T01:57:25.5872356Z thread 'main' panicked at 'Some tests failed', src/tools/compiletest/src/main.rs:536:22
2019-08-13T01:57:25.5872836Z note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace.
2019-08-13T01:57:25.5887552Z 
2019-08-13T01:57:25.5887816Z 
2019-08-13T01:57:25.5890275Z command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools-bin/compiletest" "--compile-lib-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib" "--run-lib-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/lib" "--rustc-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "--src-base" "/checkout/src/test/ui" "--build-base" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui" "--stage-id" "stage2-x86_64-unknown-linux-gnu" "--mode" "ui" "--target" "x86_64-unknown-linux-gnu" "--host" "x86_64-unknown-linux-gnu" "--llvm-filecheck" "/usr/lib/llvm-6.0/bin/FileCheck" "--host-rustcflags" "-Crpath -O -Cdebuginfo=0 -Zunstable-options  -Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "--target-rustcflags" "-Crpath -O -Cdebuginfo=0 -Zunstable-options  -Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "--docck-python" "/usr/bin/python2.7" "--lldb-python" "/usr/bin/python2.7" "--gdb" "/usr/bin/gdb" "--quiet" "--llvm-version" "6.0.0\n" "--system-llvm" "--cc" "" "--cxx" "" "--cflags" "" "--llvm-components" "" "--llvm-cxxflags" "" "--adb-path" "adb" "--adb-test-dir" "/data/tmp/work" "--android-cross-path" "" "--color" "always"
2019-08-13T01:57:25.5891302Z 
2019-08-13T01:57:25.5891556Z 
2019-08-13T01:57:25.5896908Z failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test
2019-08-13T01:57:25.5896995Z Build completed unsuccessfully in 1:01:46
2019-08-13T01:57:25.5896995Z Build completed unsuccessfully in 1:01:46
2019-08-13T01:57:26.2658301Z ##[error]Bash exited with code '1'.
2019-08-13T01:57:26.2716469Z ##[section]Starting: Checkout
2019-08-13T01:57:26.2717898Z ==============================================================================
2019-08-13T01:57:26.2717938Z Task         : Get sources
2019-08-13T01:57:26.2717973Z Description  : Get sources from a repository. Supports Git, TfsVC, and SVN repositories.

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

@Mark-Simulacrum Mark-Simulacrum force-pushed the Mark-Simulacrum:rustc-depdep branch from 40da969 to 417f9ea Aug 13, 2019

@rust-highfive

This comment has been minimized.

Copy link
Collaborator

commented Aug 13, 2019

The job mingw-check of your PR failed (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
2019-08-13T21:05:29.6285889Z ##[command]git remote add origin https://github.com/rust-lang/rust
2019-08-13T21:05:29.6299092Z ##[command]git config gc.auto 0
2019-08-13T21:05:30.6272037Z ##[command]git config --get-all http.https://github.com/rust-lang/rust.extraheader
2019-08-13T21:05:30.6277301Z ##[command]git config --get-all http.proxy
2019-08-13T21:05:30.6282982Z ##[command]git -c http.extraheader="AUTHORIZATION: basic ***" fetch --force --tags --prune --progress --no-recurse-submodules --depth=2 origin +refs/heads/*:refs/remotes/origin/* +refs/pull/63470/merge:refs/remotes/pull/63470/merge
---
2019-08-13T21:06:06.3462857Z do so (now or later) by using -b with the checkout command again. Example:
2019-08-13T21:06:06.3462885Z 
2019-08-13T21:06:06.3463225Z   git checkout -b <new-branch-name>
2019-08-13T21:06:06.3463251Z 
2019-08-13T21:06:06.3463317Z HEAD is now at 13221cb90 Merge 417f9ea90cf9bcccd8a1fa569a11a4fc071e3b8c into 60960a260f7b5c695fd0717311d72ce62dd4eb43
2019-08-13T21:06:06.3629640Z ##[section]Starting: Collect CPU-usage statistics in the background
2019-08-13T21:06:06.3632375Z ==============================================================================
2019-08-13T21:06:06.3632642Z Task         : Bash
2019-08-13T21:06:06.3632703Z Description  : Run a Bash script on macOS, Linux, or Windows

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

@Mark-Simulacrum

This comment has been minimized.

Copy link
Member Author

commented Aug 13, 2019

(This PR has dropped the necessary bootstrap update, by splitting it into #63534).

bors added a commit that referenced this pull request Aug 15, 2019

Auto merge of #63470 - Mark-Simulacrum:rustc-depdep, r=alexcrichton
Utilize -Zbinary-dep-depinfo in rustbuild

The last commit moves us over to using binary-dep-depinfo while the first two permit us to bootstrap from what will become future beta, to be released in the next week (it's the `cfg(bootstrap)` processing).

We no longer utilize stamp-file mtimes at all inside rustbuild, and a future PR may be able to entirely eliminate them by eagerly copying to the appropriate sysroot. The only mtime-based dependency tracking left is for documentation because we lie to Cargo about the rustdoc binary, so Cargo does not track changes to the real binary, and codegen-backends because binary-dep-depinfo does not emit that information into the depfiles.

Both of these are fixable in the longer term but this existing patch gives us the following benefits:
 * We no longer delete Cargo target directories manually within a stage. Cross-stage, changes to codegen backends will still clear out target directories. This means that incremental state persists across individual steps (e.g., rebuilding libstd does not clear out librustc incremental state). Fixes #54712.
 * Dependency tracking across steps within a given stage is now fully precise. We will not clear out all codegen backend dependencies due to changes in librustc_driver, for example, only deleting the final librustc_codegen_llvm crate. Fixes #54008, fixes #50481.
 * We properly track codegen backends as a dependency (equivalent to rustc) across changes. Fixes #53284, and fixes #52719.
 * Cross-stage dependency tracking of crates is also much more accurate and reliable. Most likely fixes #49979 (but no reproduction steps in that issue). Fixes #59105.

cc #63012
@rust-highfive

This comment has been minimized.

Copy link
Collaborator

commented Aug 15, 2019

Your PR failed (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
2019-08-15T22:27:48.4433130Z [command]/bin/bash --noprofile --norc /Users/vsts/agent/2.155.1/work/_temp/1ca1bbff-70cf-4f29-82c4-27dc5c23f5c9.sh
2019-08-15T22:27:48.4912950Z   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
2019-08-15T22:27:48.4913640Z                                  Dload  Upload   Total   Spent    Left  Speed
2019-08-15T22:27:48.4913910Z 
2019-08-15T22:27:48.4927440Z   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (6) Could not resolve host: rust-lang-ci-mirrors.s3-us-west-1.amazonaws.com
2019-08-15T22:27:48.5107870Z ##[error]Bash exited with code '6'.
2019-08-15T22:27:48.5386740Z ##[section]Starting: Upload CPU usage statistics
2019-08-15T22:27:48.5391590Z ==============================================================================
2019-08-15T22:27:48.5391690Z Task         : Bash
2019-08-15T22:27:48.5391790Z Description  : Run a Bash script on macOS, Linux, or Windows
---
2019-08-15T22:27:48.7372740Z Script contents:
2019-08-15T22:27:48.7374240Z aws s3 cp --acl public-read cpu-usage.csv s3://$DEPLOY_BUCKET/rustc-builds/$BUILD_SOURCEVERSION/cpu-$SYSTEM_JOBNAME.csv
2019-08-15T22:27:48.7402300Z ========================== Starting Command Output ===========================
2019-08-15T22:27:48.7428490Z [command]/bin/bash --noprofile --norc /Users/vsts/agent/2.155.1/work/_temp/d63caf75-f264-4e34-9cbc-80c2be3d782d.sh
2019-08-15T22:27:48.7602840Z /Users/vsts/agent/2.155.1/work/_temp/d63caf75-f264-4e34-9cbc-80c2be3d782d.sh: line 1: aws: command not found
2019-08-15T22:27:48.7690540Z ##[error]Bash exited with code '127'.
2019-08-15T22:27:48.7735790Z ##[section]Starting: Checkout
2019-08-15T22:27:48.7738930Z ==============================================================================
2019-08-15T22:27:48.7739140Z Task         : Get sources
2019-08-15T22:27:48.7739250Z Description  : Get sources from a repository. Supports Git, TfsVC, and SVN repositories.

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

@bors

This comment has been minimized.

Copy link
Contributor

commented Aug 15, 2019

💔 Test failed - checks-azure

@Centril

This comment has been minimized.

Copy link
Member

commented Aug 15, 2019

@bors retry spurious

@bors

This comment has been minimized.

Copy link
Contributor

commented Aug 16, 2019

⌛️ Testing commit 417f9ea with merge 445fdfd...

bors added a commit that referenced this pull request Aug 16, 2019

Auto merge of #63470 - Mark-Simulacrum:rustc-depdep, r=alexcrichton
Utilize -Zbinary-dep-depinfo in rustbuild

The last commit moves us over to using binary-dep-depinfo while the first two permit us to bootstrap from what will become future beta, to be released in the next week (it's the `cfg(bootstrap)` processing).

We no longer utilize stamp-file mtimes at all inside rustbuild, and a future PR may be able to entirely eliminate them by eagerly copying to the appropriate sysroot. The only mtime-based dependency tracking left is for documentation because we lie to Cargo about the rustdoc binary, so Cargo does not track changes to the real binary, and codegen-backends because binary-dep-depinfo does not emit that information into the depfiles.

Both of these are fixable in the longer term but this existing patch gives us the following benefits:
 * We no longer delete Cargo target directories manually within a stage. Cross-stage, changes to codegen backends will still clear out target directories. This means that incremental state persists across individual steps (e.g., rebuilding libstd does not clear out librustc incremental state). Fixes #54712.
 * Dependency tracking across steps within a given stage is now fully precise. We will not clear out all codegen backend dependencies due to changes in librustc_driver, for example, only deleting the final librustc_codegen_llvm crate. Fixes #54008, fixes #50481.
 * We properly track codegen backends as a dependency (equivalent to rustc) across changes. Fixes #53284, and fixes #52719.
 * Cross-stage dependency tracking of crates is also much more accurate and reliable. Most likely fixes #49979 (but no reproduction steps in that issue). Fixes #59105.

cc #63012
@bors

This comment has been minimized.

Copy link
Contributor

commented Aug 16, 2019

💥 Test timed out

@Mark-Simulacrum

This comment has been minimized.

Copy link
Member Author

commented Aug 16, 2019

@bors retry

Pretty sure literally every builder passed and homu just didn't get the notification (cc @pietroalbini - does that look similar to #63513?)

@pietroalbini

This comment has been minimized.

Copy link
Member

commented Aug 16, 2019

No, GitHub was correctly updated. Let me see if there are any errors on homu's side.

@pietroalbini

This comment has been minimized.

Copy link
Member

commented Aug 16, 2019

Oh this is a legit timeout: the build took 4 hours and 6 seconds, which is more than the 4 hour timeout...

@Mark-Simulacrum

This comment has been minimized.

Copy link
Member Author

commented Aug 16, 2019

Longest builder took just under 4 hours (https://dev.azure.com/rust-lang/rust/_build/results?buildId=6037&view=logs) AFAICT, and it looks like that builder is taking 3h45m or so in other auto builds so presumably this is not unexpected. We should try and move some things around as the partner i686-mingw-1 builder is only taking 3 hours (so we have an hour or so to split across two builders, in theory).

@pietroalbini

This comment has been minimized.

Copy link
Member

commented Aug 16, 2019

Longest builder took just under 4 hours

Yeah, but in the summary the recorded build time was 4 hours and 6 seconds, which is when the final check suite was updated.

We should try and move some things around as the partner i686-mingw-1 builder is only taking 3 hours (so we have an hour or so to split across two builders, in theory).

I'll look into this.

@RalfJung

This comment has been minimized.

Copy link
Member

commented Aug 16, 2019

Some of the text in the OP still sounds like it comes from an alternative universe where this landed before the beta cutoff?

@Mark-Simulacrum

This comment has been minimized.

Copy link
Member Author

commented Aug 16, 2019

Thanks for the reminder, I had forgotten to remove that text. Should be all down to just this PR now.

@bors

This comment has been minimized.

Copy link
Contributor

commented Aug 16, 2019

⌛️ Testing commit 417f9ea with merge 9a32ad0...

bors added a commit that referenced this pull request Aug 16, 2019

Auto merge of #63470 - Mark-Simulacrum:rustc-depdep, r=alexcrichton
Utilize -Zbinary-dep-depinfo in rustbuild

We no longer utilize stamp-file mtimes at all inside rustbuild, and a future PR may be able to entirely eliminate them by eagerly copying to the appropriate sysroot. The only mtime-based dependency tracking left is for documentation because we lie to Cargo about the rustdoc binary, so Cargo does not track changes to the real binary, and codegen-backends because binary-dep-depinfo does not emit that information into the depfiles.

Both of these are fixable in the longer term but this existing patch gives us the following benefits:
 * We no longer delete Cargo target directories manually within a stage. Cross-stage, changes to codegen backends will still clear out target directories. This means that incremental state persists across individual steps (e.g., rebuilding libstd does not clear out librustc incremental state). Fixes #54712.
 * Dependency tracking across steps within a given stage is now fully precise. We will not clear out all codegen backend dependencies due to changes in librustc_driver, for example, only deleting the final librustc_codegen_llvm crate. Fixes #54008, fixes #50481.
 * We properly track codegen backends as a dependency (equivalent to rustc) across changes. Fixes #53284, and fixes #52719.
 * Cross-stage dependency tracking of crates is also much more accurate and reliable. Most likely fixes #49979 (but no reproduction steps in that issue). Fixes #59105.

cc #63012
@bors

This comment has been minimized.

Copy link
Contributor

commented Aug 16, 2019

☀️ Test successful - checks-azure
Approved by: alexcrichton
Pushing 9a32ad0 to master...

@bors bors added the merged-by-bors label Aug 16, 2019

@bors bors merged commit 417f9ea into rust-lang:master Aug 16, 2019

5 checks passed

homu Test successful
Details
pr Build #20190813.58 succeeded
Details
pr (Linux mingw-check) Linux mingw-check succeeded
Details
pr (Linux x86_64-gnu-llvm-6.0) Linux x86_64-gnu-llvm-6.0 succeeded
Details
pr (LinuxTools) LinuxTools succeeded
Details
@LukasKalbertodt

This comment has been minimized.

Copy link
Contributor

commented Aug 16, 2019

Thank you a ton!

PRs like these are huge but are not the ones getting all the attention, for obvious reasons. This PR will make the lives of almost all people working on this repo a lot better! While the fixed issues sound like minor inconveniences, they made working on the compiler very annoying at times. I almost gave up on compiler-hacking twice just for annoyances like these! Reducing frustrations with the build-system goes a long way in attracting and keeping more people hacking on this repo.

Please excuse my ramblings, but I just had to express my gratitude for PRs like this. Thanks again!

@Mark-Simulacrum Mark-Simulacrum deleted the Mark-Simulacrum:rustc-depdep branch Aug 16, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
8 participants
You can’t perform that action at this time.