Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Refactor dep graph representation both in memory and on disk #60035

Open
wants to merge 17 commits into
base: master
from

Conversation

@Zoxc
Copy link
Contributor

commented Apr 17, 2019

This PR changes how we save the dep graph. Instead of storing the nodes in memory until the end of the compilation they are streamed to a file (in the background with parallel_compiler). This should hopefully reduce the memory usage with incremental compilation somewhat.

I've yet to add code to read the dep graph back into memory for dep graph debugging functionality.

I also want to get rid of CurrentDepGraph::node_to_node_index, but I'm not yet sure what the best approach is for that.

r? @michaelwoerister

let mut hasher = StableHasher::new();

for &read in task_deps.reads.iter() {
let read_dep_node = self.data[read].node;
task_deps.reads.hash(&mut hasher);

This comment has been minimized.

Copy link
@Zoxc

Zoxc Apr 17, 2019

Author Contributor

This PR changes the hashing of anon tasks to just hash the DepNodeIndexs for dependencies instead of the full DepNodes. This should be fine since the hash is supposed to be unique per session.

@rust-highfive

This comment has been minimized.

Copy link
Collaborator

commented Apr 17, 2019

The job x86_64-gnu-llvm-6.0 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
travis_time:end:09f958b8:start=1555494408314667354,finish=1555494511775175450,duration=103460508096
$ git checkout -qf FETCH_HEAD
travis_fold:end:git.checkout

Encrypted environment variables have been removed for security reasons.
See https://docs.travis-ci.com/user/pull-requests/#pull-requests-and-security-restrictions
$ export SCCACHE_BUCKET=rust-lang-ci-sccache2
$ export SCCACHE_REGION=us-west-1
$ export GCP_CACHE_BUCKET=rust-lang-ci-cache
$ export AWS_ACCESS_KEY_ID=AKIA46X5W6CZEJZ6XT55
---
[00:06:26]    Compiling rustc_macros v0.1.0 (/checkout/src/librustc_macros)
[00:06:32]    Compiling syntax_pos v0.0.0 (/checkout/src/libsyntax_pos)
[00:06:36]    Compiling rustc_errors v0.0.0 (/checkout/src/librustc_errors)
[00:07:36]    Compiling syntax_ext v0.0.0 (/checkout/src/libsyntax_ext)
[00:07:51] error[E0609]: no field `data` on type `std::cell::RefMut<'_, dep_graph::graph::CurrentDepGraph>`
[00:07:51]     --> src/librustc/dep_graph/graph.rs:1068:52
[00:07:51]      |
[00:07:51] 1068 |                                 let source = graph.data[source].node;
[00:07:51] 
[00:08:09] error: aborting due to previous error
[00:08:09] 
[00:08:09] For more information about this error, try `rustc --explain E0609`.
---
travis_time:end:06637445:start=1555495011374053955,finish=1555495011378619543,duration=4565588
travis_fold:end:after_failure.3
travis_fold:start:after_failure.4
travis_time:start:11f6f3d0
$ ln -s . checkout && for CORE in obj/cores/core.*; do EXE=$(echo $CORE | sed 's|obj/cores/core\.[0-9]*\.!checkout!\(.*\)|\1|;y|!|/|'); if [ -f "$EXE" ]; then printf travis_fold":start:crashlog\n\033[31;1m%s\033[0m\n" "$CORE"; gdb --batch -q -c "$CORE" "$EXE" -iex 'set auto-load off' -iex 'dir src/' -iex 'set sysroot .' -ex bt -ex q; echo travis_fold":"end:crashlog; fi; done || true
travis_fold:end:after_failure.4
travis_fold:start:after_failure.5
travis_time:start:1376641e
travis_time:start:1376641e
$ cat ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers || true
cat: ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers: No such file or directory
travis_fold:end:after_failure.5
travis_fold:start:after_failure.6
travis_time:start:2fb08f06
$ dmesg | grep -i kill

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

@Zoxc Zoxc force-pushed the Zoxc:dep-stream branch from eb03e3c to 6c3cfa5 Apr 17, 2019

@rust-highfive

This comment has been minimized.

Copy link
Collaborator

commented Apr 17, 2019

The job x86_64-gnu-llvm-6.0 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
travis_time:end:013c9bb2:start=1555501807089893653,finish=1555501923266532001,duration=116176638348
$ git checkout -qf FETCH_HEAD
travis_fold:end:git.checkout

Encrypted environment variables have been removed for security reasons.
See https://docs.travis-ci.com/user/pull-requests/#pull-requests-and-security-restrictions
$ export SCCACHE_BUCKET=rust-lang-ci-sccache2
$ export SCCACHE_REGION=us-west-1
$ export GCP_CACHE_BUCKET=rust-lang-ci-cache
$ export AWS_ACCESS_KEY_ID=AKIA46X5W6CZEJZ6XT55
---
[01:03:48] ................................................i................................................... 600/5541
[01:03:52] .................................................................................................... 700/5541
[01:03:57] .................................................................................................... 800/5541
[01:04:01] .................................................................................................... 900/5541
[01:04:06] ...........i...............i......FFFFFFFF.......................................................... 1000/5541
[01:04:09] ............................................iiiii................................................... 1100/5541
[01:04:15] .................................................................................................... 1300/5541
[01:04:17] .................................................................................................... 1400/5541
[01:04:21] .................................................................................................... 1500/5541
[01:04:23] .................................................................................................... 1600/5541
---
[01:06:51] failures:
[01:06:51] 
[01:06:51] ---- [ui] ui/dep-graph/dep-graph-assoc-type-codegen.rs stdout ----
[01:06:51] 
[01:06:51] error: Error: expected failure status (Some(1)) but received status Some(101).
[01:06:51] status: exit code: 101
[01:06:51] command: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "/checkout/src/test/ui/dep-graph/dep-graph-assoc-type-codegen.rs" "-Zthreads=1" "--target=x86_64-unknown-linux-gnu" "--error-format" "json" "-Zui-testing" "-C" "prefer-dynamic" "-o" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/dep-graph/dep-graph-assoc-type-codegen/a" "-Crpath" "-O" "-Zunstable-options" "-Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "-Z" "query-dep-graph" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/dep-graph/dep-graph-assoc-type-codegen/auxiliary" "-A" "unused"
[01:06:51] ------------------------------------------
[01:06:51] 
[01:06:51] ------------------------------------------
[01:06:51] stderr:
[01:06:51] stderr:
[01:06:51] ------------------------------------------
[01:06:51] thread 'rustc' panicked at 'src/librustc/session/mod.rs:811: Trying to get session directory from IncrCompSession `NotInitialized`', src/librustc/util/bug.rs:37:26
[01:06:51] 
[01:06:51] error: internal compiler error: unexpected panic
[01:06:51] 
[01:06:51] note: the compiler unexpectedly panicked. this is a bug.
[01:06:51] note: the compiler unexpectedly panicked. this is a bug.
[01:06:51] 
[01:06:51] note: we would appreciate a bug report: https://github.com/rust-lang/rust/blob/master/CONTRIBUTING.md#bug-reports
[01:06:51] 
[01:06:51] note: rustc 1.36.0-dev running on x86_64-unknown-linux-gnu
[01:06:51] 
[01:06:51] note: compiler flags: -Z threads=1 -Z ui-testing -Z unstable-options -Z query-dep-graph -C prefer-dynamic -C rpath
[01:06:51] 
[01:06:51] ------------------------------------------
[01:06:51] 
[01:06:51] 
[01:06:51] 
[01:06:51] ---- [ui] ui/dep-graph/dep-graph-caller-callee.rs stdout ----
[01:06:51] 
[01:06:51] error: Error: expected failure status (Some(1)) but received status Some(101).
[01:06:51] status: exit code: 101
[01:06:51] command: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "/checkout/src/test/ui/dep-graph/dep-graph-caller-callee.rs" "-Zthreads=1" "--target=x86_64-unknown-linux-gnu" "--error-format" "json" "-Zui-testing" "-C" "prefer-dynamic" "-o" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/dep-graph/dep-graph-caller-callee/a" "-Crpath" "-O" "-Zunstable-options" "-Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "-Z" "query-dep-graph" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/dep-graph/dep-graph-caller-callee/auxiliary" "-A" "unused"
[01:06:51] ------------------------------------------
[01:06:51] 
[01:06:51] ------------------------------------------
[01:06:51] stderr:
[01:06:51] stderr:
[01:06:51] ------------------------------------------
[01:06:51] thread 'rustc' panicked at 'src/librustc/session/mod.rs:811: Trying to get session directory from IncrCompSession `NotInitialized`', src/librustc/util/bug.rs:37:26
[01:06:51] 
[01:06:51] error: internal compiler error: unexpected panic
[01:06:51] 
[01:06:51] note: the compiler unexpectedly panicked. this is a bug.
[01:06:51] note: the compiler unexpectedly panicked. this is a bug.
[01:06:51] 
[01:06:51] note: we would appreciate a bug report: https://github.com/rust-lang/rust/blob/master/CONTRIBUTING.md#bug-reports
[01:06:51] 
[01:06:51] note: rustc 1.36.0-dev running on x86_64-unknown-linux-gnu
[01:06:51] 
[01:06:51] note: compiler flags: -Z threads=1 -Z ui-testing -Z unstable-options -Z query-dep-graph -C prefer-dynamic -C rpath
[01:06:51] 
[01:06:51] ------------------------------------------
[01:06:51] 
[01:06:51] 
[01:06:51] 
[01:06:51] ---- [ui] ui/dep-graph/dep-graph-struct-signature.rs stdout ----
[01:06:51] 
[01:06:51] error: Error: expected failure status (Some(1)) but received status Some(101).
[01:06:51] status: exit code: 101
[01:06:51] command: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "/checkout/src/test/ui/dep-graph/dep-graph-struct-signature.rs" "-Zthreads=1" "--target=x86_64-unknown-linux-gnu" "--error-format" "json" "-Zui-testing" "-C" "prefer-dynamic" "-o" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/dep-graph/dep-graph-struct-signature/a" "-Crpath" "-O" "-Zunstable-options" "-Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "-Z" "query-dep-graph" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/dep-graph/dep-graph-struct-signature/auxiliary" "-A" "unused"
[01:06:51] ------------------------------------------
[01:06:51] 
[01:06:51] ------------------------------------------
[01:06:51] stderr:
[01:06:51] stderr:
[01:06:51] ------------------------------------------
[01:06:51] thread 'rustc' panicked at 'src/librustc/session/mod.rs:811: Trying to get session directory from IncrCompSession `NotInitialized`', src/librustc/util/bug.rs:37:26
[01:06:51] 
[01:06:51] error: internal compiler error: unexpected panic
[01:06:51] 
[01:06:51] note: the compiler unexpectedly panicked. this is a bug.
[01:06:51] note: the compiler unexpectedly panicked. this is a bug.
[01:06:51] 
[01:06:51] note: we would appreciate a bug report: https://github.com/rust-lang/rust/blob/master/CONTRIBUTING.md#bug-reports
[01:06:51] 
[01:06:51] note: rustc 1.36.0-dev running on x86_64-unknown-linux-gnu
[01:06:51] 
[01:06:51] note: compiler flags: -Z threads=1 -Z ui-testing -Z unstable-options -Z query-dep-graph -C prefer-dynamic -C rpath
[01:06:51] 
[01:06:51] ------------------------------------------
[01:06:51] 
[01:06:51] 
[01:06:51] 
[01:06:51] ---- [ui] ui/dep-graph/dep-graph-trait-impl-two-traits-same-method.rs stdout ----
[01:06:51] 
[01:06:51] error: Error: expected failure status (Some(1)) but received status Some(101).
[01:06:51] status: exit code: 101
[01:06:51] command: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "/checkout/src/test/ui/dep-graph/dep-graph-trait-impl-two-traits-same-method.rs" "-Zthreads=1" "--target=x86_64-unknown-linux-gnu" "--error-format" "json" "-Zui-testing" "-C" "prefer-dynamic" "-o" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/dep-graph/dep-graph-trait-impl-two-traits-same-method/a" "-Crpath" "-O" "-Zunstable-options" "-Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "-Z" "query-dep-graph" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/dep-graph/dep-graph-trait-impl-two-traits-same-method/auxiliary" "-A" "unused"
[01:06:51] ------------------------------------------
[01:06:51] 
[01:06:51] ------------------------------------------
[01:06:51] stderr:
[01:06:51] stderr:
[01:06:51] ------------------------------------------
[01:06:51] thread 'rustc' panicked at 'src/librustc/session/mod.rs:811: Trying to get session directory from IncrCompSession `NotInitialized`', src/librustc/util/bug.rs:37:26
[01:06:51] 
[01:06:51] error: internal compiler error: unexpected panic
[01:06:51] 
[01:06:51] note: the compiler unexpectedly panicked. this is a bug.
[01:06:51] note: the compiler unexpectedly panicked. this is a bug.
[01:06:51] 
[01:06:51] note: we would appreciate a bug report: https://github.com/rust-lang/rust/blob/master/CONTRIBUTING.md#bug-reports
[01:06:51] 
[01:06:51] note: rustc 1.36.0-dev running on x86_64-unknown-linux-gnu
[01:06:51] 
[01:06:51] note: compiler flags: -Z threads=1 -Z ui-testing -Z unstable-options -Z query-dep-graph -C prefer-dynamic -C rpath
[01:06:51] 
[01:06:51] ------------------------------------------
[01:06:51] 
[01:06:51] 
[01:06:51] 
[01:06:51] ---- [ui] ui/dep-graph/dep-graph-trait-impl-two-traits.rs stdout ----
[01:06:51] 
[01:06:51] error: Error: expected failure status (Some(1)) but received status Some(101).
[01:06:51] status: exit code: 101
[01:06:51] command: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "/checkout/src/test/ui/dep-graph/dep-graph-trait-impl-two-traits.rs" "-Zthreads=1" "--target=x86_64-unknown-linux-gnu" "--error-format" "json" "-Zui-testing" "-C" "prefer-dynamic" "-o" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/dep-graph/dep-graph-trait-impl-two-traits/a" "-Crpath" "-O" "-Zunstable-options" "-Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "-Z" "query-dep-graph" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/dep-graph/dep-graph-trait-impl-two-traits/auxiliary" "-A" "unused"
[01:06:51] ------------------------------------------
[01:06:51] 
[01:06:51] ------------------------------------------
[01:06:51] stderr:
[01:06:51] stderr:
[01:06:51] ------------------------------------------
[01:06:51] thread 'rustc' panicked at 'src/librustc/session/mod.rs:811: Trying to get session directory from IncrCompSession `NotInitialized`', src/librustc/util/bug.rs:37:26
[01:06:51] 
[01:06:51] error: internal compiler error: unexpected panic
[01:06:51] 
[01:06:51] note: the compiler unexpectedly panicked. this is a bug.
[01:06:51] note: the compiler unexpectedly panicked. this is a bug.
[01:06:51] 
[01:06:51] note: we would appreciate a bug report: https://github.com/rust-lang/rust/blob/master/CONTRIBUTING.md#bug-reports
[01:06:51] 
[01:06:51] note: rustc 1.36.0-dev running on x86_64-unknown-linux-gnu
[01:06:51] 
[01:06:51] note: compiler flags: -Z threads=1 -Z ui-testing -Z unstable-options -Z query-dep-graph -C prefer-dynamic -C rpath
[01:06:51] 
[01:06:51] ------------------------------------------
[01:06:51] 
[01:06:51] 
[01:06:51] 
[01:06:51] ---- [ui] ui/dep-graph/dep-graph-trait-impl.rs stdout ----
[01:06:51] 
[01:06:51] error: Error: expected failure status (Some(1)) but received status Some(101).
[01:06:51] status: exit code: 101
[01:06:51] command: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "/checkout/src/test/ui/dep-graph/dep-graph-trait-impl.rs" "-Zthreads=1" "--target=x86_64-unknown-linux-gnu" "--error-format" "json" "-Zui-testing" "-C" "prefer-dynamic" "-o" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/dep-graph/dep-graph-trait-impl/a" "-Crpath" "-O" "-Zunstable-options" "-Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "-Z" "query-dep-graph" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/dep-graph/dep-graph-trait-impl/auxiliary" "-A" "unused"
[01:06:51] ------------------------------------------
[01:06:51] 
[01:06:51] ------------------------------------------
[01:06:51] stderr:
[01:06:51] stderr:
[01:06:51] ------------------------------------------
[01:06:51] thread 'rustc' panicked at 'src/librustc/session/mod.rs:811: Trying to get session directory from IncrCompSession `NotInitialized`', src/librustc/util/bug.rs:37:26
[01:06:51] 
[01:06:51] error: internal compiler error: unexpected panic
[01:06:51] 
[01:06:51] note: the compiler unexpectedly panicked. this is a bug.
[01:06:51] note: the compiler unexpectedly panicked. this is a bug.
[01:06:51] 
[01:06:51] note: we would appreciate a bug report: https://github.com/rust-lang/rust/blob/master/CONTRIBUTING.md#bug-reports
[01:06:51] 
[01:06:51] note: rustc 1.36.0-dev running on x86_64-unknown-linux-gnu
[01:06:51] 
[01:06:51] note: compiler flags: -Z threads=1 -Z ui-testing -Z unstable-options -Z query-dep-graph -C prefer-dynamic -C rpath
[01:06:51] 
[01:06:51] ------------------------------------------
[01:06:51] 
[01:06:51] 
[01:06:51] 
[01:06:51] ---- [ui] ui/dep-graph/dep-graph-type-alias.rs stdout ----
[01:06:51] 
[01:06:51] error: Error: expected failure status (Some(1)) but received status Some(101).
[01:06:51] status: exit code: 101
[01:06:51] command: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "/checkout/src/test/ui/dep-graph/dep-graph-type-alias.rs" "-Zthreads=1" "--target=x86_64-unknown-linux-gnu" "--error-format" "json" "-Zui-testing" "-C" "prefer-dynamic" "-o" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/dep-graph/dep-graph-type-alias/a" "-Crpath" "-O" "-Zunstable-options" "-Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "-Z" "query-dep-graph" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/dep-graph/dep-graph-type-alias/auxiliary" "-A" "unused"
[01:06:51] ------------------------------------------
[01:06:51] 
[01:06:51] ------------------------------------------
[01:06:51] stderr:
[01:06:51] stderr:
[01:06:51] ------------------------------------------
[01:06:51] thread 'rustc' panicked at 'src/librustc/session/mod.rs:811: Trying to get session directory from IncrCompSession `NotInitialized`', src/librustc/util/bug.rs:37:26
[01:06:51] 
[01:06:51] error: internal compiler error: unexpected panic
[01:06:51] 
[01:06:51] note: the compiler unexpectedly panicked. this is a bug.
[01:06:51] note: the compiler unexpectedly panicked. this is a bug.
[01:06:51] 
[01:06:51] note: we would appreciate a bug report: https://github.com/rust-lang/rust/blob/master/CONTRIBUTING.md#bug-reports
[01:06:51] 
[01:06:51] note: rustc 1.36.0-dev running on x86_64-unknown-linux-gnu
[01:06:51] 
[01:06:51] note: compiler flags: -Z threads=1 -Z ui-testing -Z unstable-options -Z query-dep-graph -C prefer-dynamic -C rpath
[01:06:51] 
[01:06:51] ------------------------------------------
[01:06:51] 
[01:06:51] 
[01:06:51] 
[01:06:51] ---- [ui] ui/dep-graph/dep-graph-variance-alias.rs stdout ----
[01:06:51] 
[01:06:51] error: Error: expected failure status (Some(1)) but received status Some(101).
[01:06:51] status: exit code: 101
[01:06:51] command: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "/checkout/src/test/ui/dep-graph/dep-graph-variance-alias.rs" "-Zthreads=1" "--target=x86_64-unknown-linux-gnu" "--error-format" "json" "-Zui-testing" "-C" "prefer-dynamic" "-o" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/dep-graph/dep-graph-variance-alias/a" "-Crpath" "-O" "-Zunstable-options" "-Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "-Z" "query-dep-graph" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/dep-graph/dep-graph-variance-alias/auxiliary" "-A" "unused"
[01:06:51] ------------------------------------------
[01:06:51] 
[01:06:51] ------------------------------------------
[01:06:51] stderr:
[01:06:51] stderr:
[01:06:51] ------------------------------------------
[01:06:51] thread 'rustc' panicked at 'src/librustc/session/mod.rs:811: Trying to get session directory from IncrCompSession `NotInitialized`', src/librustc/util/bug.rs:37:26
[01:06:51] 
[01:06:51] error: internal compiler error: unexpected panic
[01:06:51] 
[01:06:51] note: the compiler unexpectedly panicked. this is a bug.
[01:06:51] note: the compiler unexpectedly panicked. this is a bug.
[01:06:51] 
[01:06:51] note: we would appreciate a bug report: https://github.com/rust-lang/rust/blob/master/CONTRIBUTING.md#bug-reports
[01:06:51] 
[01:06:51] note: rustc 1.36.0-dev running on x86_64-unknown-linux-gnu
[01:06:51] 
[01:06:51] note: compiler flags: -Z threads=1 -Z ui-testing -Z unstable-options -Z query-dep-graph -C prefer-dynamic -C rpath
[01:06:51] 
[01:06:51] ------------------------------------------
[01:06:51] 
[01:06:51] 
---
[01:06:51] thread 'main' panicked at 'Some tests failed', src/tools/compiletest/src/main.rs:517:22
[01:06:51] note: Run with `RUST_BACKTRACE=1` environment variable to display a backtrace.
[01:06:51] 
[01:06:51] 
[01:06:51] command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools-bin/compiletest" "--compile-lib-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib" "--run-lib-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/lib" "--rustc-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "--src-base" "/checkout/src/test/ui" "--build-base" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui" "--stage-id" "stage2-x86_64-unknown-linux-gnu" "--mode" "ui" "--target" "x86_64-unknown-linux-gnu" "--host" "x86_64-unknown-linux-gnu" "--llvm-filecheck" "/usr/lib/llvm-6.0/bin/FileCheck" "--host-rustcflags" "-Crpath -O -Zunstable-options  -Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "--target-rustcflags" "-Crpath -O -Zunstable-options  -Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "--docck-python" "/usr/bin/python2.7" "--lldb-python" "/usr/bin/python2.7" "--gdb" "/usr/bin/gdb" "--quiet" "--llvm-version" "6.0.0\n" "--system-llvm" "--cc" "" "--cxx" "" "--cflags" "" "--llvm-components" "" "--llvm-cxxflags" "" "--adb-path" "adb" "--adb-test-dir" "/data/tmp/work" "--android-cross-path" "" "--color" "always"
[01:06:51] 
[01:06:51] 
[01:06:51] failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test
[01:06:51] Build completed unsuccessfully in 0:04:20
[01:06:51] Build completed unsuccessfully in 0:04:20
[01:06:51] make: *** [check] Error 1
[01:06:51] Makefile:48: recipe for target 'check' failed
The command "stamp sh -x -c "$RUN_SCRIPT"" exited with 2.
travis_time:start:0ca5a4ce
$ date && (curl -fs --head https://google.com | grep ^Date: | sed 's/Date: //g' || true)
Wed Apr 17 12:59:03 UTC 2019
---
travis_time:end:003c3fd8:start=1555505944599953281,finish=1555505944607674857,duration=7721576
travis_fold:end:after_failure.4
travis_fold:start:after_failure.5
travis_time:start:185801d9
$ cat ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers || true
cat: ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers: No such file or directory
travis_fold:end:after_failure.5
travis_fold:start:after_failure.6
travis_time:start:210b16cc
$ dmesg | grep -i kill

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

@Zoxc

This comment has been minimized.

Copy link
Contributor Author

commented Apr 17, 2019

@bors try

@bors

This comment has been minimized.

Copy link
Contributor

commented Apr 17, 2019

⌛️ Trying commit 6c3cfa5 with merge 7c3f54e...

bors added a commit that referenced this pull request Apr 17, 2019

Auto merge of #60035 - Zoxc:dep-stream, r=<try>
[WIP] Serialize dep nodes in the background

This PR changes how we save the dep graph. Instead of storing the nodes in memory until the end of the compilation they are streamed to a file (in the background with `parallel_compiler`). This should hopefully reduce the memory usage with incremental compilation somewhat.

I've yet to add code to read the dep graph back into memory for dep graph debugging functionality.

I also want to get rid of `CurrentDepGraph::node_to_node_index`, but I'm not yet sure what the best approach is for that.

r? @michaelwoerister
@bors

This comment has been minimized.

Copy link
Contributor

commented Apr 17, 2019

☀️ Try build successful - checks-travis
Build commit: 7c3f54e

@Zoxc

This comment has been minimized.

Copy link
Contributor Author

commented Apr 17, 2019

@rust-timer

This comment has been minimized.

Copy link

commented Apr 17, 2019

Success: Queued 7c3f54e with parent efe2f32, comparison URL.

@rust-timer

This comment has been minimized.

Copy link

commented Apr 18, 2019

Finished benchmarking try commit 7c3f54e

@Zoxc Zoxc force-pushed the Zoxc:dep-stream branch from 6c3cfa5 to d2bb406 Apr 20, 2019

@Zoxc

This comment has been minimized.

Copy link
Contributor Author

commented Apr 20, 2019

@bors try

@bors

This comment has been minimized.

Copy link
Contributor

commented Apr 20, 2019

⌛️ Trying commit d2bb406 with merge d8c21d8...

bors added a commit that referenced this pull request Apr 20, 2019

Auto merge of #60035 - Zoxc:dep-stream, r=<try>
[WIP] Serialize dep nodes in the background

This PR changes how we save the dep graph. Instead of storing the nodes in memory until the end of the compilation they are streamed to a file (in the background with `parallel_compiler`). This should hopefully reduce the memory usage with incremental compilation somewhat.

I've yet to add code to read the dep graph back into memory for dep graph debugging functionality.

I also want to get rid of `CurrentDepGraph::node_to_node_index`, but I'm not yet sure what the best approach is for that.

r? @michaelwoerister
@rust-highfive

This comment has been minimized.

Copy link
Collaborator

commented Apr 20, 2019

The job x86_64-gnu-llvm-6.0 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
travis_time:end:05974380:start=1555728581885664227,finish=1555728583909455504,duration=2023791277
$ git checkout -qf FETCH_HEAD
travis_fold:end:git.checkout

Encrypted environment variables have been removed for security reasons.
See https://docs.travis-ci.com/user/pull-requests/#pull-requests-and-security-restrictions
$ export SCCACHE_BUCKET=rust-lang-ci-sccache2
$ export SCCACHE_REGION=us-west-1
$ export GCP_CACHE_BUCKET=rust-lang-ci-cache
$ export AWS_ACCESS_KEY_ID=AKIA46X5W6CZEJZ6XT55
---
[01:03:39] ................................................i................................................... 600/5547
[01:03:43] .................................................................................................... 700/5547
[01:03:47] .................................................................................................... 800/5547
[01:03:51] .................................................................................................... 900/5547
[01:03:56] ............i...............i......FFFFFFFF......................................................... 1000/5547
[01:04:00] .............................................iiiii.................................................. 1100/5547
[01:04:06] .................................................................................................... 1300/5547
[01:04:08] .................................................................................................... 1400/5547
[01:04:12] .................................................................................................... 1500/5547
[01:04:14] .................................................................................................... 1600/5547
---
[01:06:45] failures:
[01:06:45] 
[01:06:45] ---- [ui] ui/dep-graph/dep-graph-assoc-type-codegen.rs stdout ----
[01:06:45] 
[01:06:45] error: Error: expected failure status (Some(1)) but received status Some(101).
[01:06:45] status: exit code: 101
[01:06:45] command: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "/checkout/src/test/ui/dep-graph/dep-graph-assoc-type-codegen.rs" "-Zthreads=1" "--target=x86_64-unknown-linux-gnu" "--error-format" "json" "-Zui-testing" "-C" "prefer-dynamic" "-o" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/dep-graph/dep-graph-assoc-type-codegen/a" "-Crpath" "-O" "-Zunstable-options" "-Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "-Z" "query-dep-graph" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/dep-graph/dep-graph-assoc-type-codegen/auxiliary" "-A" "unused"
[01:06:45] ------------------------------------------
[01:06:45] 
[01:06:45] ------------------------------------------
[01:06:45] stderr:
[01:06:45] stderr:
[01:06:45] ------------------------------------------
[01:06:45] thread 'rustc' panicked at 'src/librustc/session/mod.rs:815: Trying to get session directory from IncrCompSession `NotInitialized`', src/librustc/util/bug.rs:37:26
[01:06:45] 
[01:06:45] error: internal compiler error: unexpected panic
[01:06:45] 
[01:06:45] note: the compiler unexpectedly panicked. this is a bug.
[01:06:45] note: the compiler unexpectedly panicked. this is a bug.
[01:06:45] 
[01:06:45] note: we would appreciate a bug report: https://github.com/rust-lang/rust/blob/master/CONTRIBUTING.md#bug-reports
[01:06:45] 
[01:06:45] note: rustc 1.36.0-dev running on x86_64-unknown-linux-gnu
[01:06:45] 
[01:06:45] note: compiler flags: -Z threads=1 -Z ui-testing -Z unstable-options -Z query-dep-graph -C prefer-dynamic -C rpath
[01:06:45] 
[01:06:45] ------------------------------------------
[01:06:45] 
[01:06:45] 
[01:06:45] 
[01:06:45] ---- [ui] ui/dep-graph/dep-graph-caller-callee.rs stdout ----
[01:06:45] 
[01:06:45] error: Error: expected failure status (Some(1)) but received status Some(101).
[01:06:45] status: exit code: 101
[01:06:45] command: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "/checkout/src/test/ui/dep-graph/dep-graph-caller-callee.rs" "-Zthreads=1" "--target=x86_64-unknown-linux-gnu" "--error-format" "json" "-Zui-testing" "-C" "prefer-dynamic" "-o" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/dep-graph/dep-graph-caller-callee/a" "-Crpath" "-O" "-Zunstable-options" "-Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "-Z" "query-dep-graph" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/dep-graph/dep-graph-caller-callee/auxiliary" "-A" "unused"
[01:06:45] ------------------------------------------
[01:06:45] 
[01:06:45] ------------------------------------------
[01:06:45] stderr:
[01:06:45] stderr:
[01:06:45] ------------------------------------------
[01:06:45] thread 'rustc' panicked at 'src/librustc/session/mod.rs:815: Trying to get session directory from IncrCompSession `NotInitialized`', src/librustc/util/bug.rs:37:26
[01:06:45] 
[01:06:45] error: internal compiler error: unexpected panic
[01:06:45] 
[01:06:45] note: the compiler unexpectedly panicked. this is a bug.
[01:06:45] note: the compiler unexpectedly panicked. this is a bug.
[01:06:45] 
[01:06:45] note: we would appreciate a bug report: https://github.com/rust-lang/rust/blob/master/CONTRIBUTING.md#bug-reports
[01:06:45] 
[01:06:45] note: rustc 1.36.0-dev running on x86_64-unknown-linux-gnu
[01:06:45] 
[01:06:45] note: compiler flags: -Z threads=1 -Z ui-testing -Z unstable-options -Z query-dep-graph -C prefer-dynamic -C rpath
[01:06:45] 
[01:06:45] ------------------------------------------
[01:06:45] 
[01:06:45] 
[01:06:45] 
[01:06:45] ---- [ui] ui/dep-graph/dep-graph-struct-signature.rs stdout ----
[01:06:45] 
[01:06:45] error: Error: expected failure status (Some(1)) but received status Some(101).
[01:06:45] status: exit code: 101
[01:06:45] command: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "/checkout/src/test/ui/dep-graph/dep-graph-struct-signature.rs" "-Zthreads=1" "--target=x86_64-unknown-linux-gnu" "--error-format" "json" "-Zui-testing" "-C" "prefer-dynamic" "-o" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/dep-graph/dep-graph-struct-signature/a" "-Crpath" "-O" "-Zunstable-options" "-Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "-Z" "query-dep-graph" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/dep-graph/dep-graph-struct-signature/auxiliary" "-A" "unused"
[01:06:45] ------------------------------------------
[01:06:45] 
[01:06:45] ------------------------------------------
[01:06:45] stderr:
[01:06:45] stderr:
[01:06:45] ------------------------------------------
[01:06:45] thread 'rustc' panicked at 'src/librustc/session/mod.rs:815: Trying to get session directory from IncrCompSession `NotInitialized`', src/librustc/util/bug.rs:37:26
[01:06:45] 
[01:06:45] error: internal compiler error: unexpected panic
[01:06:45] 
[01:06:45] note: the compiler unexpectedly panicked. this is a bug.
[01:06:45] note: the compiler unexpectedly panicked. this is a bug.
[01:06:45] 
[01:06:45] note: we would appreciate a bug report: https://github.com/rust-lang/rust/blob/master/CONTRIBUTING.md#bug-reports
[01:06:45] 
[01:06:45] note: rustc 1.36.0-dev running on x86_64-unknown-linux-gnu
[01:06:45] 
[01:06:45] note: compiler flags: -Z threads=1 -Z ui-testing -Z unstable-options -Z query-dep-graph -C prefer-dynamic -C rpath
[01:06:45] 
[01:06:45] ------------------------------------------
[01:06:45] 
[01:06:45] 
[01:06:45] 
[01:06:45] ---- [ui] ui/dep-graph/dep-graph-trait-impl-two-traits-same-method.rs stdout ----
[01:06:45] 
[01:06:45] error: Error: expected failure status (Some(1)) but received status Some(101).
[01:06:45] status: exit code: 101
[01:06:45] command: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "/checkout/src/test/ui/dep-graph/dep-graph-trait-impl-two-traits-same-method.rs" "-Zthreads=1" "--target=x86_64-unknown-linux-gnu" "--error-format" "json" "-Zui-testing" "-C" "prefer-dynamic" "-o" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/dep-graph/dep-graph-trait-impl-two-traits-same-method/a" "-Crpath" "-O" "-Zunstable-options" "-Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "-Z" "query-dep-graph" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/dep-graph/dep-graph-trait-impl-two-traits-same-method/auxiliary" "-A" "unused"
[01:06:45] ------------------------------------------
[01:06:45] 
[01:06:45] ------------------------------------------
[01:06:45] stderr:
[01:06:45] stderr:
[01:06:45] ------------------------------------------
[01:06:45] thread 'rustc' panicked at 'src/librustc/session/mod.rs:815: Trying to get session directory from IncrCompSession `NotInitialized`', src/librustc/util/bug.rs:37:26
[01:06:45] 
[01:06:45] error: internal compiler error: unexpected panic
[01:06:45] 
[01:06:45] note: the compiler unexpectedly panicked. this is a bug.
[01:06:45] note: the compiler unexpectedly panicked. this is a bug.
[01:06:45] 
[01:06:45] note: we would appreciate a bug report: https://github.com/rust-lang/rust/blob/master/CONTRIBUTING.md#bug-reports
[01:06:45] 
[01:06:45] note: rustc 1.36.0-dev running on x86_64-unknown-linux-gnu
[01:06:45] 
[01:06:45] note: compiler flags: -Z threads=1 -Z ui-testing -Z unstable-options -Z query-dep-graph -C prefer-dynamic -C rpath
[01:06:45] 
[01:06:45] ------------------------------------------
[01:06:45] 
[01:06:45] 
[01:06:45] 
[01:06:45] ---- [ui] ui/dep-graph/dep-graph-trait-impl-two-traits.rs stdout ----
[01:06:45] 
[01:06:45] error: Error: expected failure status (Some(1)) but received status Some(101).
[01:06:45] status: exit code: 101
[01:06:45] command: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "/checkout/src/test/ui/dep-graph/dep-graph-trait-impl-two-traits.rs" "-Zthreads=1" "--target=x86_64-unknown-linux-gnu" "--error-format" "json" "-Zui-testing" "-C" "prefer-dynamic" "-o" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/dep-graph/dep-graph-trait-impl-two-traits/a" "-Crpath" "-O" "-Zunstable-options" "-Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "-Z" "query-dep-graph" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/dep-graph/dep-graph-trait-impl-two-traits/auxiliary" "-A" "unused"
[01:06:45] ------------------------------------------
[01:06:45] 
[01:06:45] ------------------------------------------
[01:06:45] stderr:
[01:06:45] stderr:
[01:06:45] ------------------------------------------
[01:06:45] thread 'rustc' panicked at 'src/librustc/session/mod.rs:815: Trying to get session directory from IncrCompSession `NotInitialized`', src/librustc/util/bug.rs:37:26
[01:06:45] 
[01:06:45] error: internal compiler error: unexpected panic
[01:06:45] 
[01:06:45] note: the compiler unexpectedly panicked. this is a bug.
[01:06:45] note: the compiler unexpectedly panicked. this is a bug.
[01:06:45] 
[01:06:45] note: we would appreciate a bug report: https://github.com/rust-lang/rust/blob/master/CONTRIBUTING.md#bug-reports
[01:06:45] 
[01:06:45] note: rustc 1.36.0-dev running on x86_64-unknown-linux-gnu
[01:06:45] 
[01:06:45] note: compiler flags: -Z threads=1 -Z ui-testing -Z unstable-options -Z query-dep-graph -C prefer-dynamic -C rpath
[01:06:45] 
[01:06:45] ------------------------------------------
[01:06:45] 
[01:06:45] 
[01:06:45] 
[01:06:45] ---- [ui] ui/dep-graph/dep-graph-trait-impl.rs stdout ----
[01:06:45] 
[01:06:45] error: Error: expected failure status (Some(1)) but received status Some(101).
[01:06:45] status: exit code: 101
[01:06:45] command: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "/checkout/src/test/ui/dep-graph/dep-graph-trait-impl.rs" "-Zthreads=1" "--target=x86_64-unknown-linux-gnu" "--error-format" "json" "-Zui-testing" "-C" "prefer-dynamic" "-o" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/dep-graph/dep-graph-trait-impl/a" "-Crpath" "-O" "-Zunstable-options" "-Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "-Z" "query-dep-graph" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/dep-graph/dep-graph-trait-impl/auxiliary" "-A" "unused"
[01:06:45] ------------------------------------------
[01:06:45] 
[01:06:45] ------------------------------------------
[01:06:45] stderr:
[01:06:45] stderr:
[01:06:45] ------------------------------------------
[01:06:45] thread 'rustc' panicked at 'src/librustc/session/mod.rs:815: Trying to get session directory from IncrCompSession `NotInitialized`', src/librustc/util/bug.rs:37:26
[01:06:45] 
[01:06:45] error: internal compiler error: unexpected panic
[01:06:45] 
[01:06:45] note: the compiler unexpectedly panicked. this is a bug.
[01:06:45] note: the compiler unexpectedly panicked. this is a bug.
[01:06:45] 
[01:06:45] note: we would appreciate a bug report: https://github.com/rust-lang/rust/blob/master/CONTRIBUTING.md#bug-reports
[01:06:45] 
[01:06:45] note: rustc 1.36.0-dev running on x86_64-unknown-linux-gnu
[01:06:45] 
[01:06:45] note: compiler flags: -Z threads=1 -Z ui-testing -Z unstable-options -Z query-dep-graph -C prefer-dynamic -C rpath
[01:06:45] 
[01:06:45] ------------------------------------------
[01:06:45] 
[01:06:45] 
[01:06:45] 
[01:06:45] ---- [ui] ui/dep-graph/dep-graph-variance-alias.rs stdout ----
[01:06:45] 
[01:06:45] error: Error: expected failure status (Some(1)) but received status Some(101).
[01:06:45] status: exit code: 101
[01:06:45] command: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "/checkout/src/test/ui/dep-graph/dep-graph-variance-alias.rs" "-Zthreads=1" "--target=x86_64-unknown-linux-gnu" "--error-format" "json" "-Zui-testing" "-C" "prefer-dynamic" "-o" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/dep-graph/dep-graph-variance-alias/a" "-Crpath" "-O" "-Zunstable-options" "-Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "-Z" "query-dep-graph" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/dep-graph/dep-graph-variance-alias/auxiliary" "-A" "unused"
[01:06:45] ------------------------------------------
[01:06:45] 
[01:06:45] ------------------------------------------
[01:06:45] stderr:
[01:06:45] stderr:
[01:06:45] ------------------------------------------
[01:06:45] thread 'rustc' panicked at 'src/librustc/session/mod.rs:815: Trying to get session directory from IncrCompSession `NotInitialized`', src/librustc/util/bug.rs:37:26
[01:06:45] 
[01:06:45] error: internal compiler error: unexpected panic
[01:06:45] 
[01:06:45] note: the compiler unexpectedly panicked. this is a bug.
[01:06:45] note: the compiler unexpectedly panicked. this is a bug.
[01:06:45] 
[01:06:45] note: we would appreciate a bug report: https://github.com/rust-lang/rust/blob/master/CONTRIBUTING.md#bug-reports
[01:06:45] 
[01:06:45] note: rustc 1.36.0-dev running on x86_64-unknown-linux-gnu
[01:06:45] 
[01:06:45] note: compiler flags: -Z threads=1 -Z ui-testing -Z unstable-options -Z query-dep-graph -C prefer-dynamic -C rpath
[01:06:45] 
[01:06:45] ------------------------------------------
[01:06:45] 
[01:06:45] 
[01:06:45] 
[01:06:45] ---- [ui] ui/dep-graph/dep-graph-type-alias.rs stdout ----
[01:06:45] 
[01:06:45] error: Error: expected failure status (Some(1)) but received status Some(101).
[01:06:45] status: exit code: 101
[01:06:45] command: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "/checkout/src/test/ui/dep-graph/dep-graph-type-alias.rs" "-Zthreads=1" "--target=x86_64-unknown-linux-gnu" "--error-format" "json" "-Zui-testing" "-C" "prefer-dynamic" "-o" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/dep-graph/dep-graph-type-alias/a" "-Crpath" "-O" "-Zunstable-options" "-Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "-Z" "query-dep-graph" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/dep-graph/dep-graph-type-alias/auxiliary" "-A" "unused"
[01:06:45] ------------------------------------------
[01:06:45] 
[01:06:45] ------------------------------------------
[01:06:45] stderr:
[01:06:45] stderr:
[01:06:45] ------------------------------------------
[01:06:45] thread 'rustc' panicked at 'src/librustc/session/mod.rs:815: Trying to get session directory from IncrCompSession `NotInitialized`', src/librustc/util/bug.rs:37:26
[01:06:45] 
[01:06:45] error: internal compiler error: unexpected panic
[01:06:45] 
[01:06:45] note: the compiler unexpectedly panicked. this is a bug.
[01:06:45] note: the compiler unexpectedly panicked. this is a bug.
[01:06:45] 
[01:06:45] note: we would appreciate a bug report: https://github.com/rust-lang/rust/blob/master/CONTRIBUTING.md#bug-reports
[01:06:45] 
[01:06:45] note: rustc 1.36.0-dev running on x86_64-unknown-linux-gnu
[01:06:45] 
[01:06:45] note: compiler flags: -Z threads=1 -Z ui-testing -Z unstable-options -Z query-dep-graph -C prefer-dynamic -C rpath
[01:06:45] 
[01:06:45] ------------------------------------------
[01:06:45] 
[01:06:45] 
---
[01:06:45] thread 'main' panicked at 'Some tests failed', src/tools/compiletest/src/main.rs:517:22
[01:06:45] note: Run with `RUST_BACKTRACE=1` environment variable to display a backtrace.
[01:06:45] 
[01:06:45] 
[01:06:45] command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools-bin/compiletest" "--compile-lib-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib" "--run-lib-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/lib" "--rustc-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "--src-base" "/checkout/src/test/ui" "--build-base" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui" "--stage-id" "stage2-x86_64-unknown-linux-gnu" "--mode" "ui" "--target" "x86_64-unknown-linux-gnu" "--host" "x86_64-unknown-linux-gnu" "--llvm-filecheck" "/usr/lib/llvm-6.0/bin/FileCheck" "--host-rustcflags" "-Crpath -O -Zunstable-options  -Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "--target-rustcflags" "-Crpath -O -Zunstable-options  -Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "--docck-python" "/usr/bin/python2.7" "--lldb-python" "/usr/bin/python2.7" "--gdb" "/usr/bin/gdb" "--quiet" "--llvm-version" "6.0.0\n" "--system-llvm" "--cc" "" "--cxx" "" "--cflags" "" "--llvm-components" "" "--llvm-cxxflags" "" "--adb-path" "adb" "--adb-test-dir" "/data/tmp/work" "--android-cross-path" "" "--color" "always"
[01:06:45] 
[01:06:45] 
[01:06:45] failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test
[01:06:45] Build completed unsuccessfully in 0:04:25
[01:06:45] Build completed unsuccessfully in 0:04:25
[01:06:45] make: *** [check] Error 1
[01:06:45] Makefile:48: recipe for target 'check' failed
The command "stamp sh -x -c "$RUN_SCRIPT"" exited with 2.
travis_time:start:1f6a53a2
$ date && (curl -fs --head https://google.com | grep ^Date: | sed 's/Date: //g' || true)
Sat Apr 20 03:56:40 UTC 2019

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

@bors

This comment has been minimized.

Copy link
Contributor

commented Apr 20, 2019

☀️ Try build successful - checks-travis
Build commit: d8c21d8

@Zoxc

This comment has been minimized.

Copy link
Contributor Author

commented Apr 20, 2019

@rust-timer

This comment has been minimized.

Copy link

commented Apr 20, 2019

Success: Queued d8c21d8 with parent 8aaae42, comparison URL.

@rust-timer

This comment has been minimized.

Copy link

commented Apr 20, 2019

Finished benchmarking try commit d8c21d8

@Zoxc

This comment has been minimized.

Copy link
Contributor Author

commented Apr 20, 2019

I merged SerializedDepNodeIndex and DepNodeIndex in the latest commit and made the indices persistent between sessions. The dep graph file format is now a list of changes to apply to the dep graph. Each compiler session will just appended the changes it did to the dep graph file.

These changes make the wall times pretty green (max-rss looks very green as well).

I've yet to add a GC scheme so currently the dep graph file will just keep growing.

@Zoxc

This comment has been minimized.

Copy link
Contributor Author

commented Apr 21, 2019

cc @nnethercote since you wanted to get rid of the dep graph memory ;)

@Zoxc Zoxc force-pushed the Zoxc:dep-stream branch from d4f2a87 to 7d0fe76 Apr 21, 2019

@Zoxc

This comment has been minimized.

Copy link
Contributor Author

commented Apr 21, 2019

@bors try

@bors

This comment has been minimized.

Copy link
Contributor

commented Apr 21, 2019

⌛️ Trying commit 7d0fe76 with merge 40424a5...

bors added a commit that referenced this pull request Apr 21, 2019

Auto merge of #60035 - Zoxc:dep-stream, r=<try>
[WIP] Serialize dep nodes in the background

This PR changes how we save the dep graph. Instead of storing the nodes in memory until the end of the compilation they are streamed to a file (in the background with `parallel_compiler`). This should hopefully reduce the memory usage with incremental compilation somewhat.

I've yet to add code to read the dep graph back into memory for dep graph debugging functionality.

I also want to get rid of `CurrentDepGraph::node_to_node_index`, but I'm not yet sure what the best approach is for that.

r? @michaelwoerister
@rust-highfive

This comment has been minimized.

Copy link
Collaborator

commented Apr 21, 2019

The job x86_64-gnu-llvm-6.0 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
travis_time:end:010b54cd:start=1555882601830158502,finish=1555882603885075442,duration=2054916940
$ git checkout -qf FETCH_HEAD
travis_fold:end:git.checkout

Encrypted environment variables have been removed for security reasons.
See https://docs.travis-ci.com/user/pull-requests/#pull-requests-and-security-restrictions
$ export SCCACHE_BUCKET=rust-lang-ci-sccache2
$ export SCCACHE_REGION=us-west-1
$ export GCP_CACHE_BUCKET=rust-lang-ci-cache
$ export AWS_ACCESS_KEY_ID=AKIA46X5W6CZEJZ6XT55
---
tidy check
[00:03:47] * 569 error codes
[00:03:47] * highest error code: E0725
[00:03:47] * 253 features
[00:03:48] invalid source: "git+https://github.com/Zoxc/rayon.git?branch=alt_scope#b6f36bdb56d85799115f86c097a64c8ffc0873d9"
[00:03:48] invalid source: "git+https://github.com/Zoxc/rayon.git?branch=alt_scope#b6f36bdb56d85799115f86c097a64c8ffc0873d9"
[00:03:48] some tidy checks failed
[00:03:48] 
[00:03:48] 
[00:03:48] command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools-bin/tidy" "/checkout/src" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "--no-vendor" "--quiet"
[00:03:48] 
[00:03:48] 
[00:03:48] failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test src/tools/tidy
[00:03:48] Build completed unsuccessfully in 0:00:44
[00:03:48] Build completed unsuccessfully in 0:00:44
[00:03:48] make: *** [tidy] Error 1
[00:03:48] Makefile:67: recipe for target 'tidy' failed
The command "stamp sh -x -c "$RUN_SCRIPT"" exited with 2.
travis_time:start:3bc7099e
$ date && (curl -fs --head https://google.com | grep ^Date: | sed 's/Date: //g' || true)
Sun Apr 21 21:40:43 UTC 2019
---
travis_time:end:26188ce0:start=1555882844491351764,finish=1555882844495880577,duration=4528813
travis_fold:end:after_failure.3
travis_fold:start:after_failure.4
travis_time:start:0d55d1c8
$ ln -s . checkout && for CORE in obj/cores/core.*; do EXE=$(echo $CORE | sed 's|obj/cores/core\.[0-9]*\.!checkout!\(.*\)|\1|;y|!|/|'); if [ -f "$EXE" ]; then printf travis_fold":start:crashlog\n\033[31;1m%s\033[0m\n" "$CORE"; gdb --batch -q -c "$CORE" "$EXE" -iex 'set auto-load off' -iex 'dir src/' -iex 'set sysroot .' -ex bt -ex q; echo travis_fold":"end:crashlog; fi; done || true
travis_fold:end:after_failure.4
travis_fold:start:after_failure.5
travis_time:start:1324b8d8
travis_time:start:1324b8d8
$ cat ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers || true
cat: ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers: No such file or directory
travis_fold:end:after_failure.5
travis_fold:start:after_failure.6
travis_time:start:0e2b1575
$ dmesg | grep -i kill

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

@rust-highfive

This comment has been minimized.

Copy link
Collaborator

commented Apr 21, 2019

The job x86_64-gnu-llvm-6.0 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
travis_time:end:12efebb2:start=1555884783437625845,finish=1555884785519084160,duration=2081458315
$ git checkout -qf FETCH_HEAD
travis_fold:end:git.checkout

Encrypted environment variables have been removed for security reasons.
See https://docs.travis-ci.com/user/pull-requests/#pull-requests-and-security-restrictions
$ export SCCACHE_BUCKET=rust-lang-ci-sccache2
$ export SCCACHE_REGION=us-west-1
$ export GCP_CACHE_BUCKET=rust-lang-ci-cache
$ export AWS_ACCESS_KEY_ID=AKIA46X5W6CZEJZ6XT55
---
tidy check
[00:03:58] * 569 error codes
[00:03:58] * highest error code: E0725
[00:03:58] * 253 features
[00:03:59] invalid source: "git+https://github.com/Zoxc/rayon.git?branch=alt_scope#b6f36bdb56d85799115f86c097a64c8ffc0873d9"
[00:03:59] invalid source: "git+https://github.com/Zoxc/rayon.git?branch=alt_scope#b6f36bdb56d85799115f86c097a64c8ffc0873d9"
[00:03:59] some tidy checks failed
[00:03:59] 
[00:03:59] 
[00:03:59] command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools-bin/tidy" "/checkout/src" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "--no-vendor" "--quiet"
[00:03:59] 
[00:03:59] 
[00:03:59] failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test src/tools/tidy
[00:03:59] Build completed unsuccessfully in 0:00:45
[00:03:59] Build completed unsuccessfully in 0:00:45
[00:03:59] make: *** [tidy] Error 1
[00:03:59] Makefile:67: recipe for target 'tidy' failed
The command "stamp sh -x -c "$RUN_SCRIPT"" exited with 2.
travis_time:start:0ec29b60
$ date && (curl -fs --head https://google.com | grep ^Date: | sed 's/Date: //g' || true)
Sun Apr 21 22:17:15 UTC 2019
---
travis_time:end:098dac86:start=1555885036797733927,finish=1555885036802058481,duration=4324554
travis_fold:end:after_failure.3
travis_fold:start:after_failure.4
travis_time:start:302f302e
$ ln -s . checkout && for CORE in obj/cores/core.*; do EXE=$(echo $CORE | sed 's|obj/cores/core\.[0-9]*\.!checkout!\(.*\)|\1|;y|!|/|'); if [ -f "$EXE" ]; then printf travis_fold":start:crashlog\n\033[31;1m%s\033[0m\n" "$CORE"; gdb --batch -q -c "$CORE" "$EXE" -iex 'set auto-load off' -iex 'dir src/' -iex 'set sysroot .' -ex bt -ex q; echo travis_fold":"end:crashlog; fi; done || true
travis_fold:end:after_failure.4
travis_fold:start:after_failure.5
travis_time:start:13509773
travis_time:start:13509773
$ cat ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers || true
cat: ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers: No such file or directory
travis_fold:end:after_failure.5
travis_fold:start:after_failure.6
travis_time:start:00d66de8
$ dmesg | grep -i kill

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

@Zoxc Zoxc force-pushed the Zoxc:dep-stream branch from 5ecb8fd to 9f268ad Jun 18, 2019

@rust-highfive

This comment has been minimized.

Copy link
Collaborator

commented Jun 18, 2019

The job x86_64-gnu-llvm-6.0 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
travis_time:end:14d734e4:start=1560821262578010413,finish=1560821264956695416,duration=2378685003
$ git checkout -qf FETCH_HEAD
travis_fold:end:git.checkout

Encrypted environment variables have been removed for security reasons.
See https://docs.travis-ci.com/user/pull-requests/#pull-requests-and-security-restrictions
$ export SCCACHE_BUCKET=rust-lang-ci-sccache2
$ export SCCACHE_REGION=us-west-1
$ export GCP_CACHE_BUCKET=rust-lang-ci-cache
$ export AWS_ACCESS_KEY_ID=AKIA46X5W6CZEJZ6XT55
---
[00:57:26] .........................................................................ii......................... 3900/5680
[00:57:28] ..............................................................................................i..... 4000/5680
[00:57:31] .................................................................................................... 4100/5680
[00:57:33] ..........................................................i......................................... 4200/5680
[00:57:35] ..............................................................................F..................... 4300/5680
[00:57:54] .................................................................................................... 4500/5680
[00:57:57] .................................................................................................... 4600/5680
[00:58:01] .................................................................................................... 4700/5680
[00:58:05] .................................................................................................... 4800/5680
[00:58:05] .................................................................................................... 4800/5680
[00:58:13] .................................................................................................... 4900/5680
[00:58:17] .................................................................................................... 5000/5680
[00:58:21] .................................................................................................... 5100/5680
[00:58:26] .................................................................................................... 5200/5680
[00:58:29] .........................F.......................................................................... 5300/5680
[00:58:36] .................................................................................................... 5500/5680
[00:58:39] .................................................................................................... 5600/5680
[00:58:41] ..................i.............................................................
[00:58:41] failures:
[00:58:41] failures:
[00:58:41] 
[00:58:41] ---- [ui] ui/pattern/const-pat-ice.rs stdout ----
[00:58:41] diff of stderr:
[00:58:41] 
[00:58:41] - thread 'rustc' panicked at 'assertion failed: rows.iter().all(|r| r.len() == v.len())', src/librustc_mir/hair/pattern/_match.rs:1084:5
[00:58:41] + thread '<unnamed>' panicked at 'assertion failed: rows.iter().all(|r| r.len() == v.len())', src/librustc_mir/hair/pattern/_match.rs:1084:5
[00:58:41] 3 
[00:58:41] 4 error: internal compiler error: unexpected panic
[00:58:41] 
[00:58:41] 
[00:58:41] 
[00:58:41] The actual stderr differed from the expected stderr.
[00:58:41] Actual stderr saved to /checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/pattern/const-pat-ice/const-pat-ice.stderr
[00:58:41] To update references, rerun the tests and pass the `--bless` flag
[00:58:41] To only update this specific test, also pass `--test-args pattern/const-pat-ice.rs`
[00:58:41] error: 1 errors occurred comparing output.
[00:58:41] status: exit code: 101
[00:58:41] status: exit code: 101
[00:58:41] command: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "/checkout/src/test/ui/pattern/const-pat-ice.rs" "-Zthreads=1" "--target=x86_64-unknown-linux-gnu" "--error-format" "json" "-Zui-testing" "-C" "prefer-dynamic" "--out-dir" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/pattern/const-pat-ice" "-Crpath" "-O" "-Cdebuginfo=0" "-Zunstable-options" "-Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/pattern/const-pat-ice/auxiliary" "-A" "unused"
[00:58:41] ------------------------------------------
[00:58:41] 
[00:58:41] ------------------------------------------
[00:58:41] stderr:
[00:58:41] stderr:
[00:58:41] ------------------------------------------
[00:58:41] thread '<unnamed>' panicked at 'assertion failed: rows.iter().all(|r| r.len() == v.len())', src/librustc_mir/hair/pattern/_match.rs:1084:5
[00:58:41] 
[00:58:41] error: internal compiler error: unexpected panic
[00:58:41] 
[00:58:41] note: the compiler unexpectedly panicked. this is a bug.
[00:58:41] note: the compiler unexpectedly panicked. this is a bug.
[00:58:41] 
[00:58:41] note: we would appreciate a bug report: https://github.com/rust-lang/rust/blob/master/CONTRIBUTING.md#bug-reports
[00:58:41] 
[00:58:41] note: rustc 1.37.0-dev running on x86_64-unknown-linux-gnu
[00:58:41] 
[00:58:41] note: compiler flags: -Z threads=1 -Z ui-testing -Z unstable-options -C prefer-dynamic -C rpath -C debuginfo=0
[00:58:41] 
[00:58:41] ------------------------------------------
[00:58:41] 
[00:58:41] 
[00:58:41] 
[00:58:41] ---- [ui] ui/traits/cycle-cache-err-60010.rs stdout ----
[00:58:41] diff of stderr:
[00:58:41] 
[00:58:41] 6    |
[00:58:41] 7    = note: required because of the requirements on the impl of `Query<RootDatabase>` for `ParseQuery`
[00:58:41] 8 
[00:58:41] - error[E0275]: overflow evaluating the requirement `RootDatabase: SourceDatabase`
[00:58:41] -   --> $DIR/cycle-cache-err-60010.rs:30:6
[00:58:41] -    |
[00:58:41] - LL | impl Database for RootDatabase {
[00:58:41] -    |
[00:58:41] -    |
[00:58:41] -    = note: required because of the requirements on the impl of `Query<RootDatabase>` for `ParseQuery`
[00:58:41] -    = note: required because it appears within the type `SalsaStorage`
[00:58:41] - error: aborting due to 2 previous errors
[00:58:41] + error: aborting due to previous error
[00:58:41] 19 
[00:58:41] 20 For more information about this error, try `rustc --explain E0275`.
[00:58:41] 20 For more information about this error, try `rustc --explain E0275`.
[00:58:41] 21 
[00:58:41] 
[00:58:41] 
[00:58:41] The actual stderr differed from the expected stderr.
[00:58:41] Actual stderr saved to /checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/traits/cycle-cache-err-60010/cycle-cache-err-60010.stderr
[00:58:41] To update references, rerun the tests and pass the `--bless` flag
[00:58:41] To only update this specific test, also pass `--test-args traits/cycle-cache-err-60010.rs`
[00:58:41] error: 1 errors occurred comparing output.
[00:58:41] status: exit code: 1
[00:58:41] status: exit code: 1
[00:58:41] command: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "/checkout/src/test/ui/traits/cycle-cache-err-60010.rs" "-Zthreads=1" "--target=x86_64-unknown-linux-gnu" "--error-format" "json" "-Zui-testing" "-C" "prefer-dynamic" "--out-dir" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/traits/cycle-cache-err-60010" "-Crpath" "-O" "-Cdebuginfo=0" "-Zunstable-options" "-Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/traits/cycle-cache-err-60010/auxiliary" "-A" "unused"
[00:58:41] ------------------------------------------
[00:58:41] 
[00:58:41] ------------------------------------------
[00:58:41] stderr:
[00:58:41] stderr:
[00:58:41] ------------------------------------------
[00:58:41] error[E0275]: overflow evaluating the requirement `RootDatabase: SourceDatabase`
[00:58:41]    |
[00:58:41]    |
[00:58:41] LL |     _parse: <ParseQuery as Query<RootDatabase>>::Data, //~ ERROR overflow
[00:58:41]    |
[00:58:41]    |
[00:58:41]    = note: required because of the requirements on the impl of `Query<RootDatabase>` for `ParseQuery`
[00:58:41] error: aborting due to previous error
[00:58:41] 
[00:58:41] For more information about this error, try `rustc --explain E0275`.
[00:58:41] 
---
[00:58:41] thread 'main' panicked at 'Some tests failed', src/tools/compiletest/src/main.rs:521:22
[00:58:41] note: Run with `RUST_BACKTRACE=1` environment variable to display a backtrace.
[00:58:41] 
[00:58:41] 
[00:58:41] command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools-bin/compiletest" "--compile-lib-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib" "--run-lib-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/lib" "--rustc-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "--src-base" "/checkout/src/test/ui" "--build-base" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui" "--stage-id" "stage2-x86_64-unknown-linux-gnu" "--mode" "ui" "--target" "x86_64-unknown-linux-gnu" "--host" "x86_64-unknown-linux-gnu" "--llvm-filecheck" "/usr/lib/llvm-6.0/bin/FileCheck" "--host-rustcflags" "-Crpath -O -Cdebuginfo=0 -Zunstable-options  -Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "--target-rustcflags" "-Crpath -O -Cdebuginfo=0 -Zunstable-options  -Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "--docck-python" "/usr/bin/python2.7" "--lldb-python" "/usr/bin/python2.7" "--gdb" "/usr/bin/gdb" "--quiet" "--llvm-version" "6.0.0\n" "--system-llvm" "--cc" "" "--cxx" "" "--cflags" "" "--llvm-components" "" "--llvm-cxxflags" "" "--adb-path" "adb" "--adb-test-dir" "/data/tmp/work" "--android-cross-path" "" "--color" "always"
[00:58:41] 
[00:58:41] 
[00:58:41] failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test
[00:58:41] Build completed unsuccessfully in 0:53:52
---
travis_time:end:03673819:start=1560824799886129220,finish=1560824799891641554,duration=5512334
travis_fold:end:after_failure.3
travis_fold:start:after_failure.4
travis_time:start:180c615b
$ ln -s . checkout && for CORE in obj/cores/core.*; do EXE=$(echo $CORE | sed 's|obj/cores/core\.[0-9]*\.!checkout!\(.*\)|\1|;y|!|/|'); if [ -f "$EXE" ]; then printf travis_fold":start:crashlog\n\033[31;1m%s\033[0m\n" "$CORE"; gdb --batch -q -c "$CORE" "$EXE" -iex 'set auto-load off' -iex 'dir src/' -iex 'set sysroot .' -ex bt -ex q; echo travis_fold":"end:crashlog; fi; done || true
travis_fold:end:after_failure.4
travis_fold:start:after_failure.5
travis_time:start:0265b5e8
travis_time:start:0265b5e8
$ cat ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers || true
cat: ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers: No such file or directory
travis_fold:end:after_failure.5
travis_fold:start:after_failure.6
travis_time:start:10754a6c
$ dmesg | grep -i kill

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

bors added a commit that referenced this pull request Jun 22, 2019

Auto merge of #62038 - Zoxc:pre-gc-dep-graph, r=<try>
[WIP] Make dep node indices persistent between sessions

This makes marking dep nodes green faster (and lock free in the case with no diagnostics). This change is split out from #60035.

Unlike #60035 this makes loading the dep graph slower because it loads 2 copies of the dep graph, one immutable and one mutable.

Based on #61845, #61779 and #61923.

bors added a commit that referenced this pull request Jun 22, 2019

Auto merge of #61923 - Zoxc:dep-stream-prefix-2, r=<try>
Prerequisites from dep graph refactoring #2

Split out from #60035 and overlaps with #60559.

bors added a commit that referenced this pull request Jun 22, 2019

Auto merge of #61845 - Zoxc:sharded-dep-graph, r=<try>
Use a sharded dep node to dep node index map

This reduces lock contention a bit for incremental compilation. #60035 gets rid of the locks, but is much larger and probably won't land until @michaelwoerister returns. This reduces clean incremental times for `winapi` from 18.37s to 16.43s on 16 threads / 8 cores.

Based on #61779.

r? @eddyb

bors added a commit that referenced this pull request Jun 22, 2019

Auto merge of #62038 - Zoxc:pre-gc-dep-graph, r=<try>
[WIP] Make dep node indices persistent between sessions

This makes marking dep nodes green faster (and lock free in the case with no diagnostics). This change is split out from #60035.

Unlike #60035 this makes loading the dep graph slower because it loads 2 copies of the dep graph, one immutable and one mutable.

Based on #61845, #61779 and #61923.

@Zoxc Zoxc changed the title Refactor dep graph representation both in memory and on disk [WIP] Refactor dep graph representation both in memory and on disk Jul 1, 2019

Centril added a commit to Centril/rust that referenced this pull request Jul 11, 2019

Rollup merge of rust-lang#61923 - Zoxc:dep-stream-prefix-2, r=pnkfelix
Prerequisites from dep graph refactoring rust-lang#2

Split out from rust-lang#60035 and overlaps with rust-lang#60559.
@joelpalmer

This comment has been minimized.

Copy link

commented Jul 22, 2019

Ping from Triage: @Zoxc is there an update on this PR? Thanks!

@nnethercote

This comment has been minimized.

Copy link
Contributor

commented Jul 22, 2019

I want to encourage progress on this PR because the memory savings are big. (It's also a perf win -- although instruction counts increase in most cases, wall times mostly drop.)

A Firefox developer was complaining recently that compilation of Stylo within Firefox was causing OOMs because memory usage was exceeding 3.5 GiB. I did some profiling runs with Massif and DHAT, and for incremental builds the dep graph dominates the memory profiles -- most of the top entries involve the dep graph.

Here are some of the dep graph records for a style-servo-Debug-PatchIncr1 run. The "At t-gmax" line is the most important one -- it shows the fraction of the heap at its peak taken by blocks allocated at that point at the global heap maximum. E.g. the first record shows that a single 285 MB allocation took up 10.51% of heap memory at the heap's peak. The records below combine to account for 47.91% of the heap at its peak.

-----------------------------------------------------------------------------
At t-gmax: 285,212,672 bytes (10.51%) in 1 blocks (0%), avg size 285,212,672 bytes
Reads:     187,921,121 bytes (0.1%, 432.36/Minstr), 0.33/byte
Writes:    187,921,133 bytes (0.34%, 432.36/Minstr), 0.33/byte
Allocated at {
  ^1: 0x599FA20: alloc (alloc.rs:81)
  ^2: 0x599FA20: alloc (alloc.rs:169)
  ^3: 0x599FA20: reserve_internal<u8,alloc::alloc::Global> (raw_vec.rs:668)
  ^4: 0x599FA20: reserve<u8,alloc::alloc::Global> (raw_vec.rs:491)
  ^5: 0x599FA20: reserve<u8> (vec.rs:457)
  ^6: 0x599FA20: spec_extend<u8> (vec.rs:2019)
  ^7: 0x599FA20: alloc::vec::Vec<T>::extend_from_slice (vec.rs:1464)
  ^8: 0x59EC89F: emit_raw_bytes (opaque.rs:26)
  ^9: 0x59EC89F: rustc_incremental::persist::file_format::write_file_header (file_format.rs:32)
  #10: 0x59E88EA: save_in<closure> (save.rs:114)
  #11: 0x59E88EA: rustc_incremental::persist::save::save_dep_graph::{{closure}}::{{closure}}::{{closure}} (save.rs:39)
}

-----------------------------------------------------------------------------
At t-gmax: 242,995,920 bytes (8.95%) in 1 blocks (0%), avg size 242,995,920 bytes
Reads:     219,685,224 bytes (0.12%, 505.44/Minstr), 0.9/byte
Writes:    237,912,640 bytes (0.43%, 547.37/Minstr), 0.98/byte
Allocated at {
  #1: 0x5F072D1: alloc (alloc.rs:81)
  #2: 0x5F072D1: alloc (alloc.rs:169)
  #3: 0x5F072D1: allocate_in<rustc::dep_graph::graph::DepNodeData,alloc::alloc::Global> (raw_vec.rs:95)
  #4: 0x5F072D1: with_capacity<rustc::dep_graph::graph::DepNodeData> (raw_vec.rs:139)
  #5: 0x5F072D1: with_capacity<rustc::dep_graph::graph::DepNodeData> (vec.rs:355)
  #6: 0x5F072D1: with_capacity<rustc::dep_graph::graph::DepNodeIndex,rustc::dep_graph::graph::DepNodeData> (indexed_vec.rs:559)
  #7: 0x5F072D1: new (graph.rs:1004)
  #8: 0x5F072D1: rustc::dep_graph::graph::DepGraph::new (graph.rs:102)
  #9: 0x4F846EB: {{closure}} (queries.rs:182)
  #10: 0x4F846EB: compute<rustc::dep_graph::graph::DepGraph,closure> (queries.rs:40)
  #11: 0x4F846EB: rustc_interface::queries::<impl rustc_interface::interface::Compiler>::dep_graph (queries.rs:172)
  #12: 0x4F982D5: {{closure}} (queries.rs:198)
  #13: 0x4F982D5: rustc_interface::passes::BoxedResolver::access::{{closure}} (<::rustc_data_structures::box_region::declare_box_region_type macros>:22)
}

-----------------------------------------------------------------------------
At t-gmax: 138,412,048 bytes (5.1%) in 1 blocks (0%), avg size 138,412,048 bytes
Reads:     189,062,071 bytes (0.1%, 434.98/Minstr), 1.37/byte
Writes:    96,502,896 bytes (0.18%, 222.03/Minstr), 0.7/byte
Allocated at {
  ^1: 0x5EF80D9: alloc (alloc.rs:81)
  ^2: 0x5EF80D9: new_uninitialized<(rustc::hir::item_local_id_inner::ItemLocalId, alloc::vec::Vec<rustc::ty::adjustment::Adjustment>)> (mod.rs:379)
  ^3: 0x5EF80D9: hashbrown::raw::RawTable<T>::try_with_capacity (mod.rs:403)
  #4: 0x5EB882B: resize<(rustc::dep_graph::dep_node::DepNode, rustc::dep_graph::serialized::SerializedDepNodeIndex),closure> (mod.rs:776)
  #5: 0x5EB882B: hashbrown::raw::RawTable<T>::reserve_rehash (mod.rs:661)
  #6: 0x5F0B93E: reserve<(rustc::dep_graph::dep_node::DepNode, rustc::dep_graph::serialized::SerializedDepNodeIndex),closure> (mod.rs:621)
  #7: 0x5F0B93E: reserve<rustc::dep_graph::dep_node::DepNode,rustc::dep_graph::serialized::SerializedDepNodeIndex,core::hash::BuildHasherDefault<rustc_hash::FxHasher>> (map.rs:576)
  #8: 0x5F0B93E: extend<rustc::dep_graph::dep_node::DepNode,rustc::dep_graph::serialized::SerializedDepNodeIndex,core::hash::BuildHasherDefault<rustc_hash::FxHasher>,core::iter::adapters::Map<core::iter::adapters::Map<core::iter::adapters::Enumerate<core::slice::Iter<rustc::dep_graph::dep_node::DepNode>>, rustc_data_structures::indexed_vec::IntoIdx<rustc::dep_graph::serialized::SerializedDepNodeIndex>>, closure>> (map.rs:2465)
  #9: 0x5F0B93E: extend<rustc::dep_graph::dep_node::DepNode,rustc::dep_graph::serialized::SerializedDepNodeIndex,core::hash::BuildHasherDefault<rustc_hash::FxHasher>,core::iter::adapters::Map<core::iter::adapters::Map<core::iter::adapters::Enumerate<core::slice::Iter<rustc::dep_graph::dep_node::DepNode>>, rustc_data_structures::indexed_vec::IntoIdx<rustc::dep_graph::serialized::SerializedDepNodeIndex>>, closure>> (map.rs:2379)
  #10: 0x5F0B93E: from_iter<rustc::dep_graph::dep_node::DepNode,rustc::dep_graph::serialized::SerializedDepNodeIndex,core::hash::BuildHasherDefault<rustc_hash::FxHasher>,core::iter::adapters::Map<core::iter::adapters::Map<core::iter::adapters::Enumerate<core::slice::Iter<rustc::dep_graph::dep_node::DepNode>>, rustc_data_structures::indexed_vec::IntoIdx<rustc::dep_graph::serialized::SerializedDepNodeIndex>>, closure>> (map.rs:2366)
  #11: 0x5F0B93E: collect<core::iter::adapters::Map<core::iter::adapters::Map<core::iter::adapters::Enumerate<core::slice::Iter<rustc::dep_graph::dep_node::DepNode>>, rustc_data_structures::indexed_vec::IntoIdx<rustc::dep_graph::serialized::SerializedDepNodeIndex>>, closure>,std::collections::hash::map::HashMap<rustc::dep_graph::dep_node::DepNode, rustc::dep_graph::serialized::SerializedDepNodeIndex, core::hash::BuildHasherDefault<rustc_hash::FxHasher>>> (iterator.rs:1464)
  #12: 0x5F0B93E: rustc::dep_graph::prev::PreviousDepGraph::new (prev.rs:14)
}

-----------------------------------------------------------------------------
At t-gmax: 138,412,048 bytes (5.1%) in 1 blocks (0%), avg size 138,412,048 bytes
Reads:     318,775,911 bytes (0.17%, 733.42/Minstr), 2.3/byte
Writes:    93,411,560 bytes (0.17%, 214.92/Minstr), 0.67/byte
Allocated at {
  ^1: 0x5EF80D9: alloc (alloc.rs:81)
  ^2: 0x5EF80D9: new_uninitialized<(rustc::hir::item_local_id_inner::ItemLocalId, alloc::vec::Vec<rustc::ty::adjustment::Adjustment>)> (mod.rs:379)
  ^3: 0x5EF80D9: hashbrown::raw::RawTable<T>::try_with_capacity (mod.rs:403)
  #4: 0x5F07302: with_capacity<(rustc::dep_graph::dep_node::DepNode, rustc::dep_graph::graph::DepNodeIndex)> (mod.rs:414)
  #5: 0x5F07302: with_capacity_and_hasher<rustc::dep_graph::dep_node::DepNode,rustc::dep_graph::graph::DepNodeIndex,core::hash::BuildHasherDefault<rustc_hash::FxHasher>> (map.rs:288)
  #6: 0x5F07302: with_capacity_and_hasher<rustc::dep_graph::dep_node::DepNode,rustc::dep_graph::graph::DepNodeIndex,core::hash::BuildHasherDefault<rustc_hash::FxHasher>> (map.rs:525)
  #7: 0x5F07302: new (graph.rs:1005)
  #8: 0x5F07302: rustc::dep_graph::graph::DepGraph::new (graph.rs:102)
  #9: 0x4F846EB: {{closure}} (queries.rs:182)
  #10: 0x4F846EB: compute<rustc::dep_graph::graph::DepGraph,closure> (queries.rs:40)
  #11: 0x4F846EB: rustc_interface::queries::<impl rustc_interface::interface::Compiler>::dep_graph (queries.rs:172)
}

-----------------------------------------------------------------------------
At t-gmax: 85,836,744 bytes (3.16%) in 1 blocks (0%), avg size 85,836,744 bytes
Reads:     85,836,744 bytes (0.05%, 197.49/Minstr), 1/byte
Writes:    85,836,744 bytes (0.16%, 197.49/Minstr), 1/byte
Allocated at {
  #1: 0x5F09062: alloc (alloc.rs:81)
  #2: 0x5F09062: alloc (alloc.rs:169)
  #3: 0x5F09062: allocate_in<rustc::dep_graph::serialized::SerializedDepNodeIndex,alloc::alloc::Global> (raw_vec.rs:95)
  #4: 0x5F09062: with_capacity<rustc::dep_graph::serialized::SerializedDepNodeIndex> (raw_vec.rs:139)
  #5: 0x5F09062: with_capacity<rustc::dep_graph::serialized::SerializedDepNodeIndex> (vec.rs:355)
  #6: 0x5F09062: rustc::dep_graph::graph::DepGraph::serialize (graph.rs:517)
  #7: 0x59E6808: {{closure}} (save.rs:138)
  #8: 0x59E6808: time_ext<rustc::dep_graph::serialized::SerializedDepGraph,closure> (common.rs:151)
  #9: 0x59E6808: time<rustc::dep_graph::serialized::SerializedDepGraph,closure> (common.rs:145)
  #10: 0x59E6808: encode_dep_graph (save.rs:137)
  #11: 0x59E6808: rustc_incremental::persist::save::save_dep_graph::{{closure}}::{{closure}}::{{closure}}::{{closure}}::{{closure}} (save.rs:43)
  #12: 0x59E8982: time_ext<(),closure> (common.rs:151)
  #13: 0x59E8982: time<(),closure> (common.rs:145)
  #14: 0x59E8982: {{closure}} (save.rs:42)
  #15: 0x59E8982: save_in<closure> (save.rs:115)
  #16: 0x59E8982: rustc_incremental::persist::save::save_dep_graph::{{closure}}::{{closure}}::{{closure}} (save.rs:39)
}

-----------------------------------------------------------------------------
At t-gmax: 85,828,080 bytes (3.16%) in 1 blocks (0%), avg size 85,828,080 bytes
Reads:     77,326,412 bytes (0.04%, 177.91/Minstr), 0.9/byte
Writes:    85,828,080 bytes (0.16%, 197.47/Minstr), 1/byte
Allocated at {
  #1: 0x59E4462: alloc (alloc.rs:81)
  #2: 0x59E4462: alloc (alloc.rs:169)
  #3: 0x59E4462: allocate_in<rustc::dep_graph::serialized::SerializedDepNodeIndex,alloc::alloc::Global> (raw_vec.rs:95)
  #4: 0x59E4462: with_capacity<rustc::dep_graph::serialized::SerializedDepNodeIndex> (raw_vec.rs:139)
  #5: 0x59E4462: with_capacity<rustc::dep_graph::serialized::SerializedDepNodeIndex> (vec.rs:355)
  #6: 0x59E4462: {{closure}}<rustc::dep_graph::serialized::SerializedDepNodeIndex,serialize::opaque::Decoder> (serialize.rs:616)
  #7: 0x59E4462: read_seq<serialize::opaque::Decoder,alloc::vec::Vec<rustc::dep_graph::serialized::SerializedDepNodeIndex>,closure> (serialize.rs:267)
  #8: 0x59E4462: decode<rustc::dep_graph::serialized::SerializedDepNodeIndex,serialize::opaque::Decoder> (serialize.rs:615)
  #9: 0x59E4462: call_once<fn(&mut serialize::opaque::Decoder) -> core::result::Result<alloc::vec::Vec<rustc::dep_graph::serialized::SerializedDepNodeIndex>, alloc::string::String>,(&mut serialize::opaque::Decoder)> (function.rs:231)
  #10: 0x59E4462: read_struct_field<serialize::opaque::Decoder,alloc::vec::Vec<rustc::dep_graph::serialized::SerializedDepNodeIndex>,fn(&mut serialize::opaque::Decoder) -> core::result::Result<alloc::vec::Vec<rustc::dep_graph::serialized::SerializedDepNodeIndex>, alloc::string::String>> (serialize.rs:221)
  #11: 0x59E4462: {{closure}}<serialize::opaque::Decoder> (serialized.rs:25)
  #12: 0x59E4462: read_struct<serialize::opaque::Decoder,rustc::dep_graph::serialized::SerializedDepGraph,closure> (serialize.rs:215)
  #13: 0x59E4462: decode<serialize::opaque::Decoder> (serialized.rs:12)
  #14: 0x59E4462: rustc_incremental::persist::load::load_dep_graph::{{closure}}::{{closure}} (load.rs:185)
  #15: 0x59931F6: time_ext<rustc_incremental::persist::load::LoadResult<(rustc::dep_graph::prev::PreviousDepGraph, std::collections::hash::map::HashMap<rustc::dep_graph::dep_node::WorkProductId, rustc::dep_graph::graph::WorkProduct, core::hash::BuildHasherDefault<rustc_hash::FxHasher>>)>,closure> (common.rs:151)
  #16: 0x59931F6: {{closure}} (load.rs:163)
  #17: 0x59931F6: std::sys_common::backtrace::__rust_begin_short_backtrace (backtrace.rs:77)
  #18: 0x59938B7: {{closure}}<closure,rustc_incremental::persist::load::LoadResult<(rustc::dep_graph::prev::PreviousDepGraph, std::collections::hash::map::HashMap<rustc::dep_graph::dep_node::WorkProductId, rustc::dep_graph::graph::WorkProduct, core::hash::BuildHasherDefault<rustc_hash::FxHasher>>)>> (mod.rs:470)
  #19: 0x59938B7: call_once<rustc_incremental::persist::load::LoadResult<(rustc::dep_graph::prev::PreviousDepGraph, std::collections::hash::map::HashMap<rustc::dep_graph::dep_node::WorkProductId, rustc::dep_graph::graph::WorkProduct, core::hash::BuildHasherDefault<rustc_hash::FxHasher>>)>,closure> (panic.rs:315)
  #20: 0x59938B7: std::panicking::try::do_call (panicking.rs:296)
}

-----------------------------------------------------------------------------
At t-gmax: 71,464,704 bytes (2.63%) in 1 blocks (0%), avg size 71,464,704 bytes
Reads:     196,646,536 bytes (0.1%, 452.43/Minstr), 2.75/byte
Writes:    74,442,400 bytes (0.14%, 171.27/Minstr), 1.04/byte
Allocated at {
  #1: 0x59E3B0D: alloc (alloc.rs:81)
  #2: 0x59E3B0D: alloc (alloc.rs:169)
  #3: 0x59E3B0D: allocate_in<rustc::dep_graph::dep_node::DepNode,alloc::alloc::Global> (raw_vec.rs:95)
  #4: 0x59E3B0D: with_capacity<rustc::dep_graph::dep_node::DepNode> (raw_vec.rs:139)
  #5: 0x59E3B0D: with_capacity<rustc::dep_graph::dep_node::DepNode> (vec.rs:355)
  #6: 0x59E3B0D: {{closure}}<rustc::dep_graph::dep_node::DepNode,serialize::opaque::Decoder> (serialize.rs:616)
  #7: 0x59E3B0D: read_seq<serialize::opaque::Decoder,alloc::vec::Vec<rustc::dep_graph::dep_node::DepNode>,closure> (serialize.rs:267)
  #8: 0x59E3B0D: decode<rustc::dep_graph::dep_node::DepNode,serialize::opaque::Decoder> (serialize.rs:615)
  #9: 0x59E3B0D: decode<rustc::dep_graph::serialized::SerializedDepNodeIndex,rustc::dep_graph::dep_node::DepNode,serialize::opaque::Decoder> (indexed_vec.rs:532)
  #10: 0x59E3B0D: call_once<fn(&mut serialize::opaque::Decoder) -> core::result::Result<rustc_data_structures::indexed_vec::IndexVec<rustc::dep_graph::serialized::SerializedDepNodeIndex, rustc::dep_graph::dep_node::DepNode>, alloc::string::String>,(&mut serialize::opaque::Decoder)> (function.rs:231)
  #11: 0x59E3B0D: read_struct_field<serialize::opaque::Decoder,rustc_data_structures::indexed_vec::IndexVec<rustc::dep_graph::serialized::SerializedDepNodeIndex, rustc::dep_graph::dep_node::DepNode>,fn(&mut serialize::opaque::Decoder) -> core::result::Result<rustc_data_structures::indexed_vec::IndexVec<rustc::dep_graph::serialized::SerializedDepNodeIndex, rustc::dep_graph::dep_node::DepNode>, alloc::string::String>> (serialize.rs:221)
  #12: 0x59E3B0D: {{closure}}<serialize::opaque::Decoder> (serialized.rs:15)
  #13: 0x59E3B0D: read_struct<serialize::opaque::Decoder,rustc::dep_graph::serialized::SerializedDepGraph,closure> (serialize.rs:215)
  #14: 0x59E3B0D: decode<serialize::opaque::Decoder> (serialized.rs:12)
  #15: 0x59E3B0D: rustc_incremental::persist::load::load_dep_graph::{{closure}}::{{closure}} (load.rs:185)
  #16: 0x59931F6: time_ext<rustc_incremental::persist::load::LoadResult<(rustc::dep_graph::prev::PreviousDepGraph, std::collections::hash::map::HashMap<rustc::dep_graph::dep_node::WorkProductId, rustc::dep_graph::graph::WorkProduct, core::hash::BuildHasherDefault<rustc_hash::FxHasher>>)>,closure> (common.rs:151)
  #17: 0x59931F6: {{closure}} (load.rs:163)
  #18: 0x59931F6: std::sys_common::backtrace::__rust_begin_short_backtrace (backtrace.rs:77)
  #19: 0x59938B7: {{closure}}<closure,rustc_incremental::persist::load::LoadResult<(rustc::dep_graph::prev::PreviousDepGraph, std::collections::hash::map::HashMap<rustc::dep_graph::dep_node::WorkProductId, rustc::dep_graph::graph::WorkProduct, core::hash::BuildHasherDefault<rustc_hash::FxHasher>>)>> (mod.rs:470)
  #20: 0x59938B7: call_once<rustc_incremental::persist::load::LoadResult<(rustc::dep_graph::prev::PreviousDepGraph, std::collections::hash::map::HashMap<rustc::dep_graph::dep_node::WorkProductId, rustc::dep_graph::graph::WorkProduct, core::hash::BuildHasherDefault<rustc_hash::FxHasher>>)>,closure> (panic.rs:315)
  #21: 0x59938B7: std::panicking::try::do_call (panicking.rs:296)
}

-----------------------------------------------------------------------------
At t-gmax: 71,373,792 bytes (2.63%) in 1 blocks (0%), avg size 71,373,792 bytes
Reads:     50,556,436 bytes (0.03%, 116.32/Minstr), 0.71/byte
Writes:    71,373,792 bytes (0.13%, 164.21/Minstr), 1/byte
Allocated at {
  #1: 0x5F08F3E: alloc (alloc.rs:81)
  #2: 0x5F08F3E: alloc (alloc.rs:169)
  #3: 0x5F08F3E: reserve_internal<rustc::dep_graph::dep_node::DepNode,alloc::alloc::Global> (raw_vec.rs:668)
  #4: 0x5F08F3E: reserve<rustc::dep_graph::dep_node::DepNode,alloc::alloc::Global> (raw_vec.rs:491)
  #5: 0x5F08F3E: reserve<rustc::dep_graph::dep_node::DepNode> (vec.rs:457)
  #6: 0x5F08F3E: spec_extend<rustc::dep_graph::dep_node::DepNode,core::iter::adapters::Map<core::slice::Iter<rustc::dep_graph::graph::DepNodeData>, closure>> (vec.rs:1956)
  #7: 0x5F08F3E: from_iter<rustc::dep_graph::dep_node::DepNode,core::iter::adapters::Map<core::slice::Iter<rustc::dep_graph::graph::DepNodeData>, closure>> (vec.rs:1943)
  #8: 0x5F08F3E: from_iter<rustc::dep_graph::dep_node::DepNode,core::iter::adapters::Map<core::slice::Iter<rustc::dep_graph::graph::DepNodeData>, closure>> (vec.rs:1830)
  #9: 0x5F08F3E: from_iter<rustc::dep_graph::serialized::SerializedDepNodeIndex,rustc::dep_graph::dep_node::DepNode,core::iter::adapters::Map<core::slice::Iter<rustc::dep_graph::graph::DepNodeData>, closure>> (indexed_vec.rs:776)
  #10: 0x5F08F3E: collect<core::iter::adapters::Map<core::slice::Iter<rustc::dep_graph::graph::DepNodeData>, closure>,rustc_data_structures::indexed_vec::IndexVec<rustc::dep_graph::serialized::SerializedDepNodeIndex, rustc::dep_graph::dep_node::DepNode>> (iterator.rs:1464)
  #11: 0x5F08F3E: rustc::dep_graph::graph::DepGraph::serialize (graph.rs:510)
  #12: 0x59E6808: {{closure}} (save.rs:138)
  #13: 0x59E6808: time_ext<rustc::dep_graph::serialized::SerializedDepGraph,closure> (common.rs:151)
  #14: 0x59E6808: time<rustc::dep_graph::serialized::SerializedDepGraph,closure> (common.rs:145)
  #15: 0x59E6808: encode_dep_graph (save.rs:137)
  #16: 0x59E6808: rustc_incremental::persist::save::save_dep_graph::{{closure}}::{{closure}}::{{closure}}::{{closure}}::{{closure}} (save.rs:43)
  #17: 0x59E8982: time_ext<(),closure> (common.rs:151)
  #18: 0x59E8982: time<(),closure> (common.rs:145)
  #19: 0x59E8982: {{closure}} (save.rs:42)
  #20: 0x59E8982: save_in<closure> (save.rs:115)
  #21: 0x59E8982: rustc_incremental::persist::save::save_dep_graph::{{closure}}::{{closure}}::{{closure}} (save.rs:39)
}

-----------------------------------------------------------------------------
At t-gmax: 47,643,136 bytes (1.76%) in 1 blocks (0%), avg size 47,643,136 bytes
Reads:     46,826,928 bytes (0.02%, 107.74/Minstr), 0.98/byte
Writes:    47,643,136 bytes (0.09%, 109.61/Minstr), 1/byte
Allocated at {
  #1: 0x59E3EEE: alloc (alloc.rs:81)
  #2: 0x59E3EEE: alloc (alloc.rs:169)
  #3: 0x59E3EEE: allocate_in<rustc_data_structures::fingerprint::Fingerprint,alloc::alloc::Global> (raw_vec.rs:95)
  #4: 0x59E3EEE: with_capacity<rustc_data_structures::fingerprint::Fingerprint> (raw_vec.rs:139)
  #5: 0x59E3EEE: with_capacity<rustc_data_structures::fingerprint::Fingerprint> (vec.rs:355)
  #6: 0x59E3EEE: {{closure}}<rustc_data_structures::fingerprint::Fingerprint,serialize::opaque::Decoder> (serialize.rs:616)
  #7: 0x59E3EEE: read_seq<serialize::opaque::Decoder,alloc::vec::Vec<rustc_data_structures::fingerprint::Fingerprint>,closure> (serialize.rs:267)
  #8: 0x59E3EEE: decode<rustc_data_structures::fingerprint::Fingerprint,serialize::opaque::Decoder> (serialize.rs:615)
  #9: 0x59E3EEE: decode<rustc::dep_graph::serialized::SerializedDepNodeIndex,rustc_data_structures::fingerprint::Fingerprint,serialize::opaque::Decoder> (indexed_vec.rs:532)
  #10: 0x59E3EEE: call_once<fn(&mut serialize::opaque::Decoder) -> core::result::Result<rustc_data_structures::indexed_vec::IndexVec<rustc::dep_graph::serialized::SerializedDepNodeIndex, rustc_data_structures::fingerprint::Fingerprint>, alloc::string::String>,(&mut serialize::opaque::Decoder)> (function.rs:231)
  #11: 0x59E3EEE: read_struct_field<serialize::opaque::Decoder,rustc_data_structures::indexed_vec::IndexVec<rustc::dep_graph::serialized::SerializedDepNodeIndex, rustc_data_structures::fingerprint::Fingerprint>,fn(&mut serialize::opaque::Decoder) -> core::result::Result<rustc_data_structures::indexed_vec::IndexVec<rustc::dep_graph::serialized::SerializedDepNodeIndex, rustc_data_structures::fingerprint::Fingerprint>, alloc::string::String>> (serialize.rs:221)
  #12: 0x59E3EEE: {{closure}}<serialize::opaque::Decoder> (serialized.rs:18)
  #13: 0x59E3EEE: read_struct<serialize::opaque::Decoder,rustc::dep_graph::serialized::SerializedDepGraph,closure> (serialize.rs:215)
  #14: 0x59E3EEE: decode<serialize::opaque::Decoder> (serialized.rs:12)
  #15: 0x59E3EEE: rustc_incremental::persist::load::load_dep_graph::{{closure}}::{{closure}} (load.rs:185)
  #16: 0x59931F6: time_ext<rustc_incremental::persist::load::LoadResult<(rustc::dep_graph::prev::PreviousDepGraph, std::collections::hash::map::HashMap<rustc::dep_graph::dep_node::WorkProductId, rustc::dep_graph::graph::WorkProduct, core::hash::BuildHasherDefault<rustc_hash::FxHasher>>)>,closure> (common.rs:151)
  #17: 0x59931F6: {{closure}} (load.rs:163)
  #18: 0x59931F6: std::sys_common::backtrace::__rust_begin_short_backtrace (backtrace.rs:77)
  #19: 0x59938B7: {{closure}}<closure,rustc_incremental::persist::load::LoadResult<(rustc::dep_graph::prev::PreviousDepGraph, std::collections::hash::map::HashMap<rustc::dep_graph::dep_node::WorkProductId, rustc::dep_graph::graph::WorkProduct, core::hash::BuildHasherDefault<rustc_hash::FxHasher>>)>> (mod.rs:470)
  #20: 0x59938B7: call_once<rustc_incremental::persist::load::LoadResult<(rustc::dep_graph::prev::PreviousDepGraph, std::collections::hash::map::HashMap<rustc::dep_graph::dep_node::WorkProductId, rustc::dep_graph::graph::WorkProduct, core::hash::BuildHasherDefault<rustc_hash::FxHasher>>)>,closure> (panic.rs:315)
  #21: 0x59938B7: std::panicking::try::do_call (panicking.rs:296)
}

-----------------------------------------------------------------------------
At t-gmax: 47,582,528 bytes (1.75%) in 1 blocks (0%), avg size 47,582,528 bytes
Reads:     47,582,528 bytes (0.03%, 109.47/Minstr), 1/byte
Writes:    47,582,528 bytes (0.09%, 109.47/Minstr), 1/byte
Allocated at {
  #1: 0x5F08EAC: alloc (alloc.rs:81)
  #2: 0x5F08EAC: alloc (alloc.rs:169)
  #3: 0x5F08EAC: reserve_internal<rustc_data_structures::fingerprint::Fingerprint,alloc::alloc::Global> (raw_vec.rs:668)
  #4: 0x5F08EAC: reserve<rustc_data_structures::fingerprint::Fingerprint,alloc::alloc::Global> (raw_vec.rs:491)
  #5: 0x5F08EAC: reserve<rustc_data_structures::fingerprint::Fingerprint> (vec.rs:457)
  #6: 0x5F08EAC: spec_extend<rustc_data_structures::fingerprint::Fingerprint,core::iter::adapters::Map<core::slice::Iter<rustc::dep_graph::graph::DepNodeData>, closure>> (vec.rs:1956)
  #7: 0x5F08EAC: from_iter<rustc_data_structures::fingerprint::Fingerprint,core::iter::adapters::Map<core::slice::Iter<rustc::dep_graph::graph::DepNodeData>, closure>> (vec.rs:1943)
  #8: 0x5F08EAC: from_iter<rustc_data_structures::fingerprint::Fingerprint,core::iter::adapters::Map<core::slice::Iter<rustc::dep_graph::graph::DepNodeData>, closure>> (vec.rs:1830)
  #9: 0x5F08EAC: from_iter<rustc::dep_graph::serialized::SerializedDepNodeIndex,rustc_data_structures::fingerprint::Fingerprint,core::iter::adapters::Map<core::slice::Iter<rustc::dep_graph::graph::DepNodeData>, closure>> (indexed_vec.rs:776)
  #10: 0x5F08EAC: collect<core::iter::adapters::Map<core::slice::Iter<rustc::dep_graph::graph::DepNodeData>, closure>,rustc_data_structures::indexed_vec::IndexVec<rustc::dep_graph::serialized::SerializedDepNodeIndex, rustc_data_structures::fingerprint::Fingerprint>> (iterator.rs:1464)
  #11: 0x5F08EAC: rustc::dep_graph::graph::DepGraph::serialize (graph.rs:508)
  #12: 0x59E6808: {{closure}} (save.rs:138)
  #13: 0x59E6808: time_ext<rustc::dep_graph::serialized::SerializedDepGraph,closure> (common.rs:151)
  #14: 0x59E6808: time<rustc::dep_graph::serialized::SerializedDepGraph,closure> (common.rs:145)
  #15: 0x59E6808: encode_dep_graph (save.rs:137)
  #16: 0x59E6808: rustc_incremental::persist::save::save_dep_graph::{{closure}}::{{closure}}::{{closure}}::{{closure}}::{{closure}} (save.rs:43)
  #17: 0x59E8982: time_ext<(),closure> (common.rs:151)
  #18: 0x59E8982: time<(),closure> (common.rs:145)
  #19: 0x59E8982: {{closure}} (save.rs:42)
  #20: 0x59E8982: save_in<closure> (save.rs:115)
  #21: 0x59E8982: rustc_incremental::persist::save::save_dep_graph::{{closure}}::{{closure}}::{{closure}} (save.rs:39)
}

-----------------------------------------------------------------------------
At t-gmax: 45,464,896 bytes (1.68%) in 39,171 blocks (0.81%), avg size 1,160.68 bytes
Reads:     83,735,696 bytes (0.04%, 192.65/Minstr), 0.9/byte
Writes:    85,435,688 bytes (0.16%, 196.57/Minstr), 0.92/byte
Allocated at {
  ^1: 0x5E9BC8E: alloc (alloc.rs:81)
  ^2: 0x5E9BC8E: alloc (alloc.rs:169)
  ^3: 0x5E9BC8E: allocate_in<rustc::dep_graph::graph::DepNodeIndex,alloc::alloc::Global> (raw_vec.rs:95)
  ^4: 0x5E9BC8E: with_capacity<rustc::dep_graph::graph::DepNodeIndex> (raw_vec.rs:139)
  ^5: 0x5E9BC8E: with_capacity<rustc::dep_graph::graph::DepNodeIndex> (vec.rs:355)
  ^6: 0x5E9BC8E: grow<[rustc::dep_graph::graph::DepNodeIndex; 8]> (lib.rs:659)
  ^7: 0x5E9BC8E: reserve<[rustc::dep_graph::graph::DepNodeIndex; 8]> (lib.rs:692)
  ^8: 0x5E9BC8E: smallvec::SmallVec<A>::push (lib.rs:621)
  #9: 0x5F0B7BA: {{closure}} (graph.rs:1093)
  #10: 0x5F0B7BA: with_context_opt<closure,()> (context.rs:1944)
  #11: 0x5F0B7BA: rustc::dep_graph::graph::DepGraphData::read_index (graph.rs:1085)
}

-----------------------------------------------------------------------------
At t-gmax: 40,248,640 bytes (1.48%) in 214,572 blocks (4.41%), avg size 187.58 bytes
Reads:     86,678,424 bytes (0.05%, 199.42/Minstr), 0.72/byte
Writes:    116,440,320 bytes (0.21%, 267.9/Minstr), 0.97/byte
Allocated at {
  ^1: 0x5E9BC8E: alloc (alloc.rs:81)
  ^2: 0x5E9BC8E: alloc (alloc.rs:169)
  ^3: 0x5E9BC8E: allocate_in<rustc::dep_graph::graph::DepNodeIndex,alloc::alloc::Global> (raw_vec.rs:95)
  ^4: 0x5E9BC8E: with_capacity<rustc::dep_graph::graph::DepNodeIndex> (raw_vec.rs:139)
  ^5: 0x5E9BC8E: with_capacity<rustc::dep_graph::graph::DepNodeIndex> (vec.rs:355)
  ^6: 0x5E9BC8E: grow<[rustc::dep_graph::graph::DepNodeIndex; 8]> (lib.rs:659)
  ^7: 0x5E9BC8E: reserve<[rustc::dep_graph::graph::DepNodeIndex; 8]> (lib.rs:692)
  ^8: 0x5E9BC8E: smallvec::SmallVec<A>::push (lib.rs:621)
  #9: 0x5F099DE: rustc::dep_graph::graph::DepGraph::try_mark_previous_green (graph.rs:641)
}
@Zoxc

This comment has been minimized.

Copy link
Contributor Author

commented Jul 24, 2019

I did manage to split part of this out as #62038, but that won't make any progress until @michaelwoerister returns.

@hdhoang

This comment has been minimized.

Copy link
Contributor

commented Aug 2, 2019

This PR appears to be blocked on #62038 as per the author's comment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
8 participants
You can’t perform that action at this time.