New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Compile the libstd we distribute with -Ccodegen-unit=1 #55264

Merged
merged 3 commits into from Oct 26, 2018

Conversation

Projects
None yet
6 participants
@michaelwoerister
Contributor

michaelwoerister commented Oct 22, 2018

This PR

  • adds the single-codegen-unit-std option to config.toml which allows for setting the CGU count for libstd and libtest independently of the one for the rest of the compiler, and
  • sets the new option to true for all dist jobs in CI.

Fixes #54872.

r? @Mark-Simulacrum
cc @rust-lang/release

@Mark-Simulacrum

This comment has been minimized.

Member

Mark-Simulacrum commented Oct 22, 2018

Hm, I'd prefer that we instead have this as codegen-unit-std=1 instead of single-codegen-unit-std=true; with that change, though, I think we should be able to use this.

@bors try to get at least some amount of timing information

@michaelwoerister

This comment has been minimized.

Contributor

michaelwoerister commented Oct 22, 2018

I'd prefer that we instead have this as codegen-unit-std=1 instead of single-codegen-unit-std=true

Sure, no problem.

@nikic

This comment has been minimized.

Contributor

nikic commented Oct 22, 2018

Would it be possible to add the --set rust.single-codegen-unit-std=true flag for all DEPLOY=1 jobs in run.sh, instead of specifying it in each docker file? (This is probably a stupid question, I'm not particularly familiar with the build system.)

@Mark-Simulacrum

This comment has been minimized.

Member

Mark-Simulacrum commented Oct 22, 2018

Yeah, that's probably the better way to do it. I think it should go somewhere here: https://github.com/rust-lang/rust/blob/master/src/ci/run.sh#L56

@bors

This comment has been minimized.

Contributor

bors commented Oct 22, 2018

⌛️ Trying commit c4ca23a with merge 0c6e128...

bors added a commit that referenced this pull request Oct 22, 2018

Auto merge of #55264 - michaelwoerister:single-cgu-std, r=<try>
Compile the libstd we distribute with -Ccodegen-unit=1

This PR
 - adds the `single-codegen-unit-std` option to `config.toml` which allows for setting the CGU count for `libstd` and `libtest` independently of the one for the rest of the compiler, and
 - sets the new option to `true` for all dist jobs in CI.

Fixes #54872.

r? @Mark-Simulacrum
cc @rust-lang/release
@rust-highfive

This comment has been minimized.

Collaborator

rust-highfive commented Oct 22, 2018

Your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
travis_fold:end:services

travis_fold:start:git.checkout
travis_time:start:00e7ba4e
$ git clone --depth=2 --branch=try https://github.com/rust-lang/rust.git rust-lang/rust
---
Resolving deltas: 100% (7937/7937), done.
travis_time:end:00e7ba4e:start=1540241159474675330,finish=1540241165516488837,duration=6041813507
$ cd rust-lang/rust
$ git checkout -qf 0c6e1283b0a9fe23e54018a5dcdd3ec6d84d1896
fatal: reference is not a tree: 0c6e1283b0a9fe23e54018a5dcdd3ec6d84d1896
The command "git checkout -qf 0c6e1283b0a9fe23e54018a5dcdd3ec6d84d1896" failed and exited with 128 during .
Your build has been stopped.

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

2 similar comments
@rust-highfive

This comment has been minimized.

Collaborator

rust-highfive commented Oct 22, 2018

Your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
travis_fold:end:services

travis_fold:start:git.checkout
travis_time:start:00e7ba4e
$ git clone --depth=2 --branch=try https://github.com/rust-lang/rust.git rust-lang/rust
---
Resolving deltas: 100% (7937/7937), done.
travis_time:end:00e7ba4e:start=1540241159474675330,finish=1540241165516488837,duration=6041813507
$ cd rust-lang/rust
$ git checkout -qf 0c6e1283b0a9fe23e54018a5dcdd3ec6d84d1896
fatal: reference is not a tree: 0c6e1283b0a9fe23e54018a5dcdd3ec6d84d1896
The command "git checkout -qf 0c6e1283b0a9fe23e54018a5dcdd3ec6d84d1896" failed and exited with 128 during .
Your build has been stopped.

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

@rust-highfive

This comment has been minimized.

Collaborator

rust-highfive commented Oct 22, 2018

Your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
travis_fold:end:services

travis_fold:start:git.checkout
travis_time:start:00e7ba4e
$ git clone --depth=2 --branch=try https://github.com/rust-lang/rust.git rust-lang/rust
---
Resolving deltas: 100% (7937/7937), done.
travis_time:end:00e7ba4e:start=1540241159474675330,finish=1540241165516488837,duration=6041813507
$ cd rust-lang/rust
$ git checkout -qf 0c6e1283b0a9fe23e54018a5dcdd3ec6d84d1896
fatal: reference is not a tree: 0c6e1283b0a9fe23e54018a5dcdd3ec6d84d1896
The command "git checkout -qf 0c6e1283b0a9fe23e54018a5dcdd3ec6d84d1896" failed and exited with 128 during .
Your build has been stopped.

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

@bors

This comment has been minimized.

Contributor

bors commented Oct 22, 2018

💔 Test failed - status-travis

@rust-highfive

This comment has been minimized.

Collaborator

rust-highfive commented Oct 22, 2018

Your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
travis_fold:end:services

travis_fold:start:git.checkout
travis_time:start:00e7ba4e
$ git clone --depth=2 --branch=try https://github.com/rust-lang/rust.git rust-lang/rust
---
Resolving deltas: 100% (7937/7937), done.
travis_time:end:00e7ba4e:start=1540241159474675330,finish=1540241165516488837,duration=6041813507
$ cd rust-lang/rust
$ git checkout -qf 0c6e1283b0a9fe23e54018a5dcdd3ec6d84d1896
fatal: reference is not a tree: 0c6e1283b0a9fe23e54018a5dcdd3ec6d84d1896
The command "git checkout -qf 0c6e1283b0a9fe23e54018a5dcdd3ec6d84d1896" failed and exited with 128 during .
Your build has been stopped.

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

@michaelwoerister

This comment has been minimized.

Contributor

michaelwoerister commented Oct 22, 2018

Thanks for pointing that out, @nikic. That's a much better way of setting the flag.

@michaelwoerister

This comment has been minimized.

Contributor

michaelwoerister commented Oct 23, 2018

@bors try

@bors

This comment has been minimized.

Contributor

bors commented Oct 23, 2018

⌛️ Trying commit 4545b3e with merge 00b7129...

bors added a commit that referenced this pull request Oct 23, 2018

Auto merge of #55264 - michaelwoerister:single-cgu-std, r=<try>
Compile the libstd we distribute with -Ccodegen-unit=1

This PR
 - adds the `single-codegen-unit-std` option to `config.toml` which allows for setting the CGU count for `libstd` and `libtest` independently of the one for the rest of the compiler, and
 - sets the new option to `true` for all dist jobs in CI.

Fixes #54872.

r? @Mark-Simulacrum
cc @rust-lang/release
@bors

This comment has been minimized.

Contributor

bors commented Oct 23, 2018

💔 Test failed - status-travis

@rust-highfive

This comment has been minimized.

Collaborator

rust-highfive commented Oct 23, 2018

The job dist-x86_64-linux of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
travis_fold:end:services

travis_fold:start:git.checkout
travis_time:start:0a072a42
$ git clone --depth=2 --branch=try https://github.com/rust-lang/rust.git rust-lang/rust
---
[00:03:53]    Compiling toml v0.4.6
[00:03:53]    Compiling serde_json v1.0.31
[00:04:01]    Compiling bootstrap v0.0.0 (/checkout/src/bootstrap)
[00:04:34]     Finished dev [unoptimized] target(s) in 1m 06s
[00:04:34] failed to parse TOML configuration 'config.toml': invalid type: string "1", expected u32 for key `rust.codegen-units-std`
[00:04:34] Build completed unsuccessfully in 0:01:28
[00:04:34] make: *** [prepare] Error 1
[00:04:35] Command failed. Attempt 2/5:
[00:04:35]     Finished dev [unoptimized] target(s) in 0.26s
[00:04:35]     Finished dev [unoptimized] target(s) in 0.26s
[00:04:35] failed to parse TOML configuration 'config.toml': invalid type: string "1", expected u32 for key `rust.codegen-units-std`
[00:04:35] Build completed unsuccessfully in 0:00:00
[00:04:35] make: *** [prepare] Error 1
[00:04:37] Command failed. Attempt 3/5:
[00:04:37]     Finished dev [unoptimized] target(s) in 0.26s
[00:04:37]     Finished dev [unoptimized] target(s) in 0.26s
[00:04:37] failed to parse TOML configuration 'config.toml': invalid type: string "1", expected u32 for key `rust.codegen-units-std`
[00:04:37] Build completed unsuccessfully in 0:00:00
[00:04:37] make: *** [prepare] Error 1
[00:04:40] Command failed. Attempt 4/5:
[00:04:41]     Finished dev [unoptimized] target(s) in 0.26s
[00:04:41]     Finished dev [unoptimized] target(s) in 0.26s
[00:04:41] failed to parse TOML configuration 'config.toml': invalid type: string "1", expected u32 for key `rust.codegen-units-std`
[00:04:41] Build completed unsuccessfully in 0:00:00
[00:04:41] make: *** [prepare] Error 1
[00:04:45] Command failed. Attempt 5/5:
[00:04:45]     Finished dev [unoptimized] target(s) in 0.26s
[00:04:45]     Finished dev [unoptimized] target(s) in 0.26s
[00:04:45] failed to parse TOML configuration 'config.toml': invalid type: string "1", expected u32 for key `rust.codegen-units-std`
[00:04:45] Build completed unsuccessfully in 0:00:00
[00:04:45] make: *** [prepare] Error 1
[00:04:45] The command has failed after 5 attempts.
travis_time:end:0c1d07fc:start=1540297112928164652,finish=1540297398733819004,duration=285805654352
---
travis_time:end:14670954:start=1540297399169557079,finish=1540297399177338390,duration=7781311
travis_fold:end:after_failure.3
travis_fold:start:after_failure.4
travis_time:start:2adff231
$ ln -s . checkout && for CORE in obj/cores/core.*; do EXE=$(echo $CORE | sed 's|obj/cores/core\.[0-9]*\.!checkout!\(.*\)|\1|;y|!|/|'); if [ -f "$EXE" ]; then printf travis_fold":start:crashlog\n\033[31;1m%s\033[0m\n" "$CORE"; gdb --batch -q -c "$CORE" "$EXE" -iex 'set auto-load off' -iex 'dir src/' -iex 'set sysroot .' -ex bt -ex q; echo travis_fold":"end:crashlog; fi; done || true
travis_fold:end:after_failure.4
travis_fold:start:after_failure.5
travis_time:start:07e41598
travis_time:start:07e41598
$ cat ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers || true
cat: ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers: No such file or directory
travis_fold:end:after_failure.5
travis_fold:start:after_failure.6
travis_time:start:02dd237c
$ dmesg | grep -i kill

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

@michaelwoerister

This comment has been minimized.

Contributor

michaelwoerister commented Oct 23, 2018

@bors try

@bors

This comment has been minimized.

Contributor

bors commented Oct 23, 2018

⌛️ Trying commit 03551e1 with merge d803782...

bors added a commit that referenced this pull request Oct 23, 2018

Auto merge of #55264 - michaelwoerister:single-cgu-std, r=<try>
Compile the libstd we distribute with -Ccodegen-unit=1

This PR
 - adds the `single-codegen-unit-std` option to `config.toml` which allows for setting the CGU count for `libstd` and `libtest` independently of the one for the rest of the compiler, and
 - sets the new option to `true` for all dist jobs in CI.

Fixes #54872.

r? @Mark-Simulacrum
cc @rust-lang/release
@bors

This comment has been minimized.

Contributor

bors commented Oct 23, 2018

☀️ Test successful - status-travis
State: approved= try=True

@michaelwoerister

This comment has been minimized.

Contributor

michaelwoerister commented Oct 23, 2018

@Mark-Simulacrum: Changes applied and try build succeeds :)

@Mark-Simulacrum

This comment has been minimized.

Member

Mark-Simulacrum commented Oct 23, 2018

@rust-timer build d803782

If this shows performance wins then we might want to enable this on all builds, not just dist builds, since it could make the compiler fast(er) sufficiently to make the wins on test compilations significant.

@rust-timer

This comment has been minimized.

rust-timer commented Oct 23, 2018

Success: Queued d803782 with parent d74b402, comparison URL.

@rust-timer

This comment has been minimized.

rust-timer commented Oct 24, 2018

Finished benchmarking try commit d803782

@michaelwoerister

This comment has been minimized.

Contributor

michaelwoerister commented Oct 24, 2018

Performance looks pretty good!

@Mark-Simulacrum

This comment has been minimized.

Member

Mark-Simulacrum commented Oct 24, 2018

Can you make the change to set the flag on all CI builds vs. just dist builds? I would expect that to be mildly faster with tests so maybe worth it? r=me with that

@nikic

This comment has been minimized.

Contributor

nikic commented Oct 24, 2018

Interestingly, there's also a few percent of max-rss improvement. I wonder where that comes from, wouldn't have expected codegen units to affect this. Is LLVM optimizing away allocations here?

@michaelwoerister

This comment has been minimized.

Contributor

michaelwoerister commented Oct 25, 2018

@bors r=Mark-Simulacrum

@bors

This comment has been minimized.

Contributor

bors commented Oct 25, 2018

📌 Commit c4aa361 has been approved by Mark-Simulacrum

@michaelwoerister

This comment has been minimized.

Contributor

michaelwoerister commented Oct 26, 2018

Rebasing & re-approving in the hope that that will make the PR show up in the bors queue properly.

@bors r=simulacrum

@bors

This comment has been minimized.

Contributor

bors commented Oct 26, 2018

📌 Commit 5dedf0c has been approved by simulacrum

kennytm added a commit to kennytm/rust that referenced this pull request Oct 26, 2018

Rollup merge of rust-lang#55264 - michaelwoerister:single-cgu-std, r=…
…simulacrum

Compile the libstd we distribute with -Ccodegen-unit=1

This PR
 - adds the `single-codegen-unit-std` option to `config.toml` which allows for setting the CGU count for `libstd` and `libtest` independently of the one for the rest of the compiler, and
 - sets the new option to `true` for all dist jobs in CI.

Fixes rust-lang#54872.

bors added a commit that referenced this pull request Oct 26, 2018

Auto merge of #55382 - kennytm:rollup, r=kennytm
Rollup of 21 pull requests

Successful merges:

 - #54816 (Don't try to promote already promoted out temporaries)
 - #54824 (Cleanup rustdoc tests with `@!has` and `@!matches`)
 - #54921 (Add line numbers option to rustdoc)
 - #55167 (Add a "cheap" mode for `compute_missing_ctors`.)
 - #55258 (Fix Rustdoc ICE when checking blanket impls)
 - #55264 (Compile the libstd we distribute with -Ccodegen-unit=1)
 - #55271 (Unimplement ExactSizeIterator for MIR traversing iterators)
 - #55292 (Macro diagnostics tweaks)
 - #55298 (Point at macro definition when no rules expect token)
 - #55301 (List allowed tokens after macro fragments)
 - #55302 (Extend the impl_stable_hash_for! macro for miri.)
 - #55325 (Fix link to macros chapter)
 - #55343 (rustbuild: fix remap-debuginfo when building a release)
 - #55346 (Shrink `Statement`.)
 - #55358 (Remove redundant clone (2))
 - #55370 (Update mailmap for estebank)
 - #55375 (Typo fixes in configure_cmake comments)
 - #55378 (rustbuild: use configured linker to build boostrap)
 - #55379 (validity: assert that unions are non-empty)
 - #55383 (Use `SmallVec` for the queue in `coerce_unsized`.)
 - #55391 (bootstrap: clean up a few clippy findings)

@bors bors merged commit 5dedf0c into rust-lang:master Oct 26, 2018

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment