Compile the libstd we distribute with -Ccodegen-unit=1 #55264

michaelwoerister · 2018-10-22T15:06:48Z

This PR

adds the single-codegen-unit-std option to config.toml which allows for setting the CGU count for libstd and libtest independently of the one for the rest of the compiler, and
sets the new option to true for all dist jobs in CI.

r? @Mark-Simulacrum
cc @rust-lang/release

Mark-Simulacrum · 2018-10-22T16:25:42Z

Hm, I'd prefer that we instead have this as codegen-unit-std=1 instead of single-codegen-unit-std=true; with that change, though, I think we should be able to use this.

@bors try to get at least some amount of timing information

michaelwoerister · 2018-10-22T17:00:02Z

I'd prefer that we instead have this as codegen-unit-std=1 instead of single-codegen-unit-std=true

Sure, no problem.

nikic · 2018-10-22T17:05:48Z

Would it be possible to add the --set rust.single-codegen-unit-std=true flag for all DEPLOY=1 jobs in run.sh, instead of specifying it in each docker file? (This is probably a stupid question, I'm not particularly familiar with the build system.)

Mark-Simulacrum · 2018-10-22T17:21:44Z

Yeah, that's probably the better way to do it. I think it should go somewhere here: https://github.com/rust-lang/rust/blob/master/src/ci/run.sh#L56

bors · 2018-10-22T19:35:43Z

⌛ Trying commit c4ca23afbf1138f094fb813e7fe8eaaec73ef59c with merge 0c6e1283b0a9fe23e54018a5dcdd3ec6d84d1896...

rust-highfive · 2018-10-22T21:29:20Z

Your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

travis_fold:end:services

travis_fold:start:git.checkout
travis_time:start:00e7ba4e
$ git clone --depth=2 --branch=try https://github.com/rust-lang/rust.git rust-lang/rust
---
Resolving deltas: 100% (7937/7937), done.
travis_time:end:00e7ba4e:start=1540241159474675330,finish=1540241165516488837,duration=6041813507
$ cd rust-lang/rust
$ git checkout -qf 0c6e1283b0a9fe23e54018a5dcdd3ec6d84d1896
fatal: reference is not a tree: 0c6e1283b0a9fe23e54018a5dcdd3ec6d84d1896
The command "git checkout -qf 0c6e1283b0a9fe23e54018a5dcdd3ec6d84d1896" failed and exited with 128 during .
Your build has been stopped.

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

rust-highfive · 2018-10-22T21:29:21Z

Your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

travis_fold:end:services

travis_fold:start:git.checkout
travis_time:start:00e7ba4e
$ git clone --depth=2 --branch=try https://github.com/rust-lang/rust.git rust-lang/rust
---
Resolving deltas: 100% (7937/7937), done.
travis_time:end:00e7ba4e:start=1540241159474675330,finish=1540241165516488837,duration=6041813507
$ cd rust-lang/rust
$ git checkout -qf 0c6e1283b0a9fe23e54018a5dcdd3ec6d84d1896
fatal: reference is not a tree: 0c6e1283b0a9fe23e54018a5dcdd3ec6d84d1896
The command "git checkout -qf 0c6e1283b0a9fe23e54018a5dcdd3ec6d84d1896" failed and exited with 128 during .
Your build has been stopped.

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

rust-highfive · 2018-10-22T21:29:42Z

Your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

travis_fold:end:services

travis_fold:start:git.checkout
travis_time:start:00e7ba4e
$ git clone --depth=2 --branch=try https://github.com/rust-lang/rust.git rust-lang/rust
---
Resolving deltas: 100% (7937/7937), done.
travis_time:end:00e7ba4e:start=1540241159474675330,finish=1540241165516488837,duration=6041813507
$ cd rust-lang/rust
$ git checkout -qf 0c6e1283b0a9fe23e54018a5dcdd3ec6d84d1896
fatal: reference is not a tree: 0c6e1283b0a9fe23e54018a5dcdd3ec6d84d1896
The command "git checkout -qf 0c6e1283b0a9fe23e54018a5dcdd3ec6d84d1896" failed and exited with 128 during .
Your build has been stopped.

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

bors · 2018-10-22T21:29:55Z

💔 Test failed - status-travis

rust-highfive · 2018-10-22T21:29:55Z

Your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

travis_fold:end:services

travis_fold:start:git.checkout
travis_time:start:00e7ba4e
$ git clone --depth=2 --branch=try https://github.com/rust-lang/rust.git rust-lang/rust
---
Resolving deltas: 100% (7937/7937), done.
travis_time:end:00e7ba4e:start=1540241159474675330,finish=1540241165516488837,duration=6041813507
$ cd rust-lang/rust
$ git checkout -qf 0c6e1283b0a9fe23e54018a5dcdd3ec6d84d1896
fatal: reference is not a tree: 0c6e1283b0a9fe23e54018a5dcdd3ec6d84d1896
The command "git checkout -qf 0c6e1283b0a9fe23e54018a5dcdd3ec6d84d1896" failed and exited with 128 during .
Your build has been stopped.

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

michaelwoerister · 2018-10-22T22:55:07Z

Thanks for pointing that out, @nikic. That's a much better way of setting the flag.

michaelwoerister · 2018-10-23T12:16:41Z

@bors try

bors · 2018-10-23T12:16:54Z

⌛ Trying commit 4545b3e with merge 00b7129...

@Mark-Simulacrum

Compile the libstd we distribute with -Ccodegen-unit=1 This PR - adds the `single-codegen-unit-std` option to `config.toml` which allows for setting the CGU count for `libstd` and `libtest` independently of the one for the rest of the compiler, and - sets the new option to `true` for all dist jobs in CI. Fixes #54872. r? @Mark-Simulacrum cc @rust-lang/release

bors · 2018-10-23T12:30:14Z

💔 Test failed - status-travis

rust-highfive · 2018-10-23T12:30:15Z

The job dist-x86_64-linux of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

travis_fold:end:services

travis_fold:start:git.checkout
travis_time:start:0a072a42
$ git clone --depth=2 --branch=try https://github.com/rust-lang/rust.git rust-lang/rust
---
[00:03:53]    Compiling toml v0.4.6
[00:03:53]    Compiling serde_json v1.0.31
[00:04:01]    Compiling bootstrap v0.0.0 (/checkout/src/bootstrap)
[00:04:34]     Finished dev [unoptimized] target(s) in 1m 06s
[00:04:34] failed to parse TOML configuration 'config.toml': invalid type: string "1", expected u32 for key `rust.codegen-units-std`
[00:04:34] Build completed unsuccessfully in 0:01:28
[00:04:34] make: *** [prepare] Error 1
[00:04:35] Command failed. Attempt 2/5:
[00:04:35]     Finished dev [unoptimized] target(s) in 0.26s
[00:04:35]     Finished dev [unoptimized] target(s) in 0.26s
[00:04:35] failed to parse TOML configuration 'config.toml': invalid type: string "1", expected u32 for key `rust.codegen-units-std`
[00:04:35] Build completed unsuccessfully in 0:00:00
[00:04:35] make: *** [prepare] Error 1
[00:04:37] Command failed. Attempt 3/5:
[00:04:37]     Finished dev [unoptimized] target(s) in 0.26s
[00:04:37]     Finished dev [unoptimized] target(s) in 0.26s
[00:04:37] failed to parse TOML configuration 'config.toml': invalid type: string "1", expected u32 for key `rust.codegen-units-std`
[00:04:37] Build completed unsuccessfully in 0:00:00
[00:04:37] make: *** [prepare] Error 1
[00:04:40] Command failed. Attempt 4/5:
[00:04:41]     Finished dev [unoptimized] target(s) in 0.26s
[00:04:41]     Finished dev [unoptimized] target(s) in 0.26s
[00:04:41] failed to parse TOML configuration 'config.toml': invalid type: string "1", expected u32 for key `rust.codegen-units-std`
[00:04:41] Build completed unsuccessfully in 0:00:00
[00:04:41] make: *** [prepare] Error 1
[00:04:45] Command failed. Attempt 5/5:
[00:04:45]     Finished dev [unoptimized] target(s) in 0.26s
[00:04:45]     Finished dev [unoptimized] target(s) in 0.26s
[00:04:45] failed to parse TOML configuration 'config.toml': invalid type: string "1", expected u32 for key `rust.codegen-units-std`
[00:04:45] Build completed unsuccessfully in 0:00:00
[00:04:45] make: *** [prepare] Error 1
[00:04:45] The command has failed after 5 attempts.
travis_time:end:0c1d07fc:start=1540297112928164652,finish=1540297398733819004,duration=285805654352
---
travis_time:end:14670954:start=1540297399169557079,finish=1540297399177338390,duration=7781311
travis_fold:end:after_failure.3
travis_fold:start:after_failure.4
travis_time:start:2adff231
$ ln -s . checkout && for CORE in obj/cores/core.*; do EXE=$(echo $CORE | sed 's|obj/cores/core\.[0-9]*\.!checkout!\(.*\)|\1|;y|!|/|'); if [ -f "$EXE" ]; then printf travis_fold":start:crashlog\n\033[31;1m%s\033[0m\n" "$CORE"; gdb --batch -q -c "$CORE" "$EXE" -iex 'set auto-load off' -iex 'dir src/' -iex 'set sysroot .' -ex bt -ex q; echo travis_fold":"end:crashlog; fi; done || true
travis_fold:end:after_failure.4
travis_fold:start:after_failure.5
travis_time:start:07e41598
travis_time:start:07e41598
$ cat ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers || true
cat: ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers: No such file or directory
travis_fold:end:after_failure.5
travis_fold:start:after_failure.6
travis_time:start:02dd237c
$ dmesg | grep -i kill

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

michaelwoerister · 2018-10-23T13:37:50Z

@bors try

bors · 2018-10-23T13:38:03Z

⌛ Trying commit 03551e1 with merge d803782...

@Mark-Simulacrum

Compile the libstd we distribute with -Ccodegen-unit=1 This PR - adds the `single-codegen-unit-std` option to `config.toml` which allows for setting the CGU count for `libstd` and `libtest` independently of the one for the rest of the compiler, and - sets the new option to `true` for all dist jobs in CI. Fixes #54872. r? @Mark-Simulacrum cc @rust-lang/release

bors · 2018-10-23T16:02:00Z

☀️ Test successful - status-travis
State: approved= try=True

michaelwoerister · 2018-10-23T16:20:22Z

@Mark-Simulacrum: Changes applied and try build succeeds :)

Mark-Simulacrum · 2018-10-23T16:36:38Z

@rust-timer build d803782

If this shows performance wins then we might want to enable this on all builds, not just dist builds, since it could make the compiler fast(er) sufficiently to make the wins on test compilations significant.

rust-timer · 2018-10-23T16:36:39Z

Success: Queued d803782 with parent d74b402, comparison URL.

rust-timer · 2018-10-24T09:05:47Z

Finished benchmarking try commit d803782

michaelwoerister · 2018-10-24T11:25:54Z

Performance looks pretty good!

Mark-Simulacrum · 2018-10-24T14:12:12Z

Can you make the change to set the flag on all CI builds vs. just dist builds? I would expect that to be mildly faster with tests so maybe worth it? r=me with that

nikic · 2018-10-24T14:19:04Z

Interestingly, there's also a few percent of max-rss improvement. I wonder where that comes from, wouldn't have expected codegen units to affect this. Is LLVM optimizing away allocations here?

michaelwoerister · 2018-10-25T08:48:20Z

@bors r=Mark-Simulacrum

bors · 2018-10-25T08:48:21Z

📌 Commit c4aa361e3d0771b18ad2e832c4f28eb345acd686 has been approved by Mark-Simulacrum

michaelwoerister · 2018-10-26T13:08:58Z

Rebasing & re-approving in the hope that that will make the PR show up in the bors queue properly.

@bors r=simulacrum

bors · 2018-10-26T13:08:59Z

📌 Commit 5dedf0c has been approved by simulacrum

…simulacrum Compile the libstd we distribute with -Ccodegen-unit=1 This PR - adds the `single-codegen-unit-std` option to `config.toml` which allows for setting the CGU count for `libstd` and `libtest` independently of the one for the rest of the compiler, and - sets the new option to `true` for all dist jobs in CI. Fixes rust-lang#54872.

Rollup of 21 pull requests Successful merges: - #54816 (Don't try to promote already promoted out temporaries) - #54824 (Cleanup rustdoc tests with `@!has` and `@!matches`) - #54921 (Add line numbers option to rustdoc) - #55167 (Add a "cheap" mode for `compute_missing_ctors`.) - #55258 (Fix Rustdoc ICE when checking blanket impls) - #55264 (Compile the libstd we distribute with -Ccodegen-unit=1) - #55271 (Unimplement ExactSizeIterator for MIR traversing iterators) - #55292 (Macro diagnostics tweaks) - #55298 (Point at macro definition when no rules expect token) - #55301 (List allowed tokens after macro fragments) - #55302 (Extend the impl_stable_hash_for! macro for miri.) - #55325 (Fix link to macros chapter) - #55343 (rustbuild: fix remap-debuginfo when building a release) - #55346 (Shrink `Statement`.) - #55358 (Remove redundant clone (2)) - #55370 (Update mailmap for estebank) - #55375 (Typo fixes in configure_cmake comments) - #55378 (rustbuild: use configured linker to build boostrap) - #55379 (validity: assert that unions are non-empty) - #55383 (Use `SmallVec` for the queue in `coerce_unsized`.) - #55391 (bootstrap: clean up a few clippy findings)

Upstream started this in rust-lang/rust#55264, which was released in Rust 1.32.

rust-highfive assigned Mark-Simulacrum Oct 22, 2018

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Oct 22, 2018

bors added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Oct 22, 2018

michaelwoerister force-pushed the single-cgu-std branch from c4ca23a to 4545b3e Compare October 23, 2018 12:15

michaelwoerister force-pushed the single-cgu-std branch from 03551e1 to c4aa361 Compare October 25, 2018 08:45

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Oct 25, 2018

michaelwoerister added 3 commits October 26, 2018 15:07

bootstrap: Allow for build libstd to have its own codegen-unit setting.

b8f977a

Make configure.py handle numeric arguments for --set a little better.

9e51b57

CI: Set codegen-units-std=1 for dist builds.

5dedf0c

michaelwoerister force-pushed the single-cgu-std branch from c4aa361 to 5dedf0c Compare October 26, 2018 13:07

kennytm mentioned this pull request Oct 26, 2018

Rollup of 21 pull requests #55382

Merged

bors merged commit 5dedf0c into rust-lang:master Oct 26, 2018

pietroalbini mentioned this pull request Oct 29, 2018

[beta]: Prepare the 1.31.0 beta release #55405

Merged

cuviper mentioned this pull request May 23, 2019

Failed to build rpm-ostree on ppc64le: multiple definition of `__fixdfdi' #61063

Open

im-0 pushed a commit to im-0/fedora-rpm.rust that referenced this pull request Dec 9, 2019

Set rust.codegen-units-std=1

039aee8

Upstream started this in rust-lang/rust#55264, which was released in Rust 1.32.

ian-h-chamberlain mentioned this pull request May 1, 2022

Add failing example with regex rust3ds/ctru-rs#56

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Compile the libstd we distribute with -Ccodegen-unit=1 #55264

Compile the libstd we distribute with -Ccodegen-unit=1 #55264

michaelwoerister commented Oct 22, 2018

Mark-Simulacrum commented Oct 22, 2018

michaelwoerister commented Oct 22, 2018

nikic commented Oct 22, 2018

Mark-Simulacrum commented Oct 22, 2018

bors commented Oct 22, 2018

rust-highfive commented Oct 22, 2018

rust-highfive commented Oct 22, 2018

rust-highfive commented Oct 22, 2018

bors commented Oct 22, 2018

rust-highfive commented Oct 22, 2018

michaelwoerister commented Oct 22, 2018

michaelwoerister commented Oct 23, 2018

bors commented Oct 23, 2018

bors commented Oct 23, 2018

rust-highfive commented Oct 23, 2018

michaelwoerister commented Oct 23, 2018

bors commented Oct 23, 2018

bors commented Oct 23, 2018

michaelwoerister commented Oct 23, 2018

Mark-Simulacrum commented Oct 23, 2018

rust-timer commented Oct 23, 2018

rust-timer commented Oct 24, 2018

michaelwoerister commented Oct 24, 2018

Mark-Simulacrum commented Oct 24, 2018

nikic commented Oct 24, 2018

michaelwoerister commented Oct 25, 2018

bors commented Oct 25, 2018

michaelwoerister commented Oct 26, 2018

bors commented Oct 26, 2018

Compile the libstd we distribute with -Ccodegen-unit=1 #55264

Compile the libstd we distribute with -Ccodegen-unit=1 #55264

Conversation

michaelwoerister commented Oct 22, 2018

Mark-Simulacrum commented Oct 22, 2018

michaelwoerister commented Oct 22, 2018

nikic commented Oct 22, 2018

Mark-Simulacrum commented Oct 22, 2018

bors commented Oct 22, 2018

rust-highfive commented Oct 22, 2018

rust-highfive commented Oct 22, 2018

rust-highfive commented Oct 22, 2018

bors commented Oct 22, 2018

rust-highfive commented Oct 22, 2018

michaelwoerister commented Oct 22, 2018

michaelwoerister commented Oct 23, 2018

bors commented Oct 23, 2018

bors commented Oct 23, 2018

rust-highfive commented Oct 23, 2018

michaelwoerister commented Oct 23, 2018

bors commented Oct 23, 2018

bors commented Oct 23, 2018

michaelwoerister commented Oct 23, 2018

Mark-Simulacrum commented Oct 23, 2018

rust-timer commented Oct 23, 2018

rust-timer commented Oct 24, 2018

michaelwoerister commented Oct 24, 2018

Mark-Simulacrum commented Oct 24, 2018

nikic commented Oct 24, 2018

michaelwoerister commented Oct 25, 2018

bors commented Oct 25, 2018

michaelwoerister commented Oct 26, 2018

bors commented Oct 26, 2018