Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stabilize support for Profile-guided Optimization #61268

Merged
merged 2 commits into from Jul 2, 2019

Conversation

@michaelwoerister
Copy link
Contributor

commented May 28, 2019

This PR makes profile-guided optimization available via the -C profile-generate / -C profile-use pair of commandline flags and adds end-user documentation for the feature to the rustc book. The PR thus ticks the last two remaining checkboxes of the stabilization tracking issue.

From the tracking issue:

Profile-guided optimization (PGO) is a common optimization technique for ahead-of-time compilers. It works by collecting data about a program's typical execution (e.g. probability of branches taken, typical runtime values of variables, etc) and then uses this information during program optimization for things like inlining decisions, machine code layout, or indirect call promotion.

If you are curious about how this can be used, there is a rendered version of the documentation this PR adds available here.

r? @alexcrichton
cc @rust-lang/compiler

@michaelwoerister

This comment has been minimized.

Copy link
Contributor Author

commented May 28, 2019

Dear @rust-lang/compiler, please add your check mark below.
@rfcbot fcp merge

@rfcbot

This comment has been minimized.

Copy link

commented May 28, 2019

Team member @michaelwoerister has proposed to merge this. The next step is review by the rest of the tagged team members:

No concerns currently listed.

Once a majority of reviewers approve (and at most 2 approvals are outstanding), this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up!

See this document for info about what commands tagged team members can give me.

src/doc/rustc/src/profile-guided-optimization.md Outdated Show resolved Hide resolved

- It is recommended to use *absolute paths* for the argument of
`-Cprofile-generate` and `-Cprofile-use`. Cargo can invoke `rustc` with
varying working directories, meaning that `.profraw` files might show up

This comment has been minimized.

Copy link
@alexcrichton

alexcrichton May 28, 2019

Member

Hm this isn't quite right I think, don't .profraw files show up in the cwd by default? When compiling with profile-use, however, I think you definitely want to use absolute paths.

That being said absolute paths I think are still always your best bet.

This comment has been minimized.

Copy link
@michaelwoerister

michaelwoerister May 29, 2019

Author Contributor

True, things would still work for the "generate" case. I'll re-word but keep the recommendation for using absolute paths.

src/doc/rustc/src/profile-guided-optimization.md Outdated Show resolved Hide resolved
@rust-highfive

This comment has been minimized.

Copy link
Collaborator

commented May 28, 2019

The job x86_64-gnu-llvm-6.0 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
travis_time:end:0dcd936a:start=1559056887674898160,finish=1559056888433409042,duration=758510882
$ git checkout -qf FETCH_HEAD
travis_fold:end:git.checkout

Encrypted environment variables have been removed for security reasons.
See https://docs.travis-ci.com/user/pull-requests/#pull-requests-and-security-restrictions
$ export SCCACHE_BUCKET=rust-lang-ci-sccache2
$ export SCCACHE_REGION=us-west-1
$ export GCP_CACHE_BUCKET=rust-lang-ci-cache
$ export AWS_ACCESS_KEY_ID=AKIA46X5W6CZEJZ6XT55
---
travis_time:start:test_codegen
Check compiletest suite=codegen mode=codegen (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
[01:20:41] 
[01:20:41] running 143 tests
[01:20:44] i..iii.....iii..iiii.....i......................i...i................i......i.........ii.i..i..i.ii. 100/143
[01:20:46] test result: ok. 113 passed; 0 failed; 30 ignored; 0 measured; 0 filtered out
[01:20:46] 
[01:20:46]  finished in 4.696
[01:20:46] travis_fold:end:test_codegen
---
travis_time:start:test_assembly
Check compiletest suite=assembly mode=assembly (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
[01:20:48] 
[01:20:48] running 9 tests
[01:20:48] iiiiiiiii
[01:20:48] 
[01:20:48]  finished in 0.158
[01:20:48] travis_fold:end:test_assembly

---
travis_time:start:test_debuginfo
Check compiletest suite=debuginfo mode=debuginfo-gdb+lldb (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
[01:21:04] 
[01:21:04] running 122 tests
[01:21:30] .iiiii...i.....i..i...i..i.i.i..i.ii..i.i.....i..i....i..........iiii..........i...ii...i.......ii.i 100/122
[01:21:35] .i.i......iii.i.....ii
[01:21:35] 
[01:21:35]  finished in 30.513
[01:21:35] travis_fold:end:test_debuginfo

---
[01:45:25] 
[01:45:25] stdout ----
[01:45:25] 
[01:45:25] running 1 test
[01:45:25] test /checkout/src/doc/rustc/src/profile-guided-optimization.md - Profile_Guided_Optimization::Usage (line 43) ... FAILED
[01:45:25] failures:
[01:45:25] 
[01:45:25] 
[01:45:25] ---- /checkout/src/doc/rustc/src/profile-guided-optimization.md - Profile_Guided_Optimization::Usage (line 43) stdout ----
[01:45:25] error: expected one of `!`, `.`, `::`, `;`, `?`, `{`, `}`, or an operator, found `component`
[01:45:25]  --> /checkout/src/doc/rustc/src/profile-guided-optimization.md:44:8
[01:45:25] 3 | rustup component add llvm-tools-preview
[01:45:25]   |        ^^^^^^^^^ expected one of 8 possible tokens here
[01:45:25] 
[01:45:25] error[E0425]: cannot find value `rustup` in this scope
---
[01:45:25] 
[01:45:25] error: aborting due to 2 previous errors
[01:45:25] 
[01:45:25] For more information about this error, try `rustc --explain E0425`.
[01:45:25] thread '/checkout/src/doc/rustc/src/profile-guided-optimization.md - Profile_Guided_Optimization::Usage (line 43)' panicked at 'couldn't compile the test', src/librustdoc/test.rs:320:13
[01:45:25] 
[01:45:25] 
[01:45:25] failures:
[01:45:25] failures:
[01:45:25]     /checkout/src/doc/rustc/src/profile-guided-optimization.md - Profile_Guided_Optimization::Usage (line 43)
[01:45:25] test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out
[01:45:25] 
[01:45:25] 
[01:45:25] stderr ----
[01:45:25] stderr ----
[01:45:25] 
[01:45:25] 
[01:45:25] 
[01:45:25] failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test
[01:45:25] Build completed unsuccessfully in 0:36:45
[01:45:25] make: *** [check] Error 1
[01:45:25] Makefile:48: recipe for target 'check' failed
The command "stamp sh -x -c "$RUN_SCRIPT"" exited with 2.
travis_time:start:354b0800
$ date && (curl -fs --head https://google.com | grep ^Date: | sed 's/Date: //g' || true)
Tue May 28 17:07:05 UTC 2019
---
travis_time:end:094dae8e:start=1559063227275306941,finish=1559063227280379075,duration=5072134
travis_fold:end:after_failure.3
travis_fold:start:after_failure.4
travis_time:start:00e441d8
$ ln -s . checkout && for CORE in obj/cores/core.*; do EXE=$(echo $CORE | sed 's|obj/cores/core\.[0-9]*\.!checkout!\(.*\)|\1|;y|!|/|'); if [ -f "$EXE" ]; then printf travis_fold":start:crashlog\n\033[31;1m%s\033[0m\n" "$CORE"; gdb --batch -q -c "$CORE" "$EXE" -iex 'set auto-load off' -iex 'dir src/' -iex 'set sysroot .' -ex bt -ex q; echo travis_fold":"end:crashlog; fi; done || true
travis_fold:end:after_failure.4
travis_fold:start:after_failure.5
travis_time:start:0d57b450
travis_time:start:0d57b450
$ cat ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers || true
cat: ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers: No such file or directory
travis_fold:end:after_failure.5
travis_fold:start:after_failure.6
travis_time:start:0985bd58
$ dmesg | grep -i kill

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

@bors

This comment has been minimized.

Copy link
Contributor

commented May 28, 2019

☔️ The latest upstream changes (presumably #61274) made this pull request unmergeable. Please resolve the merge conflicts.

@michaelwoerister michaelwoerister force-pushed the michaelwoerister:stabilize-pgo branch from e8b88e9 to f334f2c May 29, 2019

@michaelwoerister

This comment has been minimized.

Copy link
Contributor Author

commented May 29, 2019

@alexcrichton, I removed the -Ccodegen-units=1 flags from the examples. What do you think of defaulting to one codegen unit (with the option to opt out) when compiling with PGO? I think that would remove a common footgun. And, perhaps more importantly, it removes ThinLTO from the equation which can only make things better/more reliable.

@alexcrichton

This comment has been minimized.

Copy link
Member

commented May 29, 2019

I mentioned above that I wouldn't switch CGUs by default just yet, but I would also expect ThinLTO to take PGO data into account to know what to inline, but I could very well be wrong about that!

@michaelwoerister

This comment has been minimized.

Copy link
Contributor Author

commented May 29, 2019

I mentioned above that I wouldn't switch CGUs by default just yet, but I would also expect ThinLTO to take PGO data into account to know what to inline, but I could very well be wrong about that!

OK, let's keep the two settings independent then. ThinLTO should indeed take PGO data into account since PGO adds regular cold and inlinehint annotation to functions.

This is mostly my general slight mistrust in (Thin)LTO speaking. It's just one more step where things can go wrong (and have done so in the past). I personally would always default to one CGU for any kind of production code. But that's a topic for another discussion :)

@bors

This comment has been minimized.

Copy link
Contributor

commented May 30, 2019

☔️ The latest upstream changes (presumably #61005) made this pull request unmergeable. Please resolve the merge conflicts.

@michaelwoerister michaelwoerister force-pushed the michaelwoerister:stabilize-pgo branch from f334f2c to 96f0086 May 30, 2019

@rfcbot

This comment has been minimized.

Copy link

commented Jun 4, 2019

🔔 This is now entering its final comment period, as per the review above. 🔔

@rfcbot

This comment has been minimized.

Copy link

commented Jun 14, 2019

The final comment period, with a disposition to merge, as per the review above, is now complete.

As the automated representative of the governance process, I would like to thank the author for their work and everyone else who contributed.

The RFC will be merged soon.

@bors

This comment has been minimized.

Copy link
Contributor

commented Jun 16, 2019

☔️ The latest upstream changes (presumably #61739) made this pull request unmergeable. Please resolve the merge conflicts.

@michaelwoerister michaelwoerister force-pushed the michaelwoerister:stabilize-pgo branch from 96f0086 to b7fe2ca Jun 21, 2019

@michaelwoerister

This comment has been minimized.

Copy link
Contributor Author

commented Jun 21, 2019

@bors r=alexcrichton (right?)

@bors

This comment has been minimized.

Copy link
Contributor

commented Jun 21, 2019

📌 Commit b7fe2ca has been approved by alexcrichton

@mati865

This comment has been minimized.

Copy link
Contributor

commented Jul 2, 2019

This PR doesn't appear on https://buildbot2.rust-lang.org/homu/queue/rust
@michaelwoerister you will have to re-queue it.

@alexcrichton alexcrichton reopened this Jul 2, 2019

@alexcrichton

This comment has been minimized.

Copy link
Member

commented Jul 2, 2019

@bors: r+

@bors

This comment has been minimized.

Copy link
Contributor

commented Jul 2, 2019

💡 This pull request was already approved, no need to approve it again.

  • There's another pull request that is currently being tested, blocking this pull request: #61922
@bors

This comment has been minimized.

Copy link
Contributor

commented Jul 2, 2019

📌 Commit b7fe2ca has been approved by alexcrichton

@bors

This comment has been minimized.

Copy link
Contributor

commented Jul 2, 2019

⌛️ Testing commit b7fe2ca with merge 30fcafb...

bors added a commit that referenced this pull request Jul 2, 2019
Auto merge of #61268 - michaelwoerister:stabilize-pgo, r=alexcrichton
Stabilize support for Profile-guided Optimization

This PR makes profile-guided optimization available via the `-C profile-generate` / `-C profile-use` pair of commandline flags and adds end-user documentation for the feature to the [rustc book](https://doc.rust-lang.org/rustc/). The PR thus ticks the last two remaining checkboxes of the [stabilization tracking issue](#59913).

From the tracking issue:
> Profile-guided optimization (PGO) is a common optimization technique for ahead-of-time compilers. It works by collecting data about a program's typical execution (e.g. probability of branches taken, typical runtime values of variables, etc) and then uses this information during program optimization for things like inlining decisions, machine code layout, or indirect call promotion.

If you are curious about how this can be used, there is a rendered version of the documentation this PR adds available [here](
https://github.com/michaelwoerister/rust/blob/stabilize-pgo/src/doc/rustc/src/profile-guided-optimization.md).

r? @alexcrichton
cc @rust-lang/compiler
@Centril

This comment has been minimized.

Copy link
Member

commented Jul 2, 2019

@bors retry Yielding to stable emergency backport (#62311).

@Centril

This comment has been minimized.

Copy link
Member

commented Jul 2, 2019

@bors p=50

@Centril Centril added this to the 1.37 milestone Jul 2, 2019

@bors

This comment has been minimized.

Copy link
Contributor

commented Jul 2, 2019

⌛️ Testing commit b7fe2ca with merge 0beb2ba...

bors added a commit that referenced this pull request Jul 2, 2019
Auto merge of #61268 - michaelwoerister:stabilize-pgo, r=alexcrichton
Stabilize support for Profile-guided Optimization

This PR makes profile-guided optimization available via the `-C profile-generate` / `-C profile-use` pair of commandline flags and adds end-user documentation for the feature to the [rustc book](https://doc.rust-lang.org/rustc/). The PR thus ticks the last two remaining checkboxes of the [stabilization tracking issue](#59913).

From the tracking issue:
> Profile-guided optimization (PGO) is a common optimization technique for ahead-of-time compilers. It works by collecting data about a program's typical execution (e.g. probability of branches taken, typical runtime values of variables, etc) and then uses this information during program optimization for things like inlining decisions, machine code layout, or indirect call promotion.

If you are curious about how this can be used, there is a rendered version of the documentation this PR adds available [here](
https://github.com/michaelwoerister/rust/blob/stabilize-pgo/src/doc/rustc/src/profile-guided-optimization.md).

r? @alexcrichton
cc @rust-lang/compiler

@Centril Centril added the relnotes label Jul 2, 2019

@bors

This comment has been minimized.

Copy link
Contributor

commented Jul 2, 2019

☀️ Test successful - checks-azure, checks-travis, status-appveyor
Approved by: alexcrichton
Pushing 0beb2ba to master...

@bors bors added the merged-by-bors label Jul 2, 2019

@bors bors merged commit b7fe2ca into rust-lang:master Jul 2, 2019

4 checks passed

Travis CI - Pull Request Build Passed
Details
homu Test successful
Details
pr Build #20190702.40 succeeded
Details
pr (Linux x86_64-gnu-llvm-6.0) Linux x86_64-gnu-llvm-6.0 succeeded
Details
netbsd-srcmastr pushed a commit to NetBSD/pkgsrc that referenced this pull request Aug 29, 2019
he
Update rust to version 1.37.0
Pkgsrc changes:
 * Add a patch to llvm to deal with const dli_saddr.
 * Adapt two other patches.
 * Cross-build currently fails, so i386, powerpc and sparc64 bootstrap
   kits for 1.37.0 are built natively.  Missing aarch64 hardware, so that's
   not available yet.
 * Bump bootstrap requirements to 1.36.0 except for armv7-unknown-netbsd-eabihf
   which I've not managed to cross-build.

Upstream changes:

Version 1.37.0 (2019-08-15)
==========================

Language
--------
- `#[must_use]` will now warn if the type is contained in a [tuple][61100],
  [`Box`][62228], or an [array][62235] and unused.
- [You can now use the `cfg` and `cfg_attr` attributes on
  generic parameters.][61547]
- [You can now use enum variants through type alias.][61682] e.g. You can
  write the following:
  ```rust
  type MyOption = Option<u8>;

  fn increment_or_zero(x: MyOption) -> u8 {
      match x {
          MyOption::Some(y) => y + 1,
          MyOption::None => 0,
      }
  }
  ```
- [You can now use `_` as an identifier for consts.][61347] e.g. You can write
  `const _: u32 = 5;`.
- [You can now use `#[repr(align(X)]` on enums.][61229]
- [The  `?`/_"Kleene"_ macro operator is now available in the
  2015 edition.][60932]

Compiler
--------
- [You can now enable Profile-Guided Optimization with the `-C profile-generate`
  and `-C profile-use` flags.][61268] For more information on how to use profile
  guided optimization, please refer to the [rustc book][rustc-book-pgo].
- [The `rust-lldb` wrapper script should now work again.][61827]

Libraries
---------
- [`mem::MaybeUninit<T>` is now ABI-compatible with `T`.][61802]

Stabilized APIs
---------------
- [`BufReader::buffer`]
- [`BufWriter::buffer`]
- [`Cell::from_mut`]
- [`Cell<[T]>::as_slice_of_cells`][`Cell<slice>::as_slice_of_cells`]
- [`DoubleEndedIterator::nth_back`]
- [`Option::xor`]
- [`Wrapping::reverse_bits`]
- [`i128::reverse_bits`]
- [`i16::reverse_bits`]
- [`i32::reverse_bits`]
- [`i64::reverse_bits`]
- [`i8::reverse_bits`]
- [`isize::reverse_bits`]
- [`slice::copy_within`]
- [`u128::reverse_bits`]
- [`u16::reverse_bits`]
- [`u32::reverse_bits`]
- [`u64::reverse_bits`]
- [`u8::reverse_bits`]
- [`usize::reverse_bits`]

Cargo
-----
- [`Cargo.lock` files are now included by default when publishing executable crates
  with executables.][cargo/7026]
- [You can now specify `default-run="foo"` in `[package]` to specify the
  default executable to use for `cargo run`.][cargo/7056]

Misc
----

Compatibility Notes
-------------------
- [Using `...` for inclusive range patterns will now warn by default.][61342]
  Please transition your code to using the `..=` syntax for inclusive
  ranges instead.
- [Using a trait object without the `dyn` will now warn by default.][61203]
  Please transition your code to use `dyn Trait` for trait objects instead.

[62228]: rust-lang/rust#62228
[62235]: rust-lang/rust#62235
[61802]: rust-lang/rust#61802
[61827]: rust-lang/rust#61827
[61547]: rust-lang/rust#61547
[61682]: rust-lang/rust#61682
[61268]: rust-lang/rust#61268
[61342]: rust-lang/rust#61342
[61347]: rust-lang/rust#61347
[61100]: rust-lang/rust#61100
[61203]: rust-lang/rust#61203
[61229]: rust-lang/rust#61229
[60932]: rust-lang/rust#60932
[cargo/7026]: rust-lang/cargo#7026
[cargo/7056]: rust-lang/cargo#7056
[`BufReader::buffer`]: https://doc.rust-lang.org/std/io/struct.BufReader.html#method.buffer
[`BufWriter::buffer`]: https://doc.rust-lang.org/std/io/struct.BufWriter.html#method.buffer
[`Cell::from_mut`]: https://doc.rust-lang.org/std/cell/struct.Cell.html#method.from_mut
[`Cell<slice>::as_slice_of_cells`]: https://doc.rust-lang.org/std/cell/struct.Cell.html#method.as_slice_of_cells
[`DoubleEndedIterator::nth_back`]: https://doc.rust-lang.org/std/iter/trait.DoubleEndedIterator.html#method.nth_back
[`Option::xor`]: https://doc.rust-lang.org/std/option/enum.Option.html#method.xor
[`RefCell::try_borrow_unguarded`]: https://doc.rust-lang.org/std/cell/struct.RefCell.html#method.try_borrow_unguarded
[`Wrapping::reverse_bits`]: https://doc.rust-lang.org/std/num/struct.Wrapping.html#method.reverse_bits
[`i128::reverse_bits`]: https://doc.rust-lang.org/std/primitive.i128.html#method.reverse_bits
[`i16::reverse_bits`]: https://doc.rust-lang.org/std/primitive.i16.html#method.reverse_bits
[`i32::reverse_bits`]: https://doc.rust-lang.org/std/primitive.i32.html#method.reverse_bits
[`i64::reverse_bits`]: https://doc.rust-lang.org/std/primitive.i64.html#method.reverse_bits
[`i8::reverse_bits`]: https://doc.rust-lang.org/std/primitive.i8.html#method.reverse_bits
[`isize::reverse_bits`]: https://doc.rust-lang.org/std/primitive.isize.html#method.reverse_bits
[`slice::copy_within`]: https://doc.rust-lang.org/std/primitive.slice.html#method.copy_within
[`u128::reverse_bits`]: https://doc.rust-lang.org/std/primitive.u128.html#method.reverse_bits
[`u16::reverse_bits`]: https://doc.rust-lang.org/std/primitive.u16.html#method.reverse_bits
[`u32::reverse_bits`]: https://doc.rust-lang.org/std/primitive.u32.html#method.reverse_bits
[`u64::reverse_bits`]: https://doc.rust-lang.org/std/primitive.u64.html#method.reverse_bits
[`u8::reverse_bits`]: https://doc.rust-lang.org/std/primitive.u8.html#method.reverse_bits
[`usize::reverse_bits`]: https://doc.rust-lang.org/std/primitive.usize.html#method.reverse_bits
[rustc-book-pgo]: https://doc.rust-lang.org/rustc/profile-guided-optimization.html
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.