Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stabilize support for Profile-guided Optimization #61268

Merged
merged 2 commits into from Jul 2, 2019

Conversation

@michaelwoerister
Copy link
Member

@michaelwoerister michaelwoerister commented May 28, 2019

This PR makes profile-guided optimization available via the -C profile-generate / -C profile-use pair of commandline flags and adds end-user documentation for the feature to the rustc book. The PR thus ticks the last two remaining checkboxes of the stabilization tracking issue.

From the tracking issue:

Profile-guided optimization (PGO) is a common optimization technique for ahead-of-time compilers. It works by collecting data about a program's typical execution (e.g. probability of branches taken, typical runtime values of variables, etc) and then uses this information during program optimization for things like inlining decisions, machine code layout, or indirect call promotion.

If you are curious about how this can be used, there is a rendered version of the documentation this PR adds available here.

r? @alexcrichton
cc @rust-lang/compiler

@michaelwoerister
Copy link
Member Author

@michaelwoerister michaelwoerister commented May 28, 2019

Dear @rust-lang/compiler, please add your check mark below.
@rfcbot fcp merge

Loading

@rfcbot
Copy link

@rfcbot rfcbot commented May 28, 2019

Team member @michaelwoerister has proposed to merge this. The next step is review by the rest of the tagged team members:

No concerns currently listed.

Once a majority of reviewers approve (and at most 2 approvals are outstanding), this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up!

See this document for info about what commands tagged team members can give me.

Loading

Loading
src/doc/rustc/src/profile-guided-optimization.md Outdated Show resolved Hide resolved
Loading
Loading

- It is recommended to use *absolute paths* for the argument of
`-Cprofile-generate` and `-Cprofile-use`. Cargo can invoke `rustc` with
varying working directories, meaning that `.profraw` files might show up
Copy link
Member

@alexcrichton alexcrichton May 28, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm this isn't quite right I think, don't .profraw files show up in the cwd by default? When compiling with profile-use, however, I think you definitely want to use absolute paths.

That being said absolute paths I think are still always your best bet.

Loading

Copy link
Member Author

@michaelwoerister michaelwoerister May 29, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

True, things would still work for the "generate" case. I'll re-word but keep the recommendation for using absolute paths.

Loading

src/doc/rustc/src/profile-guided-optimization.md Outdated Show resolved Hide resolved
Loading
Loading
@rust-highfive
Copy link
Collaborator

@rust-highfive rust-highfive commented May 28, 2019

The job x86_64-gnu-llvm-6.0 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
travis_time:end:0dcd936a:start=1559056887674898160,finish=1559056888433409042,duration=758510882
$ git checkout -qf FETCH_HEAD
travis_fold:end:git.checkout

Encrypted environment variables have been removed for security reasons.
See https://docs.travis-ci.com/user/pull-requests/#pull-requests-and-security-restrictions
$ export SCCACHE_BUCKET=rust-lang-ci-sccache2
$ export SCCACHE_REGION=us-west-1
$ export GCP_CACHE_BUCKET=rust-lang-ci-cache
$ export AWS_ACCESS_KEY_ID=AKIA46X5W6CZEJZ6XT55
---
travis_time:start:test_codegen
Check compiletest suite=codegen mode=codegen (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
[01:20:41] 
[01:20:41] running 143 tests
[01:20:44] i..iii.....iii..iiii.....i......................i...i................i......i.........ii.i..i..i.ii. 100/143
[01:20:46] test result: ok. 113 passed; 0 failed; 30 ignored; 0 measured; 0 filtered out
[01:20:46] 
[01:20:46]  finished in 4.696
[01:20:46] travis_fold:end:test_codegen
---
travis_time:start:test_assembly
Check compiletest suite=assembly mode=assembly (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
[01:20:48] 
[01:20:48] running 9 tests
[01:20:48] iiiiiiiii
[01:20:48] 
[01:20:48]  finished in 0.158
[01:20:48] travis_fold:end:test_assembly

---
travis_time:start:test_debuginfo
Check compiletest suite=debuginfo mode=debuginfo-gdb+lldb (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
[01:21:04] 
[01:21:04] running 122 tests
[01:21:30] .iiiii...i.....i..i...i..i.i.i..i.ii..i.i.....i..i....i..........iiii..........i...ii...i.......ii.i 100/122
[01:21:35] .i.i......iii.i.....ii
[01:21:35] 
[01:21:35]  finished in 30.513
[01:21:35] travis_fold:end:test_debuginfo

---
[01:45:25] 
[01:45:25] stdout ----
[01:45:25] 
[01:45:25] running 1 test
[01:45:25] test /checkout/src/doc/rustc/src/profile-guided-optimization.md - Profile_Guided_Optimization::Usage (line 43) ... FAILED
[01:45:25] failures:
[01:45:25] 
[01:45:25] 
[01:45:25] ---- /checkout/src/doc/rustc/src/profile-guided-optimization.md - Profile_Guided_Optimization::Usage (line 43) stdout ----
[01:45:25] error: expected one of `!`, `.`, `::`, `;`, `?`, `{`, `}`, or an operator, found `component`
[01:45:25]  --> /checkout/src/doc/rustc/src/profile-guided-optimization.md:44:8
[01:45:25] 3 | rustup component add llvm-tools-preview
[01:45:25]   |        ^^^^^^^^^ expected one of 8 possible tokens here
[01:45:25] 
[01:45:25] error[E0425]: cannot find value `rustup` in this scope
---
[01:45:25] 
[01:45:25] error: aborting due to 2 previous errors
[01:45:25] 
[01:45:25] For more information about this error, try `rustc --explain E0425`.
[01:45:25] thread '/checkout/src/doc/rustc/src/profile-guided-optimization.md - Profile_Guided_Optimization::Usage (line 43)' panicked at 'couldn't compile the test', src/librustdoc/test.rs:320:13
[01:45:25] 
[01:45:25] 
[01:45:25] failures:
[01:45:25] failures:
[01:45:25]     /checkout/src/doc/rustc/src/profile-guided-optimization.md - Profile_Guided_Optimization::Usage (line 43)
[01:45:25] test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out
[01:45:25] 
[01:45:25] 
[01:45:25] stderr ----
[01:45:25] stderr ----
[01:45:25] 
[01:45:25] 
[01:45:25] 
[01:45:25] failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test
[01:45:25] Build completed unsuccessfully in 0:36:45
[01:45:25] make: *** [check] Error 1
[01:45:25] Makefile:48: recipe for target 'check' failed
The command "stamp sh -x -c "$RUN_SCRIPT"" exited with 2.
travis_time:start:354b0800
$ date && (curl -fs --head https://google.com | grep ^Date: | sed 's/Date: //g' || true)
Tue May 28 17:07:05 UTC 2019
---
travis_time:end:094dae8e:start=1559063227275306941,finish=1559063227280379075,duration=5072134
travis_fold:end:after_failure.3
travis_fold:start:after_failure.4
travis_time:start:00e441d8
$ ln -s . checkout && for CORE in obj/cores/core.*; do EXE=$(echo $CORE | sed 's|obj/cores/core\.[0-9]*\.!checkout!\(.*\)|\1|;y|!|/|'); if [ -f "$EXE" ]; then printf travis_fold":start:crashlog\n\033[31;1m%s\033[0m\n" "$CORE"; gdb --batch -q -c "$CORE" "$EXE" -iex 'set auto-load off' -iex 'dir src/' -iex 'set sysroot .' -ex bt -ex q; echo travis_fold":"end:crashlog; fi; done || true
travis_fold:end:after_failure.4
travis_fold:start:after_failure.5
travis_time:start:0d57b450
travis_time:start:0d57b450
$ cat ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers || true
cat: ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers: No such file or directory
travis_fold:end:after_failure.5
travis_fold:start:after_failure.6
travis_time:start:0985bd58
$ dmesg | grep -i kill

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

Loading

@bors
Copy link
Contributor

@bors bors commented May 28, 2019

The latest upstream changes (presumably #61274) made this pull request unmergeable. Please resolve the merge conflicts.

Loading

@michaelwoerister
Copy link
Member Author

@michaelwoerister michaelwoerister commented May 29, 2019

@alexcrichton, I removed the -Ccodegen-units=1 flags from the examples. What do you think of defaulting to one codegen unit (with the option to opt out) when compiling with PGO? I think that would remove a common footgun. And, perhaps more importantly, it removes ThinLTO from the equation which can only make things better/more reliable.

Loading

@alexcrichton
Copy link
Member

@alexcrichton alexcrichton commented May 29, 2019

I mentioned above that I wouldn't switch CGUs by default just yet, but I would also expect ThinLTO to take PGO data into account to know what to inline, but I could very well be wrong about that!

Loading

@michaelwoerister
Copy link
Member Author

@michaelwoerister michaelwoerister commented May 29, 2019

I mentioned above that I wouldn't switch CGUs by default just yet, but I would also expect ThinLTO to take PGO data into account to know what to inline, but I could very well be wrong about that!

OK, let's keep the two settings independent then. ThinLTO should indeed take PGO data into account since PGO adds regular cold and inlinehint annotation to functions.

This is mostly my general slight mistrust in (Thin)LTO speaking. It's just one more step where things can go wrong (and have done so in the past). I personally would always default to one CGU for any kind of production code. But that's a topic for another discussion :)

Loading

@bors
Copy link
Contributor

@bors bors commented May 30, 2019

The latest upstream changes (presumably #61005) made this pull request unmergeable. Please resolve the merge conflicts.

Loading

@rfcbot
Copy link

@rfcbot rfcbot commented Jun 4, 2019

🔔 This is now entering its final comment period, as per the review above. 🔔

Loading

@rfcbot
Copy link

@rfcbot rfcbot commented Jun 14, 2019

The final comment period, with a disposition to merge, as per the review above, is now complete.

As the automated representative of the governance process, I would like to thank the author for their work and everyone else who contributed.

The RFC will be merged soon.

Loading

@bors
Copy link
Contributor

@bors bors commented Jun 16, 2019

The latest upstream changes (presumably #61739) made this pull request unmergeable. Please resolve the merge conflicts.

Loading

@michaelwoerister
Copy link
Member Author

@michaelwoerister michaelwoerister commented Jun 21, 2019

@bors r=alexcrichton (right?)

Loading

@bors
Copy link
Contributor

@bors bors commented Jun 21, 2019

📌 Commit b7fe2ca has been approved by alexcrichton

Loading

@mati865
Copy link
Contributor

@mati865 mati865 commented Jul 2, 2019

This PR doesn't appear on https://buildbot2.rust-lang.org/homu/queue/rust
@michaelwoerister you will have to re-queue it.

Loading

@alexcrichton alexcrichton reopened this Jul 2, 2019
@alexcrichton
Copy link
Member

@alexcrichton alexcrichton commented Jul 2, 2019

@bors: r+

Loading

@bors
Copy link
Contributor

@bors bors commented Jul 2, 2019

💡 This pull request was already approved, no need to approve it again.

  • There's another pull request that is currently being tested, blocking this pull request: #61922

Loading

@bors
Copy link
Contributor

@bors bors commented Jul 2, 2019

📌 Commit b7fe2ca has been approved by alexcrichton

Loading

@bors
Copy link
Contributor

@bors bors commented Jul 2, 2019

Testing commit b7fe2ca with merge 30fcafb...

Loading

bors added a commit that referenced this issue Jul 2, 2019
Stabilize support for Profile-guided Optimization

This PR makes profile-guided optimization available via the `-C profile-generate` / `-C profile-use` pair of commandline flags and adds end-user documentation for the feature to the [rustc book](https://doc.rust-lang.org/rustc/). The PR thus ticks the last two remaining checkboxes of the [stabilization tracking issue](#59913).

From the tracking issue:
> Profile-guided optimization (PGO) is a common optimization technique for ahead-of-time compilers. It works by collecting data about a program's typical execution (e.g. probability of branches taken, typical runtime values of variables, etc) and then uses this information during program optimization for things like inlining decisions, machine code layout, or indirect call promotion.

If you are curious about how this can be used, there is a rendered version of the documentation this PR adds available [here](
https://github.com/michaelwoerister/rust/blob/stabilize-pgo/src/doc/rustc/src/profile-guided-optimization.md).

r? @alexcrichton
cc @rust-lang/compiler
@Centril
Copy link
Contributor

@Centril Centril commented Jul 2, 2019

@bors retry Yielding to stable emergency backport (#62311).

Loading

@Centril
Copy link
Contributor

@Centril Centril commented Jul 2, 2019

@bors p=50

Loading

@Centril Centril added this to the 1.37 milestone Jul 2, 2019
@bors
Copy link
Contributor

@bors bors commented Jul 2, 2019

Testing commit b7fe2ca with merge 0beb2ba...

Loading

bors added a commit that referenced this issue Jul 2, 2019
Stabilize support for Profile-guided Optimization

This PR makes profile-guided optimization available via the `-C profile-generate` / `-C profile-use` pair of commandline flags and adds end-user documentation for the feature to the [rustc book](https://doc.rust-lang.org/rustc/). The PR thus ticks the last two remaining checkboxes of the [stabilization tracking issue](#59913).

From the tracking issue:
> Profile-guided optimization (PGO) is a common optimization technique for ahead-of-time compilers. It works by collecting data about a program's typical execution (e.g. probability of branches taken, typical runtime values of variables, etc) and then uses this information during program optimization for things like inlining decisions, machine code layout, or indirect call promotion.

If you are curious about how this can be used, there is a rendered version of the documentation this PR adds available [here](
https://github.com/michaelwoerister/rust/blob/stabilize-pgo/src/doc/rustc/src/profile-guided-optimization.md).

r? @alexcrichton
cc @rust-lang/compiler
@bors
Copy link
Contributor

@bors bors commented Jul 2, 2019

☀️ Test successful - checks-azure, checks-travis, status-appveyor
Approved by: alexcrichton
Pushing 0beb2ba to master...

Loading

@bors bors merged commit b7fe2ca into rust-lang:master Jul 2, 2019
4 checks passed
Loading
netbsd-srcmastr pushed a commit to NetBSD/pkgsrc that referenced this issue Aug 29, 2019
Pkgsrc changes:
 * Add a patch to llvm to deal with const dli_saddr.
 * Adapt two other patches.
 * Cross-build currently fails, so i386, powerpc and sparc64 bootstrap
   kits for 1.37.0 are built natively.  Missing aarch64 hardware, so that's
   not available yet.
 * Bump bootstrap requirements to 1.36.0 except for armv7-unknown-netbsd-eabihf
   which I've not managed to cross-build.

Upstream changes:

Version 1.37.0 (2019-08-15)
==========================

Language
--------
- `#[must_use]` will now warn if the type is contained in a [tuple][61100],
  [`Box`][62228], or an [array][62235] and unused.
- [You can now use the `cfg` and `cfg_attr` attributes on
  generic parameters.][61547]
- [You can now use enum variants through type alias.][61682] e.g. You can
  write the following:
  ```rust
  type MyOption = Option<u8>;

  fn increment_or_zero(x: MyOption) -> u8 {
      match x {
          MyOption::Some(y) => y + 1,
          MyOption::None => 0,
      }
  }
  ```
- [You can now use `_` as an identifier for consts.][61347] e.g. You can write
  `const _: u32 = 5;`.
- [You can now use `#[repr(align(X)]` on enums.][61229]
- [The  `?`/_"Kleene"_ macro operator is now available in the
  2015 edition.][60932]

Compiler
--------
- [You can now enable Profile-Guided Optimization with the `-C profile-generate`
  and `-C profile-use` flags.][61268] For more information on how to use profile
  guided optimization, please refer to the [rustc book][rustc-book-pgo].
- [The `rust-lldb` wrapper script should now work again.][61827]

Libraries
---------
- [`mem::MaybeUninit<T>` is now ABI-compatible with `T`.][61802]

Stabilized APIs
---------------
- [`BufReader::buffer`]
- [`BufWriter::buffer`]
- [`Cell::from_mut`]
- [`Cell<[T]>::as_slice_of_cells`][`Cell<slice>::as_slice_of_cells`]
- [`DoubleEndedIterator::nth_back`]
- [`Option::xor`]
- [`Wrapping::reverse_bits`]
- [`i128::reverse_bits`]
- [`i16::reverse_bits`]
- [`i32::reverse_bits`]
- [`i64::reverse_bits`]
- [`i8::reverse_bits`]
- [`isize::reverse_bits`]
- [`slice::copy_within`]
- [`u128::reverse_bits`]
- [`u16::reverse_bits`]
- [`u32::reverse_bits`]
- [`u64::reverse_bits`]
- [`u8::reverse_bits`]
- [`usize::reverse_bits`]

Cargo
-----
- [`Cargo.lock` files are now included by default when publishing executable crates
  with executables.][cargo/7026]
- [You can now specify `default-run="foo"` in `[package]` to specify the
  default executable to use for `cargo run`.][cargo/7056]

Misc
----

Compatibility Notes
-------------------
- [Using `...` for inclusive range patterns will now warn by default.][61342]
  Please transition your code to using the `..=` syntax for inclusive
  ranges instead.
- [Using a trait object without the `dyn` will now warn by default.][61203]
  Please transition your code to use `dyn Trait` for trait objects instead.

[62228]: rust-lang/rust#62228
[62235]: rust-lang/rust#62235
[61802]: rust-lang/rust#61802
[61827]: rust-lang/rust#61827
[61547]: rust-lang/rust#61547
[61682]: rust-lang/rust#61682
[61268]: rust-lang/rust#61268
[61342]: rust-lang/rust#61342
[61347]: rust-lang/rust#61347
[61100]: rust-lang/rust#61100
[61203]: rust-lang/rust#61203
[61229]: rust-lang/rust#61229
[60932]: rust-lang/rust#60932
[cargo/7026]: rust-lang/cargo#7026
[cargo/7056]: rust-lang/cargo#7056
[`BufReader::buffer`]: https://doc.rust-lang.org/std/io/struct.BufReader.html#method.buffer
[`BufWriter::buffer`]: https://doc.rust-lang.org/std/io/struct.BufWriter.html#method.buffer
[`Cell::from_mut`]: https://doc.rust-lang.org/std/cell/struct.Cell.html#method.from_mut
[`Cell<slice>::as_slice_of_cells`]: https://doc.rust-lang.org/std/cell/struct.Cell.html#method.as_slice_of_cells
[`DoubleEndedIterator::nth_back`]: https://doc.rust-lang.org/std/iter/trait.DoubleEndedIterator.html#method.nth_back
[`Option::xor`]: https://doc.rust-lang.org/std/option/enum.Option.html#method.xor
[`RefCell::try_borrow_unguarded`]: https://doc.rust-lang.org/std/cell/struct.RefCell.html#method.try_borrow_unguarded
[`Wrapping::reverse_bits`]: https://doc.rust-lang.org/std/num/struct.Wrapping.html#method.reverse_bits
[`i128::reverse_bits`]: https://doc.rust-lang.org/std/primitive.i128.html#method.reverse_bits
[`i16::reverse_bits`]: https://doc.rust-lang.org/std/primitive.i16.html#method.reverse_bits
[`i32::reverse_bits`]: https://doc.rust-lang.org/std/primitive.i32.html#method.reverse_bits
[`i64::reverse_bits`]: https://doc.rust-lang.org/std/primitive.i64.html#method.reverse_bits
[`i8::reverse_bits`]: https://doc.rust-lang.org/std/primitive.i8.html#method.reverse_bits
[`isize::reverse_bits`]: https://doc.rust-lang.org/std/primitive.isize.html#method.reverse_bits
[`slice::copy_within`]: https://doc.rust-lang.org/std/primitive.slice.html#method.copy_within
[`u128::reverse_bits`]: https://doc.rust-lang.org/std/primitive.u128.html#method.reverse_bits
[`u16::reverse_bits`]: https://doc.rust-lang.org/std/primitive.u16.html#method.reverse_bits
[`u32::reverse_bits`]: https://doc.rust-lang.org/std/primitive.u32.html#method.reverse_bits
[`u64::reverse_bits`]: https://doc.rust-lang.org/std/primitive.u64.html#method.reverse_bits
[`u8::reverse_bits`]: https://doc.rust-lang.org/std/primitive.u8.html#method.reverse_bits
[`usize::reverse_bits`]: https://doc.rust-lang.org/std/primitive.usize.html#method.reverse_bits
[rustc-book-pgo]: https://doc.rust-lang.org/rustc/profile-guided-optimization.html
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

8 participants