enable parallel rustc front end in nightly builds #117435

SparrowLii · 2023-10-31T08:48:28Z

Refers to the MCP, this pr does:

Enable the parallel front end in nightly builds, and keep the default number of threads as 1. Then users can use the parallel rustc front end via -Z threads=n option.
Set it up to serial front end for beta/stable builds via bootstrap.
Switch over the alt builders from parallel rustc to serial, so we have artifacts without parallel to test against the artifacts with parallel.

cc @cjgillot @nnethercote @bjorn3 @Kobzol

rustbot · 2023-10-31T08:48:39Z

This PR modifies src/bootstrap/src/core/config. If appropriate, please also update CONFIG_CHANGE_HISTORY in src/bootstrap/src/lib.rs and change-id in config.example.toml.

rustbot · 2023-10-31T08:52:30Z

This PR changes config.example.toml. If appropriate, please also update CONFIG_CHANGE_HISTORY in src/bootstrap/src/lib.rs and change-id in config.example.toml.

src/bootstrap/src/core/config/config.rs

oli-obk

I'm gonna let people chime in on this for a day or two. If I don't approve by Thursday myself, r=me

lqd · 2023-11-01T14:52:46Z

Do we need to land #115220 prior to this PR? Or the fact that it applies to more than a single thread is enough not to (in my mind people are immediately going to try >1 threads).

oli-obk · 2023-11-01T15:25:51Z

More than one thread is deadlock prone and anyone opting into it with the -Z flag has to accept that I think

SparrowLii · 2023-11-03T03:05:33Z

@oli-obk I think we can merge now

@bors r=oli-obk rollup=never

bors · 2023-11-03T03:05:40Z

📌 Commit 248dd14 has been approved by oli-obk

It is now in the queue for this repository.

src/ci/run.sh

matthiaskrgr · 2023-11-03T11:43:30Z

@bors rollup=never

bors · 2023-11-03T14:40:31Z

⌛ Testing commit 248dd14 with merge bb4d1c3...

enable parallel rustc front end in nightly builds Refers to the [MCP](rust-lang/compiler-team#681), this pr does: 1. Enable the parallel front end in nightly builds, and keep the default number of threads as 1. Then users can use the parallel rustc front end via -Z threads=n option. 2. Set it up to serial front end for beta/stable builds via bootstrap. 3. Switch over the alt builders from parallel rustc to serial, so we have artifacts without parallel to test against the artifacts with parallel. r? `@oli-obk` cc `@cjgillot` `@nnethercote` `@bjorn3` `@Kobzol`

davidtwco · 2023-11-06T05:15:00Z

r? @davidtwco

@bors r=oli-obk,davidtwco

bors · 2023-11-06T05:15:02Z

📌 Commit f2a40e9 has been approved by oli-obk,davidtwco

It is now in the queue for this repository.

bors · 2023-11-06T07:41:26Z

⌛ Testing commit f2a40e9 with merge f9b6446...

bors · 2023-11-06T09:46:33Z

☀️ Test successful - checks-actions
Approved by: oli-obk,davidtwco
Pushing f9b6446 to master...

rust-timer · 2023-11-06T21:31:33Z

Finished benchmarking commit (f9b6446): comparison URL.

Overall result: ❌ regressions - ACTION NEEDED

Next Steps: If you can justify the regressions found in this perf run, please indicate this with @rustbot label: +perf-regression-triaged along with sufficient written justification. If you cannot justify the regressions please open an issue or create a new PR that fixes the regressions, add a comment linking to the newly created issue or PR, and then add the perf-regression-triaged label to this PR.

@rustbot label: +perf-regression
cc @rust-lang/wg-compiler-performance

Instruction count

This is a highly reliable metric that was used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	2.9%	[0.4%, 5.1%]	209
Regressions ❌ (secondary)	2.7%	[0.4%, 9.1%]	219
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	2.9%	[0.4%, 5.1%]	209

Max RSS (memory usage)

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	1.2%	[0.4%, 2.6%]	28
Regressions ❌ (secondary)	2.5%	[0.5%, 6.8%]	61
Improvements ✅ (primary)	-1.6%	[-2.2%, -0.9%]	2
Improvements ✅ (secondary)	-2.2%	[-2.5%, -2.0%]	3
All ❌✅ (primary)	1.0%	[-2.2%, 2.6%]	30

Cycles

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	2.5%	[0.4%, 5.2%]	186
Regressions ❌ (secondary)	3.3%	[1.3%, 8.3%]	119
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	2.5%	[0.4%, 5.2%]	186

Binary size

This benchmark run did not return any relevant results for this metric.

Bootstrap: 636.284s -> 661.435s (3.95%)
Artifact size: 304.53 MiB -> 308.99 MiB (1.46%)

89: Automated pull from upstream `master` r=Dajamante a=github-actions[bot] This PR pulls the following changes from the upstream repository: * rust-lang/rust#117006 * rust-lang/rust#117511 * rust-lang/rust#117641 * rust-lang/rust#117637 * rust-lang/rust#117631 * rust-lang/rust#117516 * rust-lang/rust#117190 * rust-lang/rust#117292 * rust-lang/rust#117603 * rust-lang/rust#116988 * rust-lang/rust#117630 * rust-lang/rust#117615 * rust-lang/rust#117613 * rust-lang/rust#117592 * rust-lang/rust#117578 * rust-lang/rust#117435 * rust-lang/rust#117607 90: bump serde and serde_derive r=tshepang a=Dajamante Trying to get around the failure seen in #86 Co-authored-by: Ralf Jung <post@ralfj.de> Co-authored-by: Esteban Küber <esteban@kuber.com.ar> Co-authored-by: SparrowLii <liyuan179@huawei.com> Co-authored-by: Matthias Krüger <matthias.krueger@famsik.de> Co-authored-by: Gurinder Singh <frederick.the.fool@gmail.com> Co-authored-by: Michael Goulet <michael@errs.io> Co-authored-by: Thom Chiovoloni <thom@shift.click> Co-authored-by: klensy <klensy@users.noreply.github.com> Co-authored-by: Jack Huey <31162821+jackh726@users.noreply.github.com> Co-authored-by: bjorn3 <17426603+bjorn3@users.noreply.github.com> Co-authored-by: hkalbasi <hamidrezakalbasi@protonmail.com> Co-authored-by: bors <bors@rust-lang.org> Co-authored-by: Sven Marnach <sven@mozilla.com> Co-authored-by: Rémy Rakic <remy.rakic+github@gmail.com> Co-authored-by: aissata <aimaiga2@gmail.com>

pnkfelix · 2023-11-07T18:02:16Z

I am not sure that wg-parallel-rustc was anticipating there being this much of a regression here.
asked for followup in zulip for wg-parallel-rustc
not marking as triaged

SparrowLii · 2023-11-08T00:59:16Z

@pnkfelix Sorry for not explaining in the MCP. We decided in the working group meeting early this year that wall-time is a better choice to measuring the perf result of parallel front end. Therefore, the measurement we choose is wall-time+non-relevent. (You may also need to exclude secondary Categories). In this case, the regression of #117435 should be 1.65%

pnkfelix · 2023-11-08T14:32:24Z

after followup in zulip for wg-parallel-rustc, they clarified that their expectation of a 1.65% regression was based on wall-time that includes so-called "non-relevant results". (Our default rustc-perf presentation filters out the non-relevant results, which will boost outliers and thus inflate the true mean effect.)
concluded that this was anticipated, and marking as triaged

@rustbot label: +perf-regression-triaged

SparrowLii · 2023-11-09T00:54:08Z

Thanks!

RalfJung · 2023-11-11T10:03:57Z

src/bootstrap/src/lib.rs

@@ -77,7 +77,7 @@ const LLD_FILE_NAMES: &[&str] = &["ld.lld", "ld64.lld", "lld-link", "wasm-ld"];
 ///
 /// If you make any major changes (such as adding new values or changing default values), please
 /// ensure that the associated PR ID is added to the end of this list.
-pub const CONFIG_CHANGE_HISTORY: &[usize] = &[115898, 116998];
+pub const CONFIG_CHANGE_HISTORY: &[usize] = &[115898, 116998, 117435];


When you do this, please explain in the PR description what we have to do to update our configs. I was sent here by bootstrap ("WARNING: there have been changes to x.py since you last updated") and now I have no idea what I'm supposed to do.

enable parallel rustc in nightly builds

ab8101d

rustbot assigned oli-obk Oct 31, 2023

SparrowLii requested review from jackh726 and pietroalbini October 31, 2023 08:49

update bootstrap change history

bf5fb76

bjorn3 reviewed Oct 31, 2023

View reviewed changes

src/bootstrap/src/core/config/config.rs Show resolved Hide resolved

SparrowLii changed the title ~~enable parallel rustc in nightly builds~~ enable parallel rustc front end in nightly builds Oct 31, 2023

update config.example.toml

248dd14

oli-obk approved these changes Oct 31, 2023

View reviewed changes

SparrowLii mentioned this pull request Nov 1, 2023

Tracking Issue for Parallel Rustc Front-end #113349

Open

28 tasks

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Nov 3, 2023

nnethercote mentioned this pull request Nov 3, 2023

Add parallel-rustc blog post. rust-lang/blog.rust-lang.org#1164

Merged

klensy reviewed Nov 3, 2023

View reviewed changes

src/ci/run.sh Show resolved Hide resolved

This comment was marked as resolved.

Sign in to view

bors added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Nov 3, 2023

rustbot added the T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. label Nov 6, 2023

rustbot assigned petrochenkov and unassigned oli-obk Nov 6, 2023

rustbot assigned davidtwco and unassigned petrochenkov Nov 6, 2023

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Nov 6, 2023

bors added the merged-by-bors This PR was explicitly merged by bors. label Nov 6, 2023

bors merged commit f9b6446 into rust-lang:master Nov 6, 2023
12 checks passed

rustbot added this to the 1.75.0 milestone Nov 6, 2023

This was referenced Nov 6, 2023

Weekly cargo update #112865

Merged

Add a new download-ci-llvm = if-unchanged option and enable it by default for profile = codegen #116881

Merged

LLVM Bitcode Linker: A self contained linker for nvptx and other targets #117458

Merged

rustbot added the perf-regression Performance regression. label Nov 6, 2023

github-actions bot mentioned this pull request Nov 7, 2023

Automated pull from upstream master ferrocene/ferrocene#89

Merged

rustbot added the perf-regression-triaged The performance regression has been triaged. label Nov 8, 2023

RalfJung reviewed Nov 11, 2023

View reviewed changes

Alexendoo mentioned this pull request Nov 24, 2023

Release nightly compilers with ability to internally parallelize #59667

Closed

bjorn3 mentioned this pull request Nov 24, 2023

[WIP] make nightly compilers able to parallelize #101566

Closed

SparrowLii mentioned this pull request Nov 25, 2023

refactor WorkerLocal for parallel compiler #109478

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

enable parallel rustc front end in nightly builds #117435

enable parallel rustc front end in nightly builds #117435

SparrowLii commented Oct 31, 2023 •

edited

rustbot commented Oct 31, 2023

rustbot commented Oct 31, 2023

oli-obk left a comment

lqd commented Nov 1, 2023 •

edited

oli-obk commented Nov 1, 2023

SparrowLii commented Nov 3, 2023 •

edited

bors commented Nov 3, 2023

matthiaskrgr commented Nov 3, 2023

bors commented Nov 3, 2023

This comment was marked as resolved.

davidtwco commented Nov 6, 2023

bors commented Nov 6, 2023

bors commented Nov 6, 2023

bors commented Nov 6, 2023

rust-timer commented Nov 6, 2023

pnkfelix commented Nov 7, 2023

SparrowLii commented Nov 8, 2023 •

edited

pnkfelix commented Nov 8, 2023

SparrowLii commented Nov 9, 2023

RalfJung Nov 11, 2023 •

edited

enable parallel rustc front end in nightly builds #117435

enable parallel rustc front end in nightly builds #117435

Conversation

SparrowLii commented Oct 31, 2023 • edited

rustbot commented Oct 31, 2023

rustbot commented Oct 31, 2023

oli-obk left a comment

Choose a reason for hiding this comment

lqd commented Nov 1, 2023 • edited

oli-obk commented Nov 1, 2023

SparrowLii commented Nov 3, 2023 • edited

bors commented Nov 3, 2023

matthiaskrgr commented Nov 3, 2023

bors commented Nov 3, 2023

This comment was marked as resolved.

davidtwco commented Nov 6, 2023

bors commented Nov 6, 2023

bors commented Nov 6, 2023

bors commented Nov 6, 2023

rust-timer commented Nov 6, 2023

Overall result: ❌ regressions - ACTION NEEDED

Instruction count

Max RSS (memory usage)

Cycles

Binary size

pnkfelix commented Nov 7, 2023

SparrowLii commented Nov 8, 2023 • edited

pnkfelix commented Nov 8, 2023

SparrowLii commented Nov 9, 2023

RalfJung Nov 11, 2023 • edited

Choose a reason for hiding this comment

SparrowLii commented Oct 31, 2023 •

edited

lqd commented Nov 1, 2023 •

edited

SparrowLii commented Nov 3, 2023 •

edited

SparrowLii commented Nov 8, 2023 •

edited

RalfJung Nov 11, 2023 •

edited