Skip to content

/rust

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rustc: Move bytecode compression into codegen #45399

Merged
merged 1 commit into from Oct 22, 2017

Conversation

Reviewers

@michaelwoerister michaelwoerister

Assignees

@michaelwoerister michaelwoerister

Labels
S-waiting-on-author
Projects
None yet
Milestone
No milestone
6 participants
@alexcrichton @rust-highfive @michaelwoerister @bors @kennytm @arielb1
@alexcrichton
Copy link
Member

alexcrichton commented Oct 20, 2017

This commit moves compression of the bytecode from the link module to the
write module, namely allowing it to be (a) cached by incremental compilation
and (b) produced in parallel. The parallelization may show up as some nice wins
during normal compilation and the caching in incremental mode should be
beneficial for incremental compiles! (no more need to recompress the entire
crate's bitcode on all builds)

@rust-highfive rust-highfive assigned arielb1 Oct 20, 2017

@rust-highfive

This comment has been minimized.

Sign in to view
Copy link
Collaborator

rust-highfive commented Oct 20, 2017

r? @arielb1

(rust_highfive has picked a reviewer for you, use r? to override)
@alexcrichton

This comment has been minimized.

Sign in to view
Copy link
Member Author

alexcrichton commented Oct 20, 2017

r? @michaelwoerister

I'm sort of perpetually unhappy with how no matter what I do the management of filenames in the backend is confusing, but hopefully it's a little less confusing than before?

@rust-highfive rust-highfive assigned michaelwoerister and unassigned arielb1 Oct 20, 2017

@kennytm kennytm added the S-waiting-on-author label Oct 20, 2017

@michaelwoerister

michaelwoerister approved these changes Oct 20, 2017
View changes

Copy link
Contributor

michaelwoerister left a comment

Great find! The new parallelization and caching should definitely be a win here.

The only downside I can see is that this will increase the size of the incr. cache. But hopefully we can get rid of bitcode compression entirely at some point by switching to MIR-only RLIBs.

r=me modulo nits and fixing the test case.
src/librustc_trans/back/write.rs Outdated
@@ -1157,17 +1183,20 @@ fn execute_work_item(cgcx: &CodegenContext,
err));
}
}
*slot = Some(obj_out);

This comment has been minimized.

Sign in to view
Copy link
@michaelwoerister

michaelwoerister Oct 20, 2017

Contributor

Couldn't you do this already in the match arms above?
@alexcrichton

This comment has been minimized.

Sign in to view
Copy link
Member Author

alexcrichton commented Oct 20, 2017

@michaelwoerister

The only downside I can see is that this will increase the size of the incr. cache

Oh actually we're no longer storing uncompressed bytecode in the cache (as that's not needed for the linking stage), but default builds (those without fancy --emit flags) switch from storing object/bytecode to object/compressed bytecode
@michaelwoerister

This comment has been minimized.

Sign in to view
Copy link
Contributor

michaelwoerister commented Oct 20, 2017

Oh actually we're no longer storing uncompressed bytecode in the cache

We'll probably have to start doing that for ThinLTO, right?
@alexcrichton

This comment has been minimized.

Sign in to view
Copy link
Member Author

alexcrichton commented Oct 20, 2017

Heh I've thought very little about how ThinLTO interacts with incremental compilation :)

In theory there's a lot of possible wins to gain there (fully incremental release builds) but I think we've got a good bit of support on the LLVM side of things to sort out before that works.

@alexcrichton alexcrichton force-pushed the alexcrichton:compress-parallel branch from 5a183e4 to 3bfa312 Oct 20, 2017

@alexcrichton

This comment has been minimized.

Sign in to view
Copy link
Member Author

alexcrichton commented Oct 20, 2017

@bors: r=michaelwoerister
@bors

This comment has been minimized.

Sign in to view
Copy link
Contributor

bors commented Oct 20, 2017

📌 Commit 3bfa312 has been approved by michaelwoerister

@kennytm kennytm added S-waiting-on-bors and removed S-waiting-on-author labels Oct 20, 2017

@alexcrichton alexcrichton force-pushed the alexcrichton:compress-parallel branch 2 times, most recently from 2effe64 to 9963d13 Oct 20, 2017

@alexcrichton

This comment has been minimized.

Sign in to view
Copy link
Member Author

alexcrichton commented Oct 20, 2017

@bors: r=michaelwoerister
@bors

This comment has been minimized.

Sign in to view
Copy link
Contributor

bors commented Oct 20, 2017

📌 Commit 9963d13 has been approved by michaelwoerister
@bors

This comment has been minimized.

Sign in to view
Copy link
Contributor

bors commented Oct 21, 2017

☔️ The latest upstream changes (presumably #45348) made this pull request unmergeable. Please resolve the merge conflicts.

@kennytm kennytm added S-waiting-on-author and removed S-waiting-on-bors labels Oct 21, 2017

@alexcrichton
rustc: Move bytecode compression into codegen 
This commit moves compression of the bytecode from the `link` module to the
`write` module, namely allowing it to be (a) cached by incremental compilation
and (b) produced in parallel. The parallelization may show up as some nice wins
during normal compilation and the caching in incremental mode should be
beneficial for incremental compiles! (no more need to recompress the entire
crate's bitcode on all builds)
Loading status checks…
8197a0b

@alexcrichton alexcrichton force-pushed the alexcrichton:compress-parallel branch from 9963d13 to 8197a0b Oct 21, 2017

@alexcrichton

This comment has been minimized.

Sign in to view
Copy link
Member Author

alexcrichton commented Oct 21, 2017

@bors: r=michaelwoerister
@bors

This comment has been minimized.

Sign in to view
Copy link
Contributor

bors commented Oct 21, 2017

📌 Commit 8197a0b has been approved by michaelwoerister

bors added a commit that referenced this pull request Oct 21, 2017

@bors
Auto merge of #45399 - alexcrichton:compress-parallel, r=michaelwoeri… 
…ster

rustc: Move bytecode compression into codegen

This commit moves compression of the bytecode from the `link` module to the
`write` module, namely allowing it to be (a) cached by incremental compilation
and (b) produced in parallel. The parallelization may show up as some nice wins
during normal compilation and the caching in incremental mode should be
beneficial for incremental compiles! (no more need to recompress the entire
crate's bitcode on all builds)
Loading status checks…
9ea3878
@bors

This comment has been minimized.

Sign in to view
Copy link
Contributor

bors commented Oct 22, 2017

☀️ Test successful - status-appveyor, status-travis
Approved by: michaelwoerister
Pushing 9ea3878 to master...

@bors bors merged commit 8197a0b into rust-lang:master Oct 22, 2017
2 checks passed

2 checks passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
homu Test successful
Details
@michaelwoerister

This comment has been minimized.

Sign in to view
Copy link
Contributor

michaelwoerister commented Oct 23, 2017

This seems to have brought some nice performance improvements: http://perf.rust-lang.org/compare.html?start=4279e2b4c14dc90191595c97ad3a019d618e7158&end=9ea3878dfec9b20993eaebfdb7566b29e49c52ab&stat=wall-time

But they were eaten up immediately by switching to ThinLTO in the next PR 😁😭

@alexcrichton alexcrichton deleted the alexcrichton:compress-parallel branch Oct 28, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.