Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Discard LLVM modules earlier when performing ThinLTO #56487
Currently ThinLTO is performed by first compiling all modules (and keeping them in memory), and then serializing them into ThinLTO buffers in a separate, synchronized step. Modules are later read back from ThinLTO buffers when running the ThinLTO optimization pipeline.
We can also find the following comment in
I don't think that what is suggested here is the right approach: One of the primary benefits of using ThinLTO over ordinary LTO is that it's not necessary to keep all the modules (merged or not) in memory for the duration of the linking step.
However, we currently don't really make use of this (at least for crate-local ThinLTO), because we keep all modules in memory until the start of the LTO step. This PR changes the implementation to instead perform the serialization into ThinLTO buffers directly after the initial optimization step.
Most of the changes here are plumbing to separate out fat and thin lto handling in
Whoa this is a great idea! This seems like the correct strategy for incremental too, although I forget if that takes different paths.
This also makes me think that we should link in fat LTO ASAP instead of synchronizing and then linking as it'd allow pipelining a bit ideally. In any case that's a patch for another time!
I'll take a closer look later, but I'm curious on the perf impact here too, it should both make builds faster (slightly) and decrease peak memory usage in theory
added a commit
this pull request
Dec 4, 2018
I guess that means that LLVM memory usage is dominated by rustc memory usage, at least for the build types used here (opt w/o debuginfo), so it has no impact on max-rss. Unfortunately I was not able to get
I've got massif to run (need to directly call the jemalloc free rustc, no
The first hump is peak optimization, the second hump is peak LTO. So in this case peak memory usage is moved from the optimization stage to the LTO stage, but it ultimately does not make much of a difference.
Click to expand the log.