codegen-units + ThinLTO is not as good as codegen-units = 1

[nagisa]

We recently had a fair amount of reports about code generation quality drop. One of the recent causes for the quality drop is the enablement of codegen-units and ThinLTO.

It seems that ThinLTO is not capable of producing results matching those obtained by compiling without codegen-units in the first place.

The list of known reports follows:

• #47665
• #47561
• #47356
• #47062 (specifically this comment)
• #47321
• #47770
• #53833

Improvements to ThinLTO quality are inbound with the soon-to-happen LLVM upgrade(s), however those do not help sufficiently, it would be nice to figure out why ThinLTO is not doing good enough job.

cc @alexcrichton @nikomatsakis

[nikomatsakis]

Thanks for filing this @nagisa =)

[alexcrichton]

Indeed thanks! I'll try to take a closer look at this when we've upgraded LLVM

[matthiaskrgr]

By the way there is a great talk about how thinlto is designed here: https://www.youtube.com/watch?v=p9nH2vZ2mNo in case people are curious. :)

[robsmith11]

Matrix multiplication is slower with thinlto + multiple codege-units using https://github.com/bluss/matrixmultiply .

I can create a minimal example if needed.

[johnthagen]

I've always thought that there should be another Cargo profile, something like:

# The publish profile, used for `cargo build --publish`.
[profile.publish]
# (...) everything else the same as profile.release except:
lto = true        # Enable full link-time optimization.
codegen-units = 1 # Use only 1 codegen-unit to enable full optimizations.

Because I feel like there should be a distinction between release builds the developer compiles on their local machine during development (not debug builds, but "fast" release builds) and truly publishable builds (like, for example the version of Firefox that is released for public consumption) in which case sacrificing build time once is more acceptable.

I realize the status-quo for C/C++ is to also not enable LTO by default, but it just seems strange to me to have to opt into these kinds of performance enhancements when the cost (for published binaries) is a one-time compile time cost.

[HadrienG2]

I think "publish" is uncomfortably close to "release". But I could get behind a "debug/optimize/release" terminology proposal.

[forrestthewoods]

Historically I've used debug, internal, release, retail.

Plus a few variations with "add-ons" such as "Retail-Logging" or "Retail-Instrumented".

For Rust instead of 'Retail' I'd propose MaxSpeed. Whatever it's called, a profile with lto=true and codegen-units=1 is definitely a good idea!

[brson]

@johnthagen I agree that today's 'release' profile seems to have two use cases that want different configurations. Is it possible to create custom cargo profiles? Is there an upstream cargo issue for this?

[johnthagen]

@brson It looks like it's not yet implemented, but it is has been discussed for several years.

• Issue Request (from 2015): rust-lang/cargo#2007
• @aturon wrote in Feb 2018 about deprecating profiles and using "workflows" instead: rust-lang/rfcs#2282 (comment)
• Another related blog post by @aturon: https://aturon.github.io/2018/04/05/workflows/

Perhaps @matklad has some more up-to-date information on this?

[matklad]

My understanding is that "custom profiles" are pretty far-away at this moment (we need to do profile overrides first), however we do have config profiles nightly features, which allows overriding profile via .cargo/config. This might be used, for example, to specify codegen-units=1 on the build-server which produces release artifacts.