Add dist.compression-profile option to control compression speed

[pietroalbini]

PR #108534 reduced the size of compressed archives, but (as expected) it also resulted in way longer compression times and memory usage during compression.

It's desirable to keep status quo (smaller archives but more CI usage), but it should also be configurable so that downstream users don't have to waste that much time on CI. As a data point, this resulted in doubling the time of Ferrocene's dist jobs, and required us to increase the RAM allocation for one of such jobs.

This PR adds a new config.toml setting, dist.compression-profile. The values can be:

• fast: equivalent to the gzip and xz preset of "1"
• balanced: equivalent to the gzip and xz preset of "6" (the CLI defaults as far as I'm aware)
• best: equivalent to the gzip present of "9", and our custom xz profile

The default has also been moved back to balanced, to try and avoid the compression time regression for downstream users. I don't feel too strongly on the default, and I'm open to changing it.

Also, for the best profile the XZ settings do not match the "9" preset used by the CLI, and it might be confusing. Should we create a custom-rustc-ci/ultra profile for that?

r? @Mark-Simulacrum

[pietroalbini]

I recommend hiding the whitespace while reviewing this change, I had to change indentation of a bunch of code :(

[Mark-Simulacrum]

Do we think it makes sense to continue maintaining these options here and a semi-duplicate in rust-lang/promote-release for the recompression support?

Mostly raising the question -- I think I'm not sure yet myself. Will review in detail soon.

[pietroalbini]

Do we think it makes sense to continue maintaining these options here and a semi-duplicate in rust-lang/promote-release for the recompression support?

I think the difference between the support here and in promote-release is that promote-release is specific to our (Rust project) builds and releases, while this knob would be relied upon by downstream users, for whom it doesn't make much sense to use promote-release.

[Mark-Simulacrum]

Right, I mean that we could (for example) only produce .tar files in rust-lang/rust's bootstrap code, or perhaps only the fast profile out of those here. Then promote-release or others can recompress if needed. It seems to me that if you're running x.py dist purely locally, then probably no compression or super-fast compression is what you want; if you're distributing you probably want balanced or best (depending on number of users).

Long-term I suspect that maybe zstd is the right thing for rust-lang/rust to produce, presumably that can give us fast, relatively well-compressed artifacts, and maybe works for ~everyone. Then from there recompression is maybe easy enough.

We could also consider a compression-pipe-through=["program", "args"] or something as another option.

[Mark-Simulacrum]

r=me (maybe with @bjorn3's comment resolved) but I think we should continue thinking about the right shape here.

[pietroalbini]

@bors r=Mark-Simulacrum rollup

[bors]

📌 Commit 0177176 has been approved by Mark-Simulacrum

It is now in the queue for this repository.