Enable zstd for debug compression. #125642

khuey · 2024-05-28T03:12:28Z

Set LLVM_ENABLE_ZSTD alongside LLVM_ENABLE_ZLIB so that --compress-debug-sections=zstd is an option.

See #120953

rustbot · 2024-05-28T03:12:36Z

rustbot has assigned @onur-ozkan.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

khuey · 2024-05-28T04:35:35Z

To elaborate a bit here, in #99464 LLVM was updated to version 15, the first to have LLVM_ENABLE_ZSTD. That was explicitly disabled (to avoid pulling in a dependency, according to the PR).

That was fine until #124129 made rust-lld the default linker. Now, because LLVM_ENABLE_ZSTD=OFF disables --compress-debug-sections=zstd, rust-lld is (potentially) a regression against the system linker.

So here I'd like to turn this on so rust-lld and thus the out of the box default is at least as capable as the system linker here.

rustbot · 2024-05-28T04:35:39Z

This PR changes how LLVM is built. Consider updating src/bootstrap/download-ci-llvm-stamp.

lqd · 2024-05-28T09:07:35Z

How are you producing this zstd-compressed debuginfo without building your own rustc/llvm to add zstd support in -Zdebuginfo-compression?

khuey · 2024-05-28T13:55:05Z

-C link-arg=-Wl,--compress-debug-sections=zstd

lqd · 2024-05-28T14:16:21Z

That's what I expected, and it feels suboptimal that rustc/llvm think zstd compression is disabled here: we could disable self-contained linking in that case, or could have emitted a warning/error. I'm sure you're already aware that the linker change is easily controllable without more link args, to fall back to system lld (-Clink-self-contained=-linker -Zunstable-options) or bfd (-Zlinker-features=-lld).

khuey · 2024-05-28T14:38:52Z

It does emit an error: note: rust-lld: error: --compress-debug-sections: LLVM was not built with LLVM_ENABLE_ZSTD or did not find zstd at build time

I'm aware that I can explicitly opt into the system linker, but this is trivial to fix in rust-lld so I feel we may as well fix it so things just work.

lqd · 2024-05-28T14:42:15Z

but this is trivial to fix in rust-lld so I feel we may as well fix it so things just work

I'm aware and I agree, I just don't know the rationale in bootstrap yet, so there may be concerns about adding the dependency and we'll have to wait for more info from t-bootstrap / infra.

workingjubilee · 2024-05-28T18:13:18Z

@nikic was the one who toggled this off. Based on the comments at the time, am I correct in I believing there was no particular reason beyond "let's not accidentally depend on libzstd.so"? Thus it should be fine to do it as long as we are doing it on purpose?

nikic · 2024-05-28T18:15:38Z

@nikic was the one who toggled this off. Based on the comments at the time, am I correct in I believing there was no particular reason beyond "let's not accidentally depend on libzstd.so"? Thus it should be fine to do it as long as we are doing it on purpose?

I think we still wouldn't want a dependency on libzstd.so, which probably has a non-negligible chance of not being already installed? If we enable zstd we should probably statically link it.

workingjubilee · 2024-05-28T18:31:34Z

Okay, "we want everything to just be rolled up into our LLVM" sounds fine to me.

onur-ozkan · 2024-05-28T19:44:53Z

src/bootstrap/src/core/build_steps/llvm.rs

-        // Disable zstd to avoid a dependency on libzstd.so.
-        cfg.define("LLVM_ENABLE_ZSTD", "OFF");
-
+        // Libraries for ELF compression.
        if !target.is_windows() {
            cfg.define("LLVM_ENABLE_ZLIB", "ON");
+            cfg.define("LLVM_ENABLE_ZSTD", "ON");
        } else {
            cfg.define("LLVM_ENABLE_ZLIB", "OFF");
+            cfg.define("LLVM_ENABLE_ZSTD", "OFF");


How about making this configurable in config.toml (off by default) instead of constantly setting it as "ON" ?

The point of this PR to enable it by default. If it's not enabled by default it's easier to disable the self-contained linker than to build rustc, including LLVM, from scratch.

I do not understand why we would make something configurable without a specific reason to do so.

Doesn't this require an additional dependency to be installed? Also, according to the LLVM docs when it's not set LLVM build enables zstd only if it exists on the system. Could that be an option here?

Fwiw using "FORCE_ON" makes the build raise a hard error when zstd can't be found on the system.

ref: https://llvm.org/docs/CMake.html

Doesn't this require an additional dependency to be installed?

Raising the requirements to build our LLVM recipe seems fine given that we support using an external LLVM. Thus we can't assume that the external LLVM has zlib or zstd support (as both are optional) and must query it at runtime (which we do), and we support building an LLVM with arbitrary configuration already: just bring your own.

Doesn't this require an additional dependency to be installed?

As the patch stands today, it requires libzstd.so to be available at runtime if libzstd was available at build time. It doesn't actually require libzstd to be available at build time (because it uses ON instead of FORCE_ON).

There's a matrix of possible outcomes here.

ON FORCE_ON

No LLVM_USE_STATIC_ZSTD Not required at build time, required at runtime if present at build time, silently disabled if not present at build time Required at build time and runtime

LLVM_USE_STATIC_ZSTD Not required at build time or runtime, silently disabled if not present at build time Required at build time, not at runtime

Where we're currently in the top left cell.

onur-ozkan · 2024-05-29T05:24:59Z

If we enable zstd we should probably statically link it.

Once this is done, I am okay with landing it.

khuey · 2024-05-29T21:03:11Z

Unfortunately statically linking zstd is not currently as simple as setting LLVM_USE_STATIC_ZSTD=True. llvm-config outputs the flags for dynamic linking regardless of the value of LLVM_USE_STATIC_ZSTD.

khuey · 2024-05-30T01:21:55Z

Alright with llvm/llvm-project#93754 and those two additional changesets everything gets linked statically.

onur-ozkan · 2024-05-30T05:00:50Z

r? compiler

fee1-dead · 2024-05-30T09:56:57Z

r? compiler

bors · 2024-07-29T21:01:03Z

💔 Test failed - checks-actions

lqd · 2024-07-29T21:17:26Z

The builder hit Failed connect to ftp.gnu.org:443; Network is unreachable while downloading gcc. Let's try again

@bors try

Enable zstd for debug compression. Set LLVM_ENABLE_ZSTD alongside LLVM_ENABLE_ZLIB so that --compress-debug-sections=zstd is an option. See rust-lang#120953

bors · 2024-07-29T21:18:38Z

⌛ Trying commit 069f0bb with merge 5f62440...

bors · 2024-07-29T21:33:40Z

💔 Test failed - checks-actions

khuey · 2024-07-30T11:11:02Z

Fetching from ftp.gnu.org failed again. Not sure what else to do besides re-retry.

jieyouxu · 2024-07-30T11:24:26Z

@bors try

bors · 2024-07-30T11:25:37Z

⌛ Trying commit 069f0bb with merge a0307d2...

Enable zstd for debug compression. Set LLVM_ENABLE_ZSTD alongside LLVM_ENABLE_ZLIB so that --compress-debug-sections=zstd is an option. See rust-lang#120953

bors · 2024-07-30T11:28:43Z

💔 Test failed - checks-actions

lqd · 2024-07-30T11:54:59Z

https://vault.centos.org/centos/7/os/x86_64/repodata/repomd.xml: [Errno 14] HTTPS Error 502 - Bad Gateway

@bors try

bors · 2024-07-30T11:56:10Z

⌛ Trying commit 069f0bb with merge aba840d...

Enable zstd for debug compression. Set LLVM_ENABLE_ZSTD alongside LLVM_ENABLE_ZLIB so that --compress-debug-sections=zstd is an option. See rust-lang#120953

bors · 2024-07-30T11:59:06Z

💔 Test failed - checks-actions

khuey · 2024-07-30T16:48:24Z

sigh, more network failures.

Is this a common occurrence?

jieyouxu · 2024-07-30T16:50:19Z

@bors try

Kobzol · 2024-07-30T16:51:13Z

It happens from time to time, but it doesn't seem happening for other PRs other than this one at the moment. But that could also be caused by the fact that this PR needs to rebuild Docker images (normally the Docker build is cached).

bors · 2024-07-30T16:51:30Z

⌛ Trying commit 069f0bb with merge 33c56f6...

Enable zstd for debug compression. Set LLVM_ENABLE_ZSTD alongside LLVM_ENABLE_ZLIB so that --compress-debug-sections=zstd is an option. See rust-lang#120953

bors · 2024-07-30T17:59:42Z

💔 Test failed - checks-actions

khuey · 2024-07-30T18:19:41Z

Ok, that's at least relevant to the PR.

khuey · 2024-07-30T23:56:32Z

Would appreciate another try run.

jieyouxu · 2024-07-31T00:06:39Z

@bors try

bors · 2024-07-31T00:07:52Z

⌛ Trying commit 25813dd with merge 79b326d...

Enable zstd for debug compression. Set LLVM_ENABLE_ZSTD alongside LLVM_ENABLE_ZLIB so that --compress-debug-sections=zstd is an option. See rust-lang#120953

rustbot assigned onur-ozkan May 28, 2024

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) labels May 28, 2024

workingjubilee mentioned this pull request May 28, 2024

Support zstd-compressed ELF sections. rust-lang/backtrace-rs#626

Open

khuey marked this pull request as ready for review May 28, 2024 04:35

onur-ozkan reviewed May 28, 2024

View reviewed changes

onur-ozkan added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels May 28, 2024

khuey force-pushed the zstd branch from 169f982 to c638c09 Compare May 29, 2024 16:55

rustbot added the T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. label May 30, 2024

rustbot assigned fee1-dead and unassigned onur-ozkan May 30, 2024

onur-ozkan added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels May 30, 2024

This comment has been minimized.

Sign in to view

Don't install zstd until after the first LLVM build.

25813dd

	ON	FORCE_ON
No LLVM_USE_STATIC_ZSTD	Not required at build time, required at runtime if present at build time, silently disabled if not present at build time	Required at build time and runtime
LLVM_USE_STATIC_ZSTD	Not required at build time or runtime, silently disabled if not present at build time	Required at build time, not at runtime

Enable zstd for debug compression. #125642

Are you sure you want to change the base?

Enable zstd for debug compression. #125642

Conversation

khuey commented May 28, 2024 • edited by jieyouxu Loading

rustbot commented May 28, 2024

khuey commented May 28, 2024

rustbot commented May 28, 2024

lqd commented May 28, 2024

khuey commented May 28, 2024

lqd commented May 28, 2024

khuey commented May 28, 2024

lqd commented May 28, 2024

workingjubilee commented May 28, 2024

nikic commented May 28, 2024

workingjubilee commented May 28, 2024

onur-ozkan May 28, 2024 • edited Loading

Choose a reason for hiding this comment

khuey May 28, 2024

Choose a reason for hiding this comment

workingjubilee May 28, 2024

Choose a reason for hiding this comment

onur-ozkan May 28, 2024

Choose a reason for hiding this comment

workingjubilee May 29, 2024 • edited Loading

Choose a reason for hiding this comment

khuey May 29, 2024

Choose a reason for hiding this comment

onur-ozkan commented May 29, 2024

khuey commented May 29, 2024

khuey commented May 30, 2024

onur-ozkan commented May 30, 2024

fee1-dead commented May 30, 2024

bors commented Jul 29, 2024

lqd commented Jul 29, 2024 • edited Loading

bors commented Jul 29, 2024

This comment has been minimized.

bors commented Jul 29, 2024

khuey commented Jul 30, 2024

jieyouxu commented Jul 30, 2024

bors commented Jul 30, 2024

This comment has been minimized.

bors commented Jul 30, 2024

lqd commented Jul 30, 2024

bors commented Jul 30, 2024

This comment has been minimized.

bors commented Jul 30, 2024

khuey commented Jul 30, 2024

jieyouxu commented Jul 30, 2024

Kobzol commented Jul 30, 2024

bors commented Jul 30, 2024

This comment has been minimized.

bors commented Jul 30, 2024

khuey commented Jul 30, 2024

khuey commented Jul 30, 2024

jieyouxu commented Jul 31, 2024

bors commented Jul 31, 2024

khuey commented May 28, 2024 •

edited by jieyouxu

Loading

onur-ozkan May 28, 2024 •

edited

Loading

workingjubilee May 29, 2024 •

edited

Loading

lqd commented Jul 29, 2024 •

edited

Loading