Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

debuginfo: add compiler option to allow compressed debuginfo sections #115358

Merged
merged 6 commits into from Sep 9, 2023
Merged

debuginfo: add compiler option to allow compressed debuginfo sections #115358

merged 6 commits into from Sep 9, 2023

Conversation

durin42
Copy link
Contributor

@durin42 durin42 commented Aug 29, 2023 •
edited

LLVM already supports emitting compressed debuginfo. In debuginfo=full builds, the debug section is often a large amount of data, and it typically compresses very well (3x is not unreasonable.) We add a new knob to allow debuginfo to be compressed when the matching LLVM functionality is present. Like clang, if a known-but-disabled compression mechanism is requested, we disable compression and emit uncompressed debuginfo sections.

The API is different enough on older LLVMs we just pretend the support
is missing on LLVM older than 16.

@rust-log-analyzer

This comment has been minimized.

Sign in to view
@rustbot
Copy link
Collaborator

rustbot commented Aug 29, 2023

r? @oli-obk

(rustbot has picked a reviewer for you, use r? to override)

@rustbot rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. labels Aug 29, 2023
@rust-log-analyzer

This comment has been minimized.

Sign in to view
@durin42
Copy link
Contributor Author

durin42 commented Aug 30, 2023 via email

@oli-obk
Copy link
Contributor

oli-obk commented Aug 30, 2023

is this failure because our CI does not have zlib installed?

@durin42
Copy link
Contributor Author

durin42 commented Aug 30, 2023

That's my assumption - I'm not familiar enough with the Rust project's LLVM setup to know how to check.

I'm also not sure how to make the test gracefully degrade if zlib is missing. :(

@tmiasko
Copy link
Contributor

tmiasko commented Aug 30, 2023

If compression is requested but not available, a warning might be appropriate. Then you could check for presence of the warning.

@durin42
Copy link
Contributor Author

durin42 commented Aug 30, 2023

Right - I'm not sure how I should be printing a warning from where I've got access to the LLVM API in question. Any pointers?

Alternatively, if I can run ld.lld in the test, I can check what compression methods it supports.

@durin42
Copy link
Contributor Author

durin42 commented Aug 30, 2023

Oh, I might have had the dumb. I may see what I missed on my last attempt to print the warning...

@durin42
Copy link
Contributor Author

durin42 commented Aug 30, 2023

Ah, I remember the stumbling block I hit: rustc_codegen_llvm isn't currently used by rustc_session, and that seems like a layering violation to me naively, so I can't call into the FFI functions to look if the requested compression is available from there. But by the time we're in a location where we can check that in rustc_codegen_llvm, I don't seem to have a way to emit warnings at all (eg from target_machine_factory) - so I'm stumped where I should do this - if there's an example on how to emit the warning based on the codegen backend, I'm sadly missing it. :(

Any hints?

@durin42
Copy link
Contributor Author

durin42 commented Aug 30, 2023

Manishearth got me sorted as far as I can tell on printing the diagnostic, and I came up with some Shell Crimes for the test in the makefile, so I think this is ready for review.

@rust-log-analyzer

This comment has been minimized.

Sign in to view
@rust-log-analyzer

This comment has been minimized.

Sign in to view
oli-obk
oli-obk requested changes Sep 5, 2023
View reviewed changes
Copy link
Contributor

@oli-obk oli-obk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just a style nit, then this lgtm

compiler/rustc_session/src/options.rs Outdated Show resolved Hide resolved
@rust-log-analyzer

This comment has been minimized.

Sign in to view
@oli-obk
Copy link
Contributor

oli-obk commented Sep 6, 2023

@bors r+ rollup

@bors
Copy link
Contributor

bors commented Sep 6, 2023

📌 Commit 8577e3e has been approved by oli-obk

It is now in the queue for this repository.

@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Sep 6, 2023
fmease added a commit to fmease/rust that referenced this pull request Sep 6, 2023
@fmease
Rollup merge of rust-lang#115358 - durin42:compress-debuginfo, r=oli-obk
6023808 
debuginfo: add compiler option to allow compressed debuginfo sections

LLVM already supports emitting compressed debuginfo. In debuginfo=full builds, the debug section is often a large amount of data, and it typically compresses very well (3x is not unreasonable.) We add a new knob to allow debuginfo to be compressed when the matching LLVM functionality is present. Like clang, if a known-but-disabled compression mechanism is requested, we disable compression and emit uncompressed debuginfo sections.

The API is different enough on older LLVMs we just pretend the support
is missing on LLVM older than 16.
@fmease fmease mentioned this pull request Sep 6, 2023
Closed
@fmease
Copy link
Member

fmease commented Sep 6, 2023

@bors r-
#115606 (comment)

@durin42
sort
a52990b
@durin42
tests: use warning output from rustc to catch missing compression
78d805c 
Using `ld.lld` may have been clever, but that was getting the /system/
ld.lld, not one we may have built as part of building llvm. By using the
warning message coming directly from rustc we now correctly skip the
zlib and zstd tests when the support is missing.
@durin42
Copy link
Contributor Author

durin42 commented Sep 8, 2023

Rebased (thus the force push) and reworked test to actually do what it says on the tin.

@durin42
Copy link
Contributor Author

durin42 commented Sep 8, 2023

@rustbot label -S-waiting-on-author +S-waiting-on-review

@rustbot rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Sep 8, 2023
@durin42 durin42 requested a review from oli-obk September 8, 2023 15:36
@oli-obk
Copy link
Contributor

oli-obk commented Sep 8, 2023

@bors r+

@bors
Copy link
Contributor

bors commented Sep 8, 2023

📌 Commit 78d805c has been approved by oli-obk

It is now in the queue for this repository.

@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Sep 8, 2023
@matthiaskrgr matthiaskrgr mentioned this pull request Sep 8, 2023
Merged
bors added a commit to rust-lang-ci/rust that referenced this pull request Sep 8, 2023
@bors
Auto merge of rust-lang#115685 - matthiaskrgr:rollup-t31gowy, r=matth…
b0b8c52 
…iaskrgr

Rollup of 5 pull requests

Successful merges:

 - rust-lang#113807 (Tests crash from inappropriate use of common linkage)
 - rust-lang#115358 (debuginfo: add compiler option to allow compressed debuginfo sections)
 - rust-lang#115630 (Dont suggest use between `use` and cfg attr)
 - rust-lang#115662 (Improve "associated type not found" diagnostics)
 - rust-lang#115673 (Fix sanitize/cfg.rs test)

r? `@ghost`
`@rustbot` modify labels: rollup
@bors bors merged commit aa78b4c into rust-lang:master Sep 9, 2023
11 checks passed
@rustbot rustbot added this to the 1.74.0 milestone Sep 9, 2023
rust-timer added a commit to rust-lang-ci/rust that referenced this pull request Sep 9, 2023
@rust-timer
Unrolled build for rust-lang#115358
c7a90ca 
Rollup merge of rust-lang#115358 - durin42:compress-debuginfo, r=oli-obk

debuginfo: add compiler option to allow compressed debuginfo sections

LLVM already supports emitting compressed debuginfo. In debuginfo=full builds, the debug section is often a large amount of data, and it typically compresses very well (3x is not unreasonable.) We add a new knob to allow debuginfo to be compressed when the matching LLVM functionality is present. Like clang, if a known-but-disabled compression mechanism is requested, we disable compression and emit uncompressed debuginfo sections.

The API is different enough on older LLVMs we just pretend the support
is missing on LLVM older than 16.
@Wyvern
Copy link

Wyvern commented Sep 11, 2023

Just tried this option got warning: unknown debuginfo compression algorithm zstd - will fall back to uncompressed debuginfo, how to use zstd as compression method?

@durin42
Copy link
Contributor Author

durin42 commented Sep 11, 2023

That means the LLVM your rustc was compiled against didn't have zstandard support enabled. zlib is probably already on, maybe try that?

@durin42 durin42 deleted the compress-debuginfo branch October 5, 2023 19:02
This was referenced Feb 11, 2024
Open
Open
fbq pushed a commit to Rust-for-Linux/linux that referenced this pull request Feb 17, 2024
@ojeda @fbq
kbuild: rust: use -Zdebuginfo-compression
ce16e96 
Rust 1.74.0 introduced (unstable) support for the
`-Zdebuginfo-compression` flag, thus use it.

Link: rust-lang/rust#120953
Link: rust-lang/rust#115358
Signed-off-by: Miguel Ojeda <ojeda@kernel.org>
Link: https://lore.kernel.org/r/20240217002622.57322-1-ojeda@kernel.org
fbq pushed a commit to Rust-for-Linux/linux that referenced this pull request Feb 19, 2024
@ojeda @fbq
kbuild: rust: use -Zdebuginfo-compression
3635710 
Rust 1.74.0 introduced (unstable) support for the
`-Zdebuginfo-compression` flag, thus use it.

Link: rust-lang/rust#120953
Link: rust-lang/rust#115358
Signed-off-by: Miguel Ojeda <ojeda@kernel.org>
Link: https://lore.kernel.org/r/20240217002622.57322-1-ojeda@kernel.org
fbq pushed a commit to Rust-for-Linux/linux that referenced this pull request Mar 5, 2024
@ojeda @fbq
kbuild: rust: use -Zdebuginfo-compression
83a1681 
Rust 1.74.0 introduced (unstable) support for the
`-Zdebuginfo-compression` flag, thus use it.

Link: rust-lang/rust#120953
Link: rust-lang/rust#115358
Signed-off-by: Miguel Ojeda <ojeda@kernel.org>
Link: https://lore.kernel.org/r/20240217002622.57322-1-ojeda@kernel.org
fbq pushed a commit to Rust-for-Linux/linux that referenced this pull request Mar 25, 2024
@ojeda @fbq
kbuild: rust: use -Zdebuginfo-compression
0201093 
Rust 1.74.0 introduced (unstable) support for the
`-Zdebuginfo-compression` flag, thus use it.

Link: rust-lang/rust#120953
Link: rust-lang/rust#115358
Signed-off-by: Miguel Ojeda <ojeda@kernel.org>
Link: https://lore.kernel.org/r/20240217002622.57322-1-ojeda@kernel.org
ojeda added a commit to ojeda/linux that referenced this pull request Mar 29, 2024
@ojeda
kbuild: rust: use -Zdebuginfo-compression
6e5574c 
Rust 1.74.0 introduced (unstable) support for the
`-Zdebuginfo-compression` flag, thus use it.

Link: rust-lang/rust#120953
Link: rust-lang/rust#115358
Link: https://lore.kernel.org/r/20240217002622.57322-1-ojeda@kernel.org
Signed-off-by: Miguel Ojeda <ojeda@kernel.org>
ojeda added a commit to Rust-for-Linux/linux that referenced this pull request Mar 29, 2024
@ojeda
kbuild: rust: use -Zdebuginfo-compression
10b592f 
Rust 1.74.0 introduced (unstable) support for the
`-Zdebuginfo-compression` flag, thus use it.

Note that the releases built by the Rust project (i.e. the ones provided
by rustup) do not enable support for zstd in their bundled LLVM (yet,
at least), thus the Rust compiler will warn, but the build will proceed:

    warning: unknown debuginfo compression algorithm zstd - will fall
    back to uncompressed debuginfo

Link: rust-lang/rust#120953
Link: rust-lang/rust#115358
Link: https://lore.kernel.org/r/20240217002622.57322-1-ojeda@kernel.org
[ Added note about zstd support in Rust-provided binaries. ]
Signed-off-by: Miguel Ojeda <ojeda@kernel.org>
ojeda added a commit to Rust-for-Linux/linux that referenced this pull request Apr 2, 2024
@ojeda
kbuild: rust: use -Zdebuginfo-compression
dbef181 
Rust 1.74.0 introduced (unstable) support for the
`-Zdebuginfo-compression` flag, thus use it.

Note that the releases built by the Rust project (i.e. the ones provided
by rustup) do not enable support for zstd in their bundled LLVM (yet,
at least), thus the Rust compiler will warn, but the build will proceed:

    warning: unknown debuginfo compression algorithm zstd - will fall
    back to uncompressed debuginfo

Link: rust-lang/rust#120953
Link: rust-lang/rust#115358
Reviewed-by: Alice Ryhl <aliceryhl@google.com>
Link: https://lore.kernel.org/r/20240217002622.57322-1-ojeda@kernel.org
[ Added note about zstd support in Rust-provided binaries. ]
Signed-off-by: Miguel Ojeda <ojeda@kernel.org>
sweettea pushed a commit to sweettea/btrfs-fscrypt that referenced this pull request Apr 3, 2024
@ojeda @sweettea
kbuild: rust: use -Zdebuginfo-compression
7e2f775 
Rust 1.74.0 introduced (unstable) support for the
`-Zdebuginfo-compression` flag, thus use it.

Note that the releases built by the Rust project (i.e. the ones provided
by rustup) do not enable support for zstd in their bundled LLVM (yet,
at least), thus the Rust compiler will warn, but the build will proceed:

    warning: unknown debuginfo compression algorithm zstd - will fall
    back to uncompressed debuginfo

Link: rust-lang/rust#120953
Link: rust-lang/rust#115358
Reviewed-by: Alice Ryhl <aliceryhl@google.com>
Link: https://lore.kernel.org/r/20240217002622.57322-1-ojeda@kernel.org
[ Added note about zstd support in Rust-provided binaries. ]
Signed-off-by: Miguel Ojeda <ojeda@kernel.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@oli-obk oli-obk Awaiting requested review from oli-obk

Assignees

@oli-obk oli-obk

Labels
S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue.
Projects
None yet
Milestone
1.74.0
Development

Successfully merging this pull request may close these issues.

None yet
8 participants
@durin42 @rust-log-analyzer @rustbot @oli-obk @tmiasko @bors @fmease @Wyvern