Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/link: support for zstd-compressed DWARF sections #55107

Open
stapelberg opened this issue Sep 16, 2022 · 11 comments
Open

cmd/link: support for zstd-compressed DWARF sections #55107

stapelberg opened this issue Sep 16, 2022 · 11 comments
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. FeatureRequest help wanted NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone

Comments

@stapelberg
Copy link
Contributor

stapelberg commented Sep 16, 2022

I recently read a blog post by @MaskRay comparing different compression algorithms for DWARF: https://maskray.me/blog/2022-09-09-zstd-compressed-debug-sections

The blog post concludes that zstd is the best choice (as it is in many other scenarios), and the author has done some work on establishing ELFCOMPRESS_ZSTD.

I don’t know if it’s too early, or if anything else is preventing us from doing so, but wanted to file this issue to track support for using zstd to make Go DWARF sections smaller and faster to write and read.

There is a good native Go implementation of zstd available: https://pkg.go.dev/github.com/klauspost/compress/zstd

@gopherbot gopherbot added the compiler/runtime Issues related to the Go compiler and/or runtime. label Sep 16, 2022
@dsnet
Copy link
Member

dsnet commented Sep 16, 2022

Would it make sense to consider a standard library "compress/zstd" package first before considering this?

@stapelberg
Copy link
Contributor Author

stapelberg commented Sep 16, 2022

Would it make sense to consider a standard library "encoding/zstd" package first before considering this?

I don’t think we need to gate landing zstd DWARF on having a zstd package in the standard library — we could just import it into internal, or vendor it, or whatever is the currently preferred approach :)

But, having encoding/zstd in the standard library would certainly be a welcome addition IMHO :)

Can you open a separate issue for that please?

@MaskRay
Copy link

MaskRay commented Sep 16, 2022

@mengzhuo added ELFCOMPRESS_ZLIB support to cmd/link and may be interested:)

@Foxboron
Copy link

Foxboron commented Sep 16, 2022

I'd like to just point out that elfutils without support for ZSTD is going to be painfull for downstream as we can't make debug packages out of it. I'd rather have us disable compression and DWARF headers by default.

@stapelberg
Copy link
Contributor Author

stapelberg commented Sep 16, 2022

Sounds like it should be behind a flag for now, or added only once elfutils support for zstd is more widespread.

@cherrymui
Copy link
Member

cherrymui commented Sep 16, 2022

Do the debuggers support this format? GDB? LLDB? Delve? Thanks.

cc @aarzilli @thanm

@dsnet
Copy link
Member

dsnet commented Sep 16, 2022

Does the runtime ever look at the DWARF section, will we need to link in a zstd decompressor?

@cherrymui
Copy link
Member

cherrymui commented Sep 16, 2022

@dsnet the runtime doesn't look at the DWARF sections.

@mengzhuo
Copy link
Contributor

mengzhuo commented Sep 17, 2022

@mengzhuo added ELFCOMPRESS_ZLIB support to cmd/link and may be interested:)

I'm glad to do that :) but as @dsnet suggests it looks like we need a encoding/zstd in the first place.

cc @rsc

@aarzilli
Copy link
Contributor

aarzilli commented Sep 17, 2022

Do the debuggers support this format? GDB? LLDB? Delve? Thanks.

cc @aarzilli @thanm

According to the blog post, it was proposed to SysV gABI in june of this year and it is supported by approximately nothing. Delve will support it when debug/elf will support it, so I think it should be added there first (or at least at the same time). Other consideration:

  • I think @dsnet is right that there should be a compress/zstd first, it would be weird if the standard library supported decompressing zstd but only if it's inside an elf container
  • this is all about elf compressed sections, it does not apply to zdebug sections, which are emitted for mach-o and PE, and will need to stay on zlib
  • given how slow binutils moves in LTS distributions this likely can not be the default (or only) option for several years
  • the list of feature requests at the end of the article is highly incomplete, for example bloaty (which the article even uses) will have to be changed to support zstd compressed elf sections
  • it probably won't be a problem but compression speed is also a factor for cmd/link.

@cherrymui cherrymui added this to the Unplanned milestone Sep 19, 2022
@cherrymui cherrymui added FeatureRequest help wanted NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Sep 19, 2022
@MaskRay
Copy link

MaskRay commented Sep 27, 2022

FWIW clang/ld.lld/llvm-objcopy has supported zstd for ~2 weeks now. I just landed zstd support to binutils and gdb https://sourceware.org/git/?p=binutils-gdb.git;a=commit;h=2cac01e3ffff74898c54fa5e6418817f5578adb6 :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. FeatureRequest help wanted NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Development

No branches or pull requests

8 participants