New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/compile: consider using DWARF 5 #26379

Open
aclements opened this Issue Jul 13, 2018 · 8 comments

Comments

Projects
None yet
7 participants
@aclements
Member

aclements commented Jul 13, 2018

DWARF 5 has several advantages over previous versions of DWARF. Notably,

  1. It supports position-independent representations, which significantly reduces the number of relocations in object files and hence the size of object files and the load on the linker. In the go binary, 49% of the 503,361 total relocations are in the DWARF.

  2. It supports much more compact location and range list formats. The location and range list sections are 6% of the 12MiB of the go binary, even when zlib compressed.

  3. It has an official language code for Go. :)

DWARF 5 is quite new, and I don't think the rest of the ecosystem is ready yet, but I wanted to get the idea floating. It is supported by the GNU and LLVM toolchains and some debuggers. Support was added in GCC 7.1 (May 2017) and GDB 8.0 (June 2017). It appears to be in the latest LLVM, which covers most of the Xcode tools, though I can't find when it was added.

It is currently not supported by LLDB or the macOS linker. We could potentially get around the macOS linker by leaving out the Go DWARF from the objects we pass to the system linker and then merging it in to the final binary (we already do a merge step). This is more feasible with DWARF5 because it's mostly position-independent, so we wouldn't need dsymutil to relocate it for us.

/cc @cherrymui @heschik @dr2chase @randall77 @ianlancetaylor

@gopherbot gopherbot added this to the Proposal milestone Jul 13, 2018

@gopherbot gopherbot added the Proposal label Jul 13, 2018

@heschik

This comment has been minimized.

Contributor

heschik commented Jul 14, 2018

Strongly in favor of doing this the moment we think we can get away with it. I'm happy to do the work for location lists.

@heschik heschik added the Debugging label Jul 14, 2018

@bradfitz

This comment has been minimized.

Member

bradfitz commented Jul 16, 2018

Un-proposaling this per discussion with @ianlancetaylor.

@bradfitz bradfitz changed the title from proposal: cmd/compile: consider using DWARF 5 to cmd/compile: consider using DWARF 5 Jul 16, 2018

@bradfitz bradfitz removed the Proposal label Jul 16, 2018

@agnivade agnivade modified the milestones: Proposal, Unplanned Jul 17, 2018

@aarzilli

This comment has been minimized.

Contributor

aarzilli commented Oct 11, 2018

For reference, the debug info for the linux_amd64/go1.11.1 version of cmd/compile is 15MB of which 8MB are in debug_loc and debug_ranges, specifically:

total size 20752567

section             compressed    uncompressed
.zdebug_abbrev             274 B         467 B
.zdebug_line            739610 B     1931456 B
.zdebug_frame           158093 B      559020 B
.zdebug_pubnames         38451 B      251078 B
.zdebug_pubtypes         45270 B      211390 B
.debug_gdb_scripts                        42 B
.zdebug_info           1257828 B     3694829 B
.zdebug_loc             989048 B     5914085 B
.zdebug_ranges          442419 B     2461408 B

Rewriting debug_loc and debug_ranges in the new format specified by DWARF 5 will reduce the size of debug_ranges to 774460 B (31% of DWARF 4 size) and of debug_loc to 2032960 B (34% of DWARF 4 size).

Edit: debug_ranges and debug_loc were accidentally swapped in the last sentence.

@dr2chase

This comment has been minimized.

Contributor

dr2chase commented Oct 11, 2018

Is compression still doing a better job at reducing binary size than DWARF 5, or can D5 also be compressed? I did just mostly finish a DWARF splitter for OSX that recreates the expected file in the expected place, and adds the expected UUID for matching the two files. I suspect I ought to be using a hash of the binary contents excluding debugging information, so that identical files will remain identical after UUIDs are added.

@aclements

This comment has been minimized.

Member

aclements commented Oct 11, 2018

The compression isn't actually part of the DWARF spec, so it's orthogonal to the DWARF version and can be used with DWARF 5.

@aarzilli, thanks for doing that experiment. Can you get the numbers for DWARF 5 if it's also zlib compressed?

@aarzilli

This comment has been minimized.

Contributor

aarzilli commented Oct 12, 2018

@aarzilli, thanks for doing that experiment. Can you get the numbers for DWARF 5 if it's also zlib compressed?

After compression the size is basically the same, 98% (of compressed DWARF4) for debug_ranges and 77% (of compressed DWARF4) for debug_loc. The compression time however is reduced by 40%.

@dr2chase

This comment has been minimized.

Contributor

dr2chase commented Oct 12, 2018

Is that compression time+space comparing current low-effort (higher speed) versus future low-effort? Sorry to be so picky, it's just that we've made mistakes here before.

@aarzilli

This comment has been minimized.

Contributor

aarzilli commented Oct 12, 2018

I'm comparing compression using zlib.BestSpeed for both, like the linker. Since measuring how much time it actually took inside the linker is hard I measured how much time it takes to recompress the compressed section.

Because I have send the whole section to the compressor and the linker doesn't, I get slightly better compression than the linker. But the difference is small (around 1%) and since I'm doing it for both DWARF 4 and DWARF 5 the result should be valid.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment