Trivial dependencies on large crates pull in massive amounts of debuginfo

[rocallahan]

extern crate rusoto_core;
const REGION: rusoto_core::Region = rusoto_core::Region::UsEast1;
fn main() {
    println!("Hello, region {:?}", &REGION);
}

rusoto_core::Region is a very simple enum type. This doesn't run any interesting code. The resulting Linux debug-build binary is 41MB. If I replace &REGION with 0 the binary is 7.5MB.

readelf -a shows that .text is 370,558 bytes. .debug_info is 10,541,677 bytes. The other debug sections account for most of the rest.

Inspecting the debuginfo shows DWARF compilation units for rusoto_core and lots of its dependencies that are entirely dead code. For example:

COMPILE_UNIT<header overall offset = 0x0034b008>:
< 0><0x0000000b>  DW_TAG_compile_unit
                    DW_AT_producer              clang LLVM (rustc version 1.30.1 (1433507eb 2018-11-07))
                    DW_AT_language              DW_LANG_Rust
                    DW_AT_name                  /home/roc/.cargo/registry/src/github.com-1ecc6299db9ec823/serde_json-1.0.33/src/lib.rs
                    DW_AT_stmt_list             0x001534c8
                    DW_AT_comp_dir              /home/roc/.cargo/registry/src/github.com-1ecc6299db9ec823/serde_json-1.0.33
                    DW_AT_GNU_pubnames          yes(1)
                    DW_AT_low_pc                0x00000000
                    DW_AT_ranges                0x000a7120
                ranges: 89 at .debug_ranges offset 684320 (0x000a7120)
                        [ 0] <offset pair low-off: 0x00000001 addr 0x00000001 high-off: 0x00000001 addr 0x00000001>
                        [ 1] <offset pair low-off: 0x00000001 addr 0x00000001 high-off: 0x00000001 addr 0x00000001>
                        [ 2] <offset pair low-off: 0x00000001 addr 0x00000001 high-off: 0x00000001 addr 0x00000001>
                        [ 3] <offset pair low-off: 0x00000001 addr 0x00000001 high-off: 0x00000001 addr 0x00000001>

... followed by 85 more identical ranges. These all indicate empty code ranges; all code for this CU has been stripped by the linker. However, this CU still has a ton of debuginfo for types and for functions. E.g.:

< 4><0x00001802>          DW_TAG_subprogram
                            DW_AT_low_pc                0x00000000
                            DW_AT_high_pc               <offset-from-lowpc>262
                            DW_AT_frame_base            len 0x0001: 57: DW_OP_reg7
                            DW_AT_linkage_name          _ZN91_$LT$core..slice..Iter$LT$$u27$a$C$$u20$T$GT$$u20$as$u20$core..iter..iterator..Iterator$GT$4next1
7hc6f1932985554eebE
                            DW_AT_name                  next<serde_json::value::Value>
                            DW_AT_decl_file             0x00000005 /home/roc/.cargo/registry/src/github.com-1ecc6299db9ec823/serde_json-1.0.33/libcore/slice/m
od.rs
                            DW_AT_decl_line             0x00000978
                            DW_AT_type                  <0x000007b0>
< 5><0x00001820>            DW_TAG_formal_parameter
                              DW_AT_location              len 0x0002: 9128: DW_OP_fbreg 40
                              DW_AT_name                  self
                              DW_AT_decl_file             0x00000003 /home/roc/.cargo/registry/src/github.com-1ecc6299db9ec823/serde_json-1.0.33/src/lib.rs
                              DW_AT_decl_line             0x00000001
                              DW_AT_type                  <0x00002f94>
< 5><0x0000182e>            DW_TAG_inlined_subroutine
                              DW_AT_abstract_origin       <0x00001a7d>
                              DW_AT_low_pc                0x00000000
                              DW_AT_high_pc               <offset-from-lowpc>123
                              DW_AT_call_file             0x00000005 /home/roc/.cargo/registry/src/github.com-1ecc6299db9ec823/serde_json-1.0.33/libcore/slice/mod.rs
                              DW_AT_call_line             0x00000982
< 6><0x00001842>              DW_TAG_formal_parameter
                                DW_AT_location              len 0x0002: 9138: DW_OP_fbreg 56
                                DW_AT_abstract_origin       <0x00001a97>
< 6><0x0000184a>              DW_TAG_formal_parameter
                                DW_AT_location              len 0x0003: 91c000: DW_OP_fbreg 64
                                DW_AT_abstract_origin       <0x00001aa2>
< 6><0x00001853>              DW_TAG_lexical_block
                                DW_AT_low_pc                0x00000000
                                DW_AT_high_pc               <offset-from-lowpc>36
< 7><0x00001860>                DW_TAG_variable
                                  DW_AT_location              len 0x0003: 91d000: DW_OP_fbreg 80
                                  DW_AT_abstract_origin       <0x00001aae>
< 5><0x0000186b>            DW_TAG_template_type_parameter
                              DW_AT_type                  <0x00000064>
                              DW_AT_name                  T

All DW_AT_low_pcs of DW_TAG_subprogram, DW_TAG_lexical_block and DW_TAG_inlined_subroutine in this CU are zero. There are a lot of CUs like this.

[rocallahan]

It seems to me that this debuginfo should have been garbage collected at some point, by the linker if not before, but for some reason that isn't happening. rustc 1.30.1 (1433507eb 2018-11-07), cargo 1.30.0 (a1a4ad372 2018-11-02). Regular cargo build with no special settings, just cargo new --bin bloat and in Cargo.toml

[dependencies]
rusoto_core = "0.35"

[rocallahan]

@tromey you might be interested in this.

[rocallahan]

Also, GNU ld version 2.29.1-23.fc28.

[rocallahan]

Similar results with LLD 7.0.0.

[rocallahan]

And similar results with gold.

It seems to me that the object files that don't have any code eventually included in the binary should be skipped by the linker, with their debuginfo not linked in at all.

[rocallahan]

It appears that because Rust divides code up into codegen units (i.e. object files) somewhat arbitrarily, as soon as you pull one from a crate, you end up having to link them all as you transitively resolve undefined symbols. Then it's up to --gc-sections to eliminate unused code, but apparently --gc-sections doesn't know about .debug_info etc. And, because the decisions about which object files to link happen before any --gc-sections processing, the debuginfo for all object files gets linked.

[rocallahan]

In that case, probably the best option here would be to teach the linker's --gc-sections implementation to drop all debuginfo sections for any object file that has had all its code and data sections discarded.

[rocallahan]

This issue has been discussed before, e.g. http://sourceware-org.1504.n7.nabble.com/Re-Debuggin-info-for-unused-sections-td112685.html.

I guess this isn't a high priority for C/C++ libraries since developers can divide them up manually into separate object files so that not all object files need to be linked into every application. So implementing the above optimization, though it needs to be done in the linker, would mainly be for the benefit of Rust.

[rocallahan]

In that case, probably the best option here would be to teach the linker's --gc-sections implementation to drop all debuginfo sections for any object file that has had all its code and data sections discarded.

I implemented this in LLD. It works well for some of my binaries, and for the testcase here, but only shrank the overall size of my built binaries from 2.4G to 2.0G, so I'm not sure whether it's worth the effort to push upstream...

[rocallahan]

Let's give it a go. https://reviews.llvm.org/D54747

[Diggsey]

I have the exact same issue, and it's really problematic because it results in up 600MB of memory usage when generating a backtrace, for only 50MB of debug info, for a binary that is only 643KB in size when stripped!

That's literally 1000x increase in memory usage 😱

[sanxiyn]

It seems LLD patch is merged. How can I take advantage of this today?

[rocallahan]

Build LLD master and use it. LLD is pretty easy to build.

[pnkfelix]

It was unfortunately reverted back in May (shortly after @rocallahan last comment above...)

[zephyrus00jp]

I am creating patches for Thunderbird mail client (originally from Mozilla) from time to time when I notice a bug here and there.

I have a nagging problem since early this year.
Since early this summer, I occasionally run out of my 50GB partition (in a VM setting.) for storing binary object only. This has not happened before.
I found that among the binary directory, the incremental compilation cache of rust is huge.
But I suspect EACH object of rust compilation is large on its own.
The cause of the problem seems to be this Rust's debug info bloat.

On my local PC, I see that the file directory size (with the sizes of
subdirectories added to it) is as follows:

MOZ_OBJ is 45.8 GB large. This is the top-most directory for storing binary object files.
(as of now: I had to stop TB compilation due
to the 50 GB smallish partition overflow where this directory is located.)
MOZ_OBJ/x86_64-unknown-linux-gnu/ is 33.9 GB large .
MOZ_OBJ/x86_64-unknown-linux-gnu/debug/ is 33.9 GB large.
I think /debug/ is because I create full debug version of TB locally.
MOZ_OBJ/MOZ_OBJ/x86_64-unknown-linux-gnu/debug/incremental is 26.5 GB large.

I have learned of this issue of large debug info from a discussion in a mozilla mailing list.
https://groups.google.com/forum/#!topic/mozilla.dev.platform/EnZbltbwUeE

It would be super if we can reduce the size of debug info (.dwo) for rust object files.
Can't the compiler itself reduce the debug info a bit more intelligently?

Thank you for the great package otherwise.
Mozilla developers seem to be creating a lot of libraries in rust at a frantic pace.

TIA

[benesch]

There's an LLVM patch out that adds a --gc-debuginfo flag to lld that seems to have exactly the desired effect: https://reviews.llvm.org/D74169

Looks to be a ways away from landing, but I tested it on a relatively large Rust project and it was able to reduce an 800MB binary to a 300MB binary (with full debug info). Details in a comment here: https://reviews.llvm.org/D74169#1990180

[zephyrus00jp]

On 2020/04/19 3:23, Nikhil Benesch wrote: There's an LLVM patch out that adds a |--gc-debuginfo| flag to |lld| that seems to have exactly the desired effect: https://reviews.llvm.org/D74169 Looks to be a ways away from landing, but I tested it on a relatively large Rust project and it was able to reduce an 800MB binary to a 300MB binary (with full debug info). Details in a comment here: https://reviews.llvm.org/D74169#1990180 — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#56068 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACUNALDPTKMXIR4M4IKAIE3RNHV2FANCNFSM4GFDXILQ>.

Thank you for the update. I will investigate the new option to lld and see if it helps me for my local development of mozilla Thunderbird mailer. It seems to me that many developers are now so used to fast SSD storage that they are unaware of binary bloat very well. If they use old-fashined hard disks, and slow ones at that, they will notice the problem very fast. Thank you again. Regards,

[khuey]

This continues to be an enormous problem. I have a binary that's 1% useful stuff and 99% debug info, almost all of which is for dead code. And that --gc-debuginfo LLD thing hasn't had any activity in almost two years.

[athre0z]

Agreed. The majority of DWARF debug records in even a hello world application are dead code with bogus PC ranges all starting at 0. This is not only bloating the binary, but also simply wrong and requires anyone parsing Rust DWARF to special case it.