Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/link: compress debug info on windows #25927

Closed
dajoo75 opened this issue Jun 17, 2018 · 19 comments

Comments

Projects
None yet
8 participants
@dajoo75
Copy link
Contributor

commented Jun 17, 2018

What did you do?

Compiled a basic hello world program on 64-bit Windows 10 with go tip (+b459e00), go 1.8.1 and go 1.10.3 and compared the binary sizes.

               hello_go1.8.1.exe  |  hello_go1.10.3.exe  |  hello_+b459e00.exe
binary size    1 633 280 bytes    |  2 057 728 bytes     |  2 848 768 bytes
                                  |                      |
.rdata                     N/A    |             N/A      |    752 128 bytes
.text          1 068 032 bytes    |  1 181 696 bytes     |    575 488 bytes
.debug_info      260 608 bytes    |    396 800 bytes     |    457 216 bytes
.debug_loc                 N/A    |             N/A      |    404 480 bytes
.symtab           98 304 bytes    |    141 312 bytes     |    142 336 bytes
.debug_frame      70 144 bytes    |     75 264 bytes     |     76 288 bytes
.debug_line       68 519 bytes    |     72 192 bytes     |    139 264 bytes
.debug_pubtypes   30 720 bytes    |     42 496 bytes     |     44 544 bytes
.debug_pubnames   17 408 bytes    |     40 960 bytes     |     41 472 bytes
.data             15 360 bytes    |     77 824 bytes     |     78 336 bytes
.debug_ranges              N/A    |     25 600 bytes     |    133 632 bytes
.idata             1 536 bytes    |      1 024 bytes     |      1 024 bytes
.debug_aranges       512 bytes    |              N/A     |              N/A
.debug_abbrev        512 bytes    |        512 bytes     |        512 bytes
.debug_gdb_scripts         N/A    |        512 bytes     |        512 bytes

What did you expect to see?

The binary size to be about the same at tip (+b459e00) as the previous release (go1.10.3 at the time of writing this). And if possible, the binary size to have gone down.

Other related issues: #6853, #11799

What did you see instead?

The binary size at tip (+b459e00) has increased by 38 percent since the previous release (go1.10.3 at the time of writing this).

Does this issue reproduce with the latest release (go1.10.3)?

Yes

System details

go version devel +b459e00 Sat Jun 16 13:35:45 2018 +0000 windows/amd64
GOARCH="amd64"
GOBIN=""
GOCACHE="C:\Users\loki\AppData\Local\go-build"
GOEXE=".exe"
GOHOSTARCH="amd64"
GOHOSTOS="windows"
GOOS="windows"
GOPATH="C:\home\projects\go"
GOPROXY=""
GORACE=""
GOROOT="C:\home\installs\go\go.tip"
GOTMPDIR=""
GOTOOLDIR="C:\home\installs\go\go.tip\pkg\tool\windows_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-m64 -mthreads -fmessage-length=0 -fdebug-prefix-map=C:\Users\loki\AppData\Local\Temp\go-build103655438=/tmp/go-build -gno-record-gcc-switches"
VGOMODROOT=""
GOROOT/bin/go version: go version devel +b459e00 Sat Jun 16 13:35:45 2018 +0000 windows/amd64
GOROOT/bin/go tool compile -V: compile version devel +b459e00 Sat Jun 16 13:35:45 2018 +0000
gdb --version: GNU gdb (GDB) 8.0
@ALTree

This comment has been minimized.

Copy link
Member

commented Jun 17, 2018

As you word it, this would technically be a dup of #6853.

What I think it's worth noting is that CL 118276 (cmd/link: compress DWARF sections in ELF binaries) only helped with ELF binaries. So I think it would be better if you renamed this issue to point out this specific issue (i.e., make this issue a TODO reminder to compress debug info on windows, too).

In theory we had #11799 (cmd/link: compress debug info) for the general problem, but it got closed after the ELF work.

@dajoo75 dajoo75 changed the title Windows binaries too big and growing cmd/link: compress debug info on windows Jun 17, 2018

@ALTree ALTree added this to the Go1.12 milestone Jun 17, 2018

@dajoo75

This comment has been minimized.

Copy link
Contributor Author

commented Jun 17, 2018

@ALTree I've renamed the issue to cmd/link: compress debug info on windows as you suggested.

Others more in the know about Windows, the PE file format and the debug story around Windows (gdb, delve etc.) have to chime in on the correct way forward.

@dajoo75

This comment has been minimized.

Copy link
Contributor Author

commented Jun 17, 2018

I tried to get this working on Windows (made some small changes to cmd/link and, debug/pe) and hello world.exe went from 2.8MB to 1.9MB. I almost got delve to start but didn't have time to finish it.

There doesn't seem to be an insurmountable amount of work left to do to get compression working on Windows as well (since most of the work seems to be done already in #11799) but I may be wrong (the devil is in the details and all that).

I would feel better if the original authors of #11799 looked at this as well (together with Alex) since they know what they are doing :).

cc @heschik, @alexbrainman, @ianlancetaylor, @aclements

@aarzilli

This comment has been minimized.

Copy link
Contributor

commented Jun 18, 2018

@dajoo75 what approach are you taking? Elf supports compression but PE doesn't (based on a ctrl-f search in the relevant MSDN page), so .zdebug?

@dajoo75

This comment has been minimized.

Copy link
Contributor Author

commented Jun 18, 2018

@aarzilli

This comment has been minimized.

Copy link
Contributor

commented Jun 18, 2018

Ah, I didn't notice the switch to zdebug. In that case: go-delve/delve#1237

@alexbrainman

This comment has been minimized.

Copy link
Member

commented Jun 18, 2018

Thank you @dajoo75 for creating the issue.

If I understand correctly, CL 118276 added compression for ELF DWARF sections. I would try and add some code to make .zdebug_... sections appear in Windows executables, and see where it gets us. I doubt I would have time before weekend. So others are welcome to this.

I also wonder if debuggers used by our users will support this. Lucky we have @aarzilli here, so he can fix Delve. Not so easy to fix gdb. We will see.

Alex

@aarzilli

This comment has been minimized.

Copy link
Contributor

commented Jun 18, 2018

Not so easy to fix gdb. We will see.

I think gdb was the original user of the zdebug convention, I imagine that should be fine. The problem is lldb.

@alexbrainman

This comment has been minimized.

Copy link
Member

commented Jun 18, 2018

The problem is lldb.

Does lldb work on Windows?

Alex

@aarzilli

This comment has been minimized.

Copy link
Contributor

commented Jun 18, 2018

Does lldb work on Windows?

Partially: https://lldb.llvm.org/status.html

@heschik

This comment has been minimized.

Copy link
Contributor

commented Jun 18, 2018

@dajoo75 I'm happy to review a CL and help you over some bumps; I'd have to set up a Windows machine to even start on this and it sounds like you're well into it. We are pretty deep into the freeze so it'd be nice to get it done soon.

Can someone confirm that on Windows we're basically ignoring the platform convention for debug info, which I guess would either be COFF or PDB? As long as that's the case and none of the standard tools work anyway, I think it makes sense to go ahead and compress it. I doubt lldb is very popular there, even if it does work.

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

commented Jun 18, 2018

@heschik That is correct: we only generate DWARF. COFF debug info is entirely inadequate. As far as I know PDB is not publicly documented.

@dajoo75

This comment has been minimized.

Copy link
Contributor Author

commented Jun 18, 2018

@heschik

This comment has been minimized.

Copy link
Contributor

commented Jun 18, 2018

Okay. @alexbrainman, if you don't claim this by tomorrow I'll take a crack at it.

@alexbrainman

This comment has been minimized.

Copy link
Member

commented Jun 19, 2018

Okay. @alexbrainman, if you don't claim this by tomorrow I'll take a crack at it

Please, do. I will try it myself, but I do not know when.

Alex

@gopherbot

This comment has been minimized.

Copy link

commented Jun 19, 2018

Change https://golang.org/cl/119816 mentions this issue: cmd/link: enable DWARF compression on Windows

@gopherbot

This comment has been minimized.

Copy link

commented Jun 19, 2018

Change https://golang.org/cl/119815 mentions this issue: debug/elf,macho,pe: support compressed DWARF

@gopherbot gopherbot closed this in 83515df Jun 19, 2018

gopherbot pushed a commit that referenced this issue Jun 19, 2018

debug/elf,macho,pe: support compressed DWARF
Since we're going to start compressing DWARF on Windows and maybe
Darwin, copy the ELF support for .zdebug sections to macho and pe. The
code is almost completely the same across the three.

While I was here I added support for compressed .debug_type sections,
which I presume were overlooked before.

Tests will come in a later CL once we can actually generate compressed
PE/Mach-O binaries, since there's no other good way to get test data.

Updates #25927, #11799

Change-Id: Ie920b6a16e9270bc3df214ce601a263837810376
Reviewed-on: https://go-review.googlesource.com/119815
Run-TryBot: Heschi Kreinick <heschi@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Reviewed-by: Austin Clements <austin@google.com>
@be5invis

This comment has been minimized.

Copy link

commented Oct 19, 2018

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

commented Oct 19, 2018

@be5invis Thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.