Join GitHub today
GitHub is home to over 31 million developers working together to host and review code, manage projects, and build software together.Sign up
proposal: cmd/link: by default, do not write out DWARF #26074
This is not as radical as it sounds.
At the very least, we need to understand what the default should be when building the Go installation: write out DWARF, or not? The costs and benefits are more subtle than some realize.
Update: As demonstrated below, dropping DWARF also causes a significant improvement in build/install time.
With this canonical, trivial, but also representative program as input, I used a sequence of Go versions to build the binary on my Mac (OSX 10.13.5, amd64). I have sorted the list into chronological order by version:
I believe the drop from 1.4 to 1.7 (1.5 and 1.6 won't run on my Mac any more) is due to various cleanups in the binary triggered by #6853.
The growth after that is pretty much all due to DWARF. Absent compression, DWARF debugging is now half the binary, as reported by @rsc's sizecmp:
That's 1.3MB of growth, almost all in debug info. Even the PC-to-line table grew massively, quite disproportionate to text size, which is inexplicable to me, but also a bit off topic.
So, DWARF is huge, but we need it, right?
I don't think we do, most of the time. Surely when we are using Delve or GDB or perhaps one day LLDB, yes, but mostly not.
The need for DWARF and other debugging support in Go programs is much less than the corresponding need in C programs, for which DWARF was designed. Go binaries already include basic type information (reflection), a simple symbol table, and PC-to-line data. These not only help the running program, they also provide valuable debugging aids as they stand.
Even without DWARF at all, stack traces cased by panic would be unchanged and would contain symbols and line numbers. Pprof, objdump, and many other tools would still work.
The DWARF tables are present only for the debuggers.
And useful though the debuggers are sometimes, they are not used often and often not used at all. I think Delve is a great tool, but I use it only once or twice a year because the existing, built-in debugging information is almost always all I need. Why then do we write bloated binaries, paying a cost in file I/O and DWARF write time (not to mention time to compress now) when half the data in the binary is almost never used?
If I look in my personal bin directory, it consists of a few shell scripts and many Go binaries, and the net size is in the gigabytes. Gigabytes of binaries! I could delete the DWARF data from all of them and get much of the space back at no cost.
Also, keep in mind that much of this data is redundant. Yes, the addresses change between binaries but the type information that we write out for the runtime, garbage collector, and so on is a megabyte or more of utter redundancy, unvarying yet unshared.
The counterargument to dropping DWARF is of course that people want to debug their programs. The recent Go developer survey reported much higher concern for good debugging support than for reducing binary size. But I stress, most programs are never shown to a debugger, and many programmers only rarely use a debugger on a Go binary.
The desire to have good debugging does not immediately translate into writing out full massive DWARF data every time we build and install a program. (Test binaries and such actually skip DWARF, by default.) I believe time might be better spent improving native debugging support in the binaries, such as more informative stack traces, but that is another topic.
So to the proposal itself:
I propose we change the Go build environment to suppress DWARF by default, saving lots of CPU time and disk space. Instead, a global shell environment variable, say GODWARF=1, could be set to cause it to be written out. Programmers that want DWARF can set that once, in their shell profile, and have full data available. Others could set it only occasionally, on bad days.
For the rest of us, the rest of the time, why bother with it?
If it is decided that DWARF is too valuable to disable by default, I would instead propose a variant of this proposal, where I could set GOWARF=0 and turn it off in perpetuity.
In other words, I am proposing two things.
Note: It's not easy enough to use
I think this is a good idea.
I too have tried to disable DWARF globally before, without much success. This proposal sounds great, and I agree with the change in default behavior. If a developer wants to use a debugger, they should be able to set an environment variable.
I also assume this wouldn't affect
This is pretty much what every other compiler does, I wouldn't call it radical.
My preference would be for a
My preference would be a golang package that have a function that can build golang package with all possible switches with good default value and nice field name like GoOs, GoArch, GoPathList, TagList, GoRoot, EnableDataRace, EnableDwarf ...
Command line flags or shell environment variables are very difficult for me to understand all the possible and the correct type of that value.
I suggest the command line of golang compiler should run a host os and arch golang package which is my golang program build script. Then I will write everything else in my golang program build script.
@davecheney, I agree with you that "recompile for debugging with Delve/gdb/lldb/etc" is not too onerous and well-established. I just want to point out for the record that the recent work is supposed to make it the case that you don't need
Profilers read DWARF as well, for what it is worth. It's nice to be able to collect a detailed profile without having to recompile. On the other hand (as with debuggers) very few people run profilers.
How about a tweak -- continue to emit DWARF, but when linking "user" programs, don't include the DWARF from the standard packages (people who use debuggers are rare, but people who debug the Go runtime are even fewer).
That's not that easy. First you'd still want the full debug_frame (at minimum) to unwind the stack (possibly also skeleton DIEs for all functions too). Secondly, delve checks the destination of all CALL instructions to filter away the calls to runtime functions inserted by the compiler. Thirdly delve needs a bunch of DIEs from runtime (off the top of my head: runtime.allgs, runtime.g, runtime.firstmoduledata) for various reasons. Basically you would need to do something like microsoft's symbol files for system dlls.
I am not the least bit happy that I'm expected to debug something different than what I run, and that a recompilation is expected. I don't run the debugger much because it's not easy to run the debugger much, and I'm trying to fix that ("not too many people swimming across the river, why do we need a bridge?"). I do think it is onerous; and I view it as one of the things that Java got relatively right.
There's the additional problem that the
The main point of adding all this debugging information to Go binaries was these two things; to remove speed bumps from Go debugging, and to make progress on the do-we/don't-we support problem for these flags.
Debugging information is also not just for debuggers. Years ago a friend of mine showed me what he called the "killer app for Python", which was just really good backtraces that did a nice job of reporting the values of all the local variables in the stack trace. Their app would wrap all that up in a neat little bundle, ask the customer if they were willing to send it in as a bug report (with certain assurances about information confidentiality that probably wouldn't fly now), and his next interaction with said customer was not to badger them for more information, but instead "here's the dot-release that fixes your problem, download it at your convenience". "And the customer thinks I'm a wizard!" We could/should do this for Go, at least as an option.
There are some alternatives, for instance, with reproducible builds we can generate debugging information on demand -- but when we mention that to nearby colleagues working on other languages and platforms, their reaction is "yeah, right". Their best practice is to build with debugging, make a copy (of either the debugging info alone, or the entire binary) and strip for shipping.
Is binary size causing some other problem that can be discussed publicly? Are we filling up disks? (not mine -- with 10 copies of the Go repo in various states of build, plus all the go-gotten binaries for tools, plus testing and benchmarking, about 2% of my "disk", counting both bin and pkg) . Is there some quota that we're exceeding? Are builds taking too long, and if so, by how much, and have we explored other ways of saving that time that don't reduce functionality? (the linker is due for a rewrite, the register allocator is definitely a time hog, Josh has done some interesting work on moving code from the front end to SSA that seems to save a little time, and has the potential to expose more of the compiler to multithreading).
"Debugging information" does not mean only "DWARF". As I said in my post, we can improve self-debugging support independently of improving debugger support, which means DWARF.
@dr2chase can always get the DWARF he wants all the time by just setting the flag once in his profile. I am not proposing to turn DWARF off, I am proposing to turn it off by default as few programmers need the support it provides every day. You are not expected to run without DWARF, you are expected just to set the flag once, for yourself, if you want DWARF on.
Binary size matters. We copy binaries around, we push binaries over networks, we put them in containers, we store them in the cloud, we fill our bin directories with many programs; in those operations, the accumulations are significant.
As to your performance problems, my experiments with building the go command (cmd/go) show that generating DWARF for a significant program is about 40% of the build CPU time. That is a major chunk.
If the DWARF is redundant -- once we add the missing information to what we store in the binary -- we could generate it from built-in debugging information, thus allowing us to replace before deprecation, instead of vice-versa.
We'd also want to be careful about how any compatibility guarantees interact with the debugging information; what "variables" can be examined at which "statements" depends on all sorts of stuff.
Another worry is to what degree we'd allow a Go program to introspect on its own execution through this debugging facility.
Would building out two separate binaries one with debugging information and one without (Using
referenced this issue
Jul 10, 2018
Now that DWARF is compressed on Mac (and I assume also on Windows), it seems like we're back to a 20% or so space overhead, and the main issue is now link latency.
Even on Linux I see DWARF aggregation approximately doubling the time spent in the linker (and it's not like the non-DWARF parts of the linker are terribly fast). So this is not a Mac-specific problem. Of course, compression might be what's taking all the time.
Putting this on hold pending better understanding of where all the CPU time is going and that there aren't significant optimizations remaining that might make it faster. If 2X really is the cost of DWARF, then we should resume this conversation about whether it makes sense to pay that cost speculatively all the time when the probability of needing it is near zero.
pushed a commit
Aug 1, 2018
As mentioned in the very first post of this issue, profiling is not affected when DWARF information is not present: