all: binaries too big and growing #6853

robpike · Nov 30, 2013

As an experiment, I build "hello, world" at the release points for go 1.0.
1.1, and 1.2. Here are the binary's sizes:

% ls -l x.1.?
-rwxr-xr-x  1 r  staff  1191952 Nov 30 10:25 x.1.0
-rwxr-xr-x  1 r  staff  1525936 Nov 30 10:20 x.1.1
-rwxr-xr-x  1 r  staff  2188576 Nov 30 10:18 x.1.2
% size x.1.?
__TEXT  __DATA  __OBJC  others  dec hex
880640  33682096    0   4112    34566848    20f72c0 x.1.0
1064960 94656   0   75952   1235568 12da70  x.1.1
1429504 147896  0   177440  1754840 1ac6d8  x.1.2
% 

A near-doubling of the binary size in two releases is a bug of a kind. I will hold on to
the files so they can be analyzed more, but am filing this issue to get the topic
registered. We need to develop a better understanding of the problem and how to address
it.

Marking this 1.3 (not maybe) because I consider it a priority.


A few months ago I exchanged mail with Russ about this topic regarding a different, much
larger binary. To avoid him having to redo the analysis, here is what he said at the
time:

====
i sent CL 13722046 to make the nm -S output a bit more useful.
for the toy binary i now get

  4a2280  1898528 D symtab
  26f3a0  1405936 D type.*
  671aa0  1058432 D pclntab
  3c6790   598056 D go.string.*
  4620c0    49600 D gcbss
  7a7c20    45496 B runtime.mheap
  46e280    21936 D gcdata
  7a29e0    21056 b bufferList
  1ed600    16480 T crypto/tls.(*Conn).clientHandshake
  79eb20    16064 b semtable
  1b3d90    14224 T net/http.init

that seems plausible to me. some notes:

symtab is the plan 9 symbol table. it in the binary but never referenced at run time. it
supports things like nm -S only. it needs to move into an unmapped section of the
binary, but it is only costing at most 8k at run time right now due to fragmentation and
it just wasn't worth the effort to try to move. the new linker will make this easier. of
course, moving it in the file doesn't shrink the file.

the thing named pclntab is a reencoding of the original pclntab and the parts of the
plan 9 symbol table that we did need at run time (mostly just a list of functions and
their names and addresses). as you can see, it is much smaller than the old form (the
symbol table dominates).

type.* is the reflect types and go.string.* is the static go string data. the *
indicates that i coalesced many symbols into one, to avoid useless individual names
bloating the symbol table. if we tried we could probably cut the reflect types by 2-4x.
it would mean packing the data a bit more compactly than an ordinary go data structure
would and then using unsafe to get it back out.

gcbss and gcdata are garbage collection bits for the bss and data segments. that's what
atom symbol did, and it's not clear whether it will last (probably not) and whether what
will replace it will be smaller. time will tell. i have a meeting with dmitriy, carl,
and keith next week to figure out what the plan is.

runtime.mheap, bufferList, and semtable are bss.

you're not seeing the gdb dwarf debug information here, because it's not a runtime
symbol.
 
g% otool -l $(which toy) | egrep '^  segname|filesize'
  segname __PAGEZERO
 filesize 0
  segname __TEXT
 filesize 7811072
  segname __DATA
 filesize 126560
  segname __LINKEDIT
 filesize 921772
  segname __DWARF
 filesize 2886943
g% 

there's another 3 MB. you can build with -ldflags -w to get rid of that at least.
if you read the full otool -l output you will find

Load command 6
     cmd LC_SYMTAB
 cmdsize 24
  symoff 10825728
   nsyms 22559
  stroff 11186924
 strsize 560576

looks like another 1 MB or so (560576+11186924-10825728 or 22559*16+560576) for the
mach-o symbol table.

when we do the new linker we can make recording this kind of information in a useful
form a priority.

Comment 1:

Note: the binaries were build on amd64 10.7.5 (Lion), with  gcc -version
i686-apple-darwin11-llvm-gcc-4.2
For the record, I couldn't do this experiment on 10.9 with Xcode 5 because the older
releases wouldn't build due to gcc/clang skew.

robpike · Nov 30, 2013

Comment 1:

Note: the binaries were build on amd64 10.7.5 (Lion), with  gcc -version
i686-apple-darwin11-llvm-gcc-4.2
For the record, I couldn't do this experiment on 10.9 with Xcode 5 because the older
releases wouldn't build due to gcc/clang skew.

Comment 2:

Labels changed: added priority-later, removed priority-triage.

robpike · Nov 30, 2013

Comment 2:

Labels changed: added priority-later, removed priority-triage.

Comment 3 by jlourenco27:

Just for added reference, this are the size on go 1.1.2 vs 1.2 with OS X 10.9 and Xcode
5 (darwin gcc llvm 5.0 x86_64):
$ ls -l *1.*
-rwxr-xr-x  1 j      staff  1525984  2 Dez 21:44 hello_1.1.2
-rwxr-xr-x  1 j      staff  2192672  2 Dez 21:40 hello_1.2
$ size *1.*
__TEXT  __DATA  __OBJC  others  dec hex
1064960 94720   0   76000   1235680 12dae0  hello_1.1.2
1433600 147896  0   177440  1758936 1ad6d8  hello_1.2

gopherbot · Dec 2, 2013

Comment 3 by jlourenco27:

Just for added reference, this are the size on go 1.1.2 vs 1.2 with OS X 10.9 and Xcode
5 (darwin gcc llvm 5.0 x86_64):
$ ls -l *1.*
-rwxr-xr-x  1 j      staff  1525984  2 Dez 21:44 hello_1.1.2
-rwxr-xr-x  1 j      staff  2192672  2 Dez 21:40 hello_1.2
$ size *1.*
__TEXT  __DATA  __OBJC  others  dec hex
1064960 94720   0   76000   1235680 12dae0  hello_1.1.2
1433600 147896  0   177440  1758936 1ad6d8  hello_1.2

Comment 4:

This issue was updated by revision f238049.

R=golang-dev, rsc
CC=golang-dev
https://golang.org/cl/35940047

randall77 · Dec 3, 2013

Comment 4:

This issue was updated by revision f238049.

R=golang-dev, rsc
CC=golang-dev
https://golang.org/cl/35940047

Comment 5:

Labels changed: added release-go1.3.

rsc · Dec 4, 2013

Comment 5:

Labels changed: added release-go1.3.

Comment 6:

Labels changed: removed go1.3.

rsc · Dec 4, 2013

Comment 6:

Labels changed: removed go1.3.

Comment 7:

Labels changed: added repo-main.

rsc · Dec 4, 2013

Comment 7:

Labels changed: added repo-main.

Comment 8:

In this context, please reevaluate if the golang binaries can be changed to work with
UPX (ultimate packer for executables).¹²
For a small amount of computing power, upx can reduce the size of a binary down to a
quarter of its original size. You can authenticate this using a existing example
›fixer‹ programm for golang binaries on linux/amd64.³
While this approach doesn't fix the root of the problem – only the symptoms – it
would be nice to have this possibility always on hand.
For technical background of this problem (PT_LOAD[0].p_offset==0), please look at the
UPX bugtracker⁴.
¹ http://upx.sourceforge.net/
² https://en.wikipedia.org/wiki/UPX
³ https://github.com/pwaller/goupx
⁴ http://sourceforge.net/p/upx/bugs/195/

zephyr · Dec 7, 2013

Comment 8:

In this context, please reevaluate if the golang binaries can be changed to work with
UPX (ultimate packer for executables).¹²
For a small amount of computing power, upx can reduce the size of a binary down to a
quarter of its original size. You can authenticate this using a existing example
›fixer‹ programm for golang binaries on linux/amd64.³
While this approach doesn't fix the root of the problem – only the symptoms – it
would be nice to have this possibility always on hand.
For technical background of this problem (PT_LOAD[0].p_offset==0), please look at the
UPX bugtracker⁴.
¹ http://upx.sourceforge.net/
² https://en.wikipedia.org/wiki/UPX
³ https://github.com/pwaller/goupx
⁴ http://sourceforge.net/p/upx/bugs/195/

Comment 9:

re #8, I don't think it's Go's problem. upx should be made more flexible to handle this.

minux · Dec 7, 2013

Comment 9:

re #8, I don't think it's Go's problem. upx should be made more flexible to handle this.

Comment 10:

minux:
We may figure out why the binaries are growing fast first (lager runtime, optimization,
etc.), before we claim that it isn't Go's problem.

leo-liu · Dec 8, 2013

Comment 10:

minux:
We may figure out why the binaries are growing fast first (lager runtime, optimization,
etc.), before we claim that it isn't Go's problem.

Comment 11:

re #10, my #9 reply is to #8, which is about an entirely different problem.
i'm not saying that the ever-growing binaries is not our problem, only that i
don't believe that upx not accepting our binaries is our problem.
it's clear that upx isn't able to handle all possible and correct ELF files (i.e.
if the kernel can execute our binaries just fine, it's upx's problem to not be
able to compress them).

minux · Dec 8, 2013

Comment 11:

re #10, my #9 reply is to #8, which is about an entirely different problem.
i'm not saying that the ever-growing binaries is not our problem, only that i
don't believe that upx not accepting our binaries is our problem.
it's clear that upx isn't able to handle all possible and correct ELF files (i.e.
if the kernel can execute our binaries just fine, it's upx's problem to not be
able to compress them).

Comment 12:

More detail. The Plan 9 symbol table is about to be deleted. Here is a reference point,
adding one new entry to the list above:
$ ls -l *1.*
-rwxr-xr-x  1 j      staff  1525984  2 Dez 21:44 hello_1.1.2
-rwxr-xr-x  1 j      staff  2192672  2 Dez 21:40 hello_1.2
-rwxr-xr-x  1 j      staff  2474512 Feb 18 20:27 hello_1.2.x
$ size *1.*
__TEXT  __DATA  __OBJC  others  dec hex
1064960 94720   0   76000   1235680 12dae0  hello_1.1.2
1433600 147896  0   177440  1758936 1ad6d8  hello_1.2
1699840 160984  0   188944  2049768 1f46e8     hello_1.2.x
Text has grown substantially, as has data. At least some of this is due to new
annotations for the garbage collector.

robpike · Feb 19, 2014

Comment 12:

More detail. The Plan 9 symbol table is about to be deleted. Here is a reference point,
adding one new entry to the list above:
$ ls -l *1.*
-rwxr-xr-x  1 j      staff  1525984  2 Dez 21:44 hello_1.1.2
-rwxr-xr-x  1 j      staff  2192672  2 Dez 21:40 hello_1.2
-rwxr-xr-x  1 j      staff  2474512 Feb 18 20:27 hello_1.2.x
$ size *1.*
__TEXT  __DATA  __OBJC  others  dec hex
1064960 94720   0   76000   1235680 12dae0  hello_1.1.2
1433600 147896  0   177440  1758936 1ad6d8  hello_1.2
1699840 160984  0   188944  2049768 1f46e8     hello_1.2.x
Text has grown substantially, as has data. At least some of this is due to new
annotations for the garbage collector.

Comment 13:

More detail. The Plan 9 symbol table is about to be deleted. Here is a reference point,
adding one new entry to the list above:
$ ls -l *1.*
-rwxr-xr-x  1 r  staff  1191952 Nov 30 10:25 x.1.0
-rwxr-xr-x  1 r  staff  1525936 Nov 30 10:20 x.1.1
-rwxr-xr-x  1 r  staff  2188576 Nov 30 10:18 x.1.2
-rwxr-xr-x  1 r  staff  2474512 Feb 18 20:27 hello_1.2.x
$ size *1.*
__TEXT  __DATA  __OBJC  others  dec hex
880640  33682096     0  4112    34566848     20f72c0    x.1.0
1064960 94656   0   75952   1235568 12da70  x.1.1
1429504 147896  0   177440  1754840 1ac6d8  x.1.2
1699840 160984  0   188944  2049768 1f46e8     hello_1.2.x
Text has grown substantially, as has data. At least some of this is due to new
annotations for the garbage collector.

robpike · Feb 19, 2014

Comment 13:

More detail. The Plan 9 symbol table is about to be deleted. Here is a reference point,
adding one new entry to the list above:
$ ls -l *1.*
-rwxr-xr-x  1 r  staff  1191952 Nov 30 10:25 x.1.0
-rwxr-xr-x  1 r  staff  1525936 Nov 30 10:20 x.1.1
-rwxr-xr-x  1 r  staff  2188576 Nov 30 10:18 x.1.2
-rwxr-xr-x  1 r  staff  2474512 Feb 18 20:27 hello_1.2.x
$ size *1.*
__TEXT  __DATA  __OBJC  others  dec hex
880640  33682096     0  4112    34566848     20f72c0    x.1.0
1064960 94656   0   75952   1235568 12da70  x.1.1
1429504 147896  0   177440  1754840 1ac6d8  x.1.2
1699840 160984  0   188944  2049768 1f46e8     hello_1.2.x
Text has grown substantially, as has data. At least some of this is due to new
annotations for the garbage collector.

Comment 14:

More detail. The Plan 9 symbol table is about to be deleted. Here is a reference point,
adding one new entry to the list above:
$ ls -l *1.*
-rwxr-xr-x  1 r  staff  1191952 Nov 30 10:25 x.1.0
-rwxr-xr-x  1 r  staff  1525936 Nov 30 10:20 x.1.1
-rwxr-xr-x  1 r  staff  2188576 Nov 30 10:18 x.1.2
-rwxr-xr-x  1 r  staff  2474512 Feb 18 20:27 hello_1.2.x
$ size *1.*
__TEXT  __DATA  __OBJC  others  dec hex
880640  33682096     0  4112    34566848     20f72c0    x.1.0
1064960 94656   0   75952   1235568 12da70  x.1.1
1429504 147896  0   177440  1754840 1ac6d8  x.1.2
1699840 160984  0   188944  2049768 1f46e8     x.1.2.x
Text has grown substantially, as has data. At least some of this is due to new
annotations for the garbage collector.

robpike · Feb 19, 2014

Comment 14:

More detail. The Plan 9 symbol table is about to be deleted. Here is a reference point,
adding one new entry to the list above:
$ ls -l *1.*
-rwxr-xr-x  1 r  staff  1191952 Nov 30 10:25 x.1.0
-rwxr-xr-x  1 r  staff  1525936 Nov 30 10:20 x.1.1
-rwxr-xr-x  1 r  staff  2188576 Nov 30 10:18 x.1.2
-rwxr-xr-x  1 r  staff  2474512 Feb 18 20:27 hello_1.2.x
$ size *1.*
__TEXT  __DATA  __OBJC  others  dec hex
880640  33682096     0  4112    34566848     20f72c0    x.1.0
1064960 94656   0   75952   1235568 12da70  x.1.1
1429504 147896  0   177440  1754840 1ac6d8  x.1.2
1699840 160984  0   188944  2049768 1f46e8     x.1.2.x
Text has grown substantially, as has data. At least some of this is due to new
annotations for the garbage collector.

Comment 15:

This issue was updated by revision 964f6d3.

Nothing reads the Plan 9 symbol table anymore.
The last holdout was 'go tool nm', but since being rewritten in Go
it uses the standard symbol table for the binary format
(ELF, Mach-O, PE) instead.
Removing the Plan 9 symbol table saves ~15% disk space
on most binaries.
Two supporting changes included in this CL:
debug/gosym: use Go 1.2 pclntab to synthesize func-only
symbol table when there is no Plan 9 symbol table
debug/elf, debug/macho, debug/pe: ignore final EOF from ReadAt
LGTM=r
R=r, bradfitz
CC=golang-codereviews
https://golang.org/cl/65740045

rsc · Feb 19, 2014

Comment 15:

This issue was updated by revision 964f6d3.

Nothing reads the Plan 9 symbol table anymore.
The last holdout was 'go tool nm', but since being rewritten in Go
it uses the standard symbol table for the binary format
(ELF, Mach-O, PE) instead.
Removing the Plan 9 symbol table saves ~15% disk space
on most binaries.
Two supporting changes included in this CL:
debug/gosym: use Go 1.2 pclntab to synthesize func-only
symbol table when there is no Plan 9 symbol table
debug/elf, debug/macho, debug/pe: ignore final EOF from ReadAt
LGTM=r
R=r, bradfitz
CC=golang-codereviews
https://golang.org/cl/65740045

Comment 16:

After revision 737767dd81fd, I see a 25% reduction:
Before:
-rwxr-xr-x  1 bradfitz  staff  23556028 Feb 18 20:47 bin/camlistored
After:
-rwxr-xr-x  1 bradfitz  staff  17727420 Feb 18 20:48 bin/camlistored

bradfitz · Feb 19, 2014

Comment 16:

After revision 737767dd81fd, I see a 25% reduction:
Before:
-rwxr-xr-x  1 bradfitz  staff  23556028 Feb 18 20:47 bin/camlistored
After:
-rwxr-xr-x  1 bradfitz  staff  17727420 Feb 18 20:48 bin/camlistored

Comment 17:

For my test case before/after deleting the Plan 9 symbol table:
% ls -l ...
-rwxr-xr-x  1 r  staff  2474512 Feb 18 20:27 hello_1.2.x
-rwxr-xr-x  1 r  staff  2150928 Feb 18 22:28 hello_1.2.y
% size ...
__TEXT  __DATA  __OBJC  others  dec hex
1699840 160984  0   188944  2049768 1f46e8     hello_1.2.x
1376256 160984  0   188944  1726184 1a56e8    hello_1.2.x
% 
So deleting the Plan 9 symbol table pretty close to exactly compensates for the GC
information. We're back at Go 1.2 levels, still far too large but it's a start.

robpike · Feb 19, 2014

Comment 17:

For my test case before/after deleting the Plan 9 symbol table:
% ls -l ...
-rwxr-xr-x  1 r  staff  2474512 Feb 18 20:27 hello_1.2.x
-rwxr-xr-x  1 r  staff  2150928 Feb 18 22:28 hello_1.2.y
% size ...
__TEXT  __DATA  __OBJC  others  dec hex
1699840 160984  0   188944  2049768 1f46e8     hello_1.2.x
1376256 160984  0   188944  1726184 1a56e8    hello_1.2.x
% 
So deleting the Plan 9 symbol table pretty close to exactly compensates for the GC
information. We're back at Go 1.2 levels, still far too large but it's a start.

Comment 18:

This issue was updated by revision 2541cc8.

Every function now has a gcargs and gclocals symbol
holding associated garbage collection information.
Put them all in the same meta-symbol as the go.func data
and then drop individual entries from symbol table.
Removing gcargs and gclocals reduces the size of a
typical binary by 10%.
LGTM=r
R=r
CC=golang-codereviews
https://golang.org/cl/65870044

rsc · Feb 19, 2014

Comment 18:

This issue was updated by revision 2541cc8.

Every function now has a gcargs and gclocals symbol
holding associated garbage collection information.
Put them all in the same meta-symbol as the go.func data
and then drop individual entries from symbol table.
Removing gcargs and gclocals reduces the size of a
typical binary by 10%.
LGTM=r
R=r
CC=golang-codereviews
https://golang.org/cl/65870044

Comment 19:

This issue was updated by revision ae38b03.

For an ephemeral binary - one created, run, and then deleted -
there is no need to write dwarf debug information, since the
binary will not be used with gdb. In this case, instruct the linker
not to spend time and disk space generating the debug information
by passing the -w flag to the linker.
Omitting dwarf information reduces the size of most binaries by 25%.
We may be more aggressive about this in the future.
LGTM=bradfitz, r
R=r, bradfitz
CC=golang-codereviews
https://golang.org/cl/65890043

rsc · Feb 19, 2014

Comment 19:

This issue was updated by revision ae38b03.

For an ephemeral binary - one created, run, and then deleted -
there is no need to write dwarf debug information, since the
binary will not be used with gdb. In this case, instruct the linker
not to spend time and disk space generating the debug information
by passing the -w flag to the linker.
Omitting dwarf information reduces the size of most binaries by 25%.
We may be more aggressive about this in the future.
LGTM=bradfitz, r
R=r, bradfitz
CC=golang-codereviews
https://golang.org/cl/65890043

Comment 20:

After removing gcargs from the symbol table (stepping across CL 65870044)
% ls -l x.1.2.[yz]
-rwxr-xr-x  1 r  staff  2150928 Feb 18 22:28 hello_1.2.y
-rwxr-xr-x  1 r  staff  1932880 Feb 19 08:14 hello_1.2.z
% size x.1.2.[yz] 
__TEXT  __DATA  __OBJC  others  dec hex
1376256 160984  0   188944  1726184 1a56e8    hello_1.2.y
1376256 160984  0   110160  1647400 192328 hello_1.2.z
% 
It's now smaller than at 1.2 but still much bigger than 1.1, let alone 1.0.

robpike · Feb 19, 2014

Comment 20:

After removing gcargs from the symbol table (stepping across CL 65870044)
% ls -l x.1.2.[yz]
-rwxr-xr-x  1 r  staff  2150928 Feb 18 22:28 hello_1.2.y
-rwxr-xr-x  1 r  staff  1932880 Feb 19 08:14 hello_1.2.z
% size x.1.2.[yz] 
__TEXT  __DATA  __OBJC  others  dec hex
1376256 160984  0   188944  1726184 1a56e8    hello_1.2.y
1376256 160984  0   110160  1647400 192328 hello_1.2.z
% 
It's now smaller than at 1.2 but still much bigger than 1.1, let alone 1.0.

Comment 21:

I would like to take a look at compressing pclntab. Is it compressible?

gopherbot · Mar 4, 2014

Comment 21:

I would like to take a look at compressing pclntab. Is it compressible?

Comment 22:

The pclntab data should be reasonably compact, and note that fast access to the data is
important, since it is used for runtime.Callers and friends.  If you can find a
significant reduction in size that would be great, but small tweaks are probably not
desirable at this point.

ianlancetaylor · Mar 4, 2014

Comment 22:

The pclntab data should be reasonably compact, and note that fast access to the data is
important, since it is used for runtime.Callers and friends.  If you can find a
significant reduction in size that would be great, but small tweaks are probably not
desirable at this point.

Comment 23 by fuzxxl:

Is there any documentation for what the pclntab contains and for what constraints its
data structure must fullfill?

gopherbot · Mar 4, 2014

Comment 23 by fuzxxl:

Is there any documentation for what the pclntab contains and for what constraints its
data structure must fullfill?

Comment 24:

Let's not use this issue as a discussion list.  Please ask questions on golang-dev. 
Thanks.

ianlancetaylor · Mar 4, 2014

Comment 24:

Let's not use this issue as a discussion list.  Please ask questions on golang-dev. 
Thanks.

Comment 25 by allard.guy.m:

From the peanut gallery, AFAICT this breaks pprof interactive command 'list'.  At tip I
get, e.g.:
(pprof) list runner
Total: 6424 samples
objdump: syminit: Success
no filename found in main.runner<400c40>
Which works as expected with 1.2.1.

gopherbot · Mar 6, 2014

Comment 25 by allard.guy.m:

From the peanut gallery, AFAICT this breaks pprof interactive command 'list'.  At tip I
get, e.g.:
(pprof) list runner
Total: 6424 samples
objdump: syminit: Success
no filename found in main.runner<400c40>
Which works as expected with 1.2.1.

Comment 26:

Let's not use this issue as a discussion list.  Please ask questions on golang-dev. 
Thanks.
pprof not working is issue #7452.

Labels changed: added restrict-addissuecomment-commit.

rsc · Apr 3, 2014

Comment 26:

Let's not use this issue as a discussion list.  Please ask questions on golang-dev. 
Thanks.
pprof not working is issue #7452.

Labels changed: added restrict-addissuecomment-commit.

Comment 27:

This is as fixed as it is going to be for Go 1.3.
Right now at tip + CL 80370045 on darwin/amd64, compiling this program:
package main
import "fmt"
func main() {
    fmt.Println("hello, world")
}
I get 1830352 bytes for the binary. Assuming this is the same case for which Rob's
numbers are reported, by this metric Go 1.3 will roll back more than half the size
increase caused by Go 1.2 (relative to Go 1.1). Will leave further improvement for Go
1.4.

Labels changed: added release-go1.4, removed release-go1.3.

rsc · Apr 3, 2014

Comment 27:

This is as fixed as it is going to be for Go 1.3.
Right now at tip + CL 80370045 on darwin/amd64, compiling this program:
package main
import "fmt"
func main() {
    fmt.Println("hello, world")
}
I get 1830352 bytes for the binary. Assuming this is the same case for which Rob's
numbers are reported, by this metric Go 1.3 will roll back more than half the size
increase caused by Go 1.2 (relative to Go 1.1). Will leave further improvement for Go
1.4.

Labels changed: added release-go1.4, removed release-go1.3.

Comment 28:

This issue was updated by revision a26c01a.

LGTM=khr
R=khr
CC=golang-codereviews
https://golang.org/cl/80370045

rsc · Apr 3, 2014

Comment 28:

This issue was updated by revision a26c01a.

LGTM=khr
R=khr
CC=golang-codereviews
https://golang.org/cl/80370045

Comment 29:

Labels changed: added release-go1.5, removed release-go1.4.

rsc · Sep 15, 2014

Comment 29:

Labels changed: added release-go1.5, removed release-go1.4.

I have some more typeOff work I was hoping to do in 1.8 (the continuation of the work linked here done in 1.7), though I am busy in other projects right now. Let's at least leave this open until the window closes in a few weeks.

crawshaw · Oct 5, 2016

I have some more typeOff work I was hoping to do in 1.8 (the continuation of the work linked here done in 1.7), though I am busy in other projects right now. Let's at least leave this open until the window closes in a few weeks.

We have very few umbrella issues, but this one is fine to keep. It's good to keep in mind and collect all the work toward this. The binaries are still much bigger than we hope they would be.

rsc · Oct 6, 2016

We have very few umbrella issues, but this one is fine to keep. It's good to keep in mind and collect all the work toward this. The binaries are still much bigger than we hope they would be.

CL https://golang.org/cl/43090 mentions this issue.

gopherbot · May 10, 2017

CL https://golang.org/cl/43090 mentions this issue.

CL https://golang.org/cl/43190 mentions this issue.

gopherbot · May 10, 2017

CL https://golang.org/cl/43190 mentions this issue.

CL https://golang.org/cl/44007 mentions this issue.

gopherbot · May 24, 2017

CL https://golang.org/cl/44007 mentions this issue.

No notable wins (maybe a slight loss) at tip (to-be Go 1.9) compared to Go 1.8:

bradfitz@gdev:~$ for x in go1.4 go1.5 go1.6 go1.7 go1.8 go; do ls -l $x/bin/go; done
-rwxr-xr-x 1 bradfitz bradfitz 9571864 Feb 18  2015 go1.4/bin/go
-rwxr-xr-x 1 bradfitz bradfitz 11290832 Jan 13  2016 go1.5/bin/go
-rwxr-xr-x 1 bradfitz bradfitz 12534640 Jul 18  2016 go1.6/bin/go
-rwxr-xr-x 1 bradfitz bradfitz 9953979 Aug 15  2016 go1.7/bin/go
-rwxr-xr-x 1 bradfitz bradfitz 10068917 Feb 16 19:28 go1.8/bin/go
-rwxr-xr-x 1 bradfitz bradfitz 10346229 Jun  6 00:03 go/bin/go
bradfitz@gdev:~$ for x in go1.4 go1.5 go1.6 go1.7 go1.8 go; do ls -l $x/pkg/tool/linux_amd64/cgo; done
-rwxr-xr-x 1 bradfitz bradfitz 4560376 Feb 18  2015 go1.4/pkg/tool/linux_amd64/cgo
-rwxr-xr-x 1 bradfitz bradfitz 5111280 Jan 13  2016 go1.5/pkg/tool/linux_amd64/cgo
-rwxr-xr-x 1 bradfitz bradfitz 5279368 Jul 18  2016 go1.6/pkg/tool/linux_amd64/cgo
-rwxr-xr-x 1 bradfitz bradfitz 4114079 Aug 15  2016 go1.7/pkg/tool/linux_amd64/cgo
-rwxr-xr-x 1 bradfitz bradfitz 3914818 Feb 16 19:28 go1.8/pkg/tool/linux_amd64/cgo
-rwxr-xr-x 1 bradfitz bradfitz 4015485 Jun  6 00:03 go/pkg/tool/linux_amd64/cgo
bradfitz@gdev:~$ for x in go1.4 go1.5 go1.6 go1.7 go1.8 go; do ls -l $x/pkg/tool/linux_amd64/objdump; done
-rwxr-xr-x 1 bradfitz bradfitz 4408008 Feb 18  2015 go1.4/pkg/tool/linux_amd64/objdump
-rwxr-xr-x 1 bradfitz bradfitz 4889240 Jan 13  2016 go1.5/pkg/tool/linux_amd64/objdump
-rwxr-xr-x 1 bradfitz bradfitz 4988648 Jul 18  2016 go1.6/pkg/tool/linux_amd64/objdump
-rwxr-xr-x 1 bradfitz bradfitz 3622669 Aug 15  2016 go1.7/pkg/tool/linux_amd64/objdump
-rwxr-xr-x 1 bradfitz bradfitz 3717826 Feb 16 19:28 go1.8/pkg/tool/linux_amd64/objdump
-rwxr-xr-x 1 bradfitz bradfitz 3836105 Jun  6 00:03 go/pkg/tool/linux_amd64/objdump
bradfitz@gdev:~$ for x in go1.4 go1.5 go1.6 go1.7 go1.8 go; do ls -l $x/bin/gofmt; done
-rwxr-xr-x 1 bradfitz bradfitz 3594744 May 11  2016 go1.4/bin/gofmt
-rwxr-xr-x 1 bradfitz bradfitz 3944064 Jan 13  2016 go1.5/bin/gofmt
-rwxr-xr-x 1 bradfitz bradfitz 3895320 Jul 18  2016 go1.6/bin/gofmt
-rwxr-xr-x 1 bradfitz bradfitz 3036195 Aug 15  2016 go1.7/bin/gofmt
-rwxr-xr-x 1 bradfitz bradfitz 3481554 Feb 16 19:28 go1.8/bin/gofmt
-rwxr-xr-x 1 bradfitz bradfitz 3257512 Jun  6 00:03 go/bin/gofmt

I'm going to move this ongoing tracking bug to Go 1.10, since I don't see anything more happening for Go 1.9.

bradfitz · Jun 8, 2017

No notable wins (maybe a slight loss) at tip (to-be Go 1.9) compared to Go 1.8:

bradfitz@gdev:~$ for x in go1.4 go1.5 go1.6 go1.7 go1.8 go; do ls -l $x/bin/go; done
-rwxr-xr-x 1 bradfitz bradfitz 9571864 Feb 18  2015 go1.4/bin/go
-rwxr-xr-x 1 bradfitz bradfitz 11290832 Jan 13  2016 go1.5/bin/go
-rwxr-xr-x 1 bradfitz bradfitz 12534640 Jul 18  2016 go1.6/bin/go
-rwxr-xr-x 1 bradfitz bradfitz 9953979 Aug 15  2016 go1.7/bin/go
-rwxr-xr-x 1 bradfitz bradfitz 10068917 Feb 16 19:28 go1.8/bin/go
-rwxr-xr-x 1 bradfitz bradfitz 10346229 Jun  6 00:03 go/bin/go
bradfitz@gdev:~$ for x in go1.4 go1.5 go1.6 go1.7 go1.8 go; do ls -l $x/pkg/tool/linux_amd64/cgo; done
-rwxr-xr-x 1 bradfitz bradfitz 4560376 Feb 18  2015 go1.4/pkg/tool/linux_amd64/cgo
-rwxr-xr-x 1 bradfitz bradfitz 5111280 Jan 13  2016 go1.5/pkg/tool/linux_amd64/cgo
-rwxr-xr-x 1 bradfitz bradfitz 5279368 Jul 18  2016 go1.6/pkg/tool/linux_amd64/cgo
-rwxr-xr-x 1 bradfitz bradfitz 4114079 Aug 15  2016 go1.7/pkg/tool/linux_amd64/cgo
-rwxr-xr-x 1 bradfitz bradfitz 3914818 Feb 16 19:28 go1.8/pkg/tool/linux_amd64/cgo
-rwxr-xr-x 1 bradfitz bradfitz 4015485 Jun  6 00:03 go/pkg/tool/linux_amd64/cgo
bradfitz@gdev:~$ for x in go1.4 go1.5 go1.6 go1.7 go1.8 go; do ls -l $x/pkg/tool/linux_amd64/objdump; done
-rwxr-xr-x 1 bradfitz bradfitz 4408008 Feb 18  2015 go1.4/pkg/tool/linux_amd64/objdump
-rwxr-xr-x 1 bradfitz bradfitz 4889240 Jan 13  2016 go1.5/pkg/tool/linux_amd64/objdump
-rwxr-xr-x 1 bradfitz bradfitz 4988648 Jul 18  2016 go1.6/pkg/tool/linux_amd64/objdump
-rwxr-xr-x 1 bradfitz bradfitz 3622669 Aug 15  2016 go1.7/pkg/tool/linux_amd64/objdump
-rwxr-xr-x 1 bradfitz bradfitz 3717826 Feb 16 19:28 go1.8/pkg/tool/linux_amd64/objdump
-rwxr-xr-x 1 bradfitz bradfitz 3836105 Jun  6 00:03 go/pkg/tool/linux_amd64/objdump
bradfitz@gdev:~$ for x in go1.4 go1.5 go1.6 go1.7 go1.8 go; do ls -l $x/bin/gofmt; done
-rwxr-xr-x 1 bradfitz bradfitz 3594744 May 11  2016 go1.4/bin/gofmt
-rwxr-xr-x 1 bradfitz bradfitz 3944064 Jan 13  2016 go1.5/bin/gofmt
-rwxr-xr-x 1 bradfitz bradfitz 3895320 Jul 18  2016 go1.6/bin/gofmt
-rwxr-xr-x 1 bradfitz bradfitz 3036195 Aug 15  2016 go1.7/bin/gofmt
-rwxr-xr-x 1 bradfitz bradfitz 3481554 Feb 16 19:28 go1.8/bin/gofmt
-rwxr-xr-x 1 bradfitz bradfitz 3257512 Jun  6 00:03 go/bin/gofmt

I'm going to move this ongoing tracking bug to Go 1.10, since I don't see anything more happening for Go 1.9.

Change https://golang.org/cl/57130 mentions this issue: cmd/compile/internal/ssa: combine consecutive loads and stores on amd64

gopherbot · Aug 24, 2017

Change https://golang.org/cl/57130 mentions this issue: cmd/compile/internal/ssa: combine consecutive loads and stores on amd64

This issue reports slight increases between 1.8 and 1.9: #21653

mvdan · Aug 28, 2017

This issue reports slight increases between 1.8 and 1.9: #21653

Change https://golang.org/cl/61190 mentions this issue: cmd/compile: specialize map creation for small hint sizes

gopherbot · Sep 9, 2017

Change https://golang.org/cl/61190 mentions this issue: cmd/compile: specialize map creation for small hint sizes

Change https://golang.org/cl/88135 mentions this issue: cmd/compile: don't combine 64-bit loads/stores on amd64

gopherbot · Jan 17, 2018

Change https://golang.org/cl/88135 mentions this issue: cmd/compile: don't combine 64-bit loads/stores on amd64

#23934 reports an ~8% increase between 1.9.4 and 1.10 for default binaries, and a slight decrease with -ldflags="-s -w". (#11799 is relevant to debug info size.)

FiloSottile · Feb 20, 2018

#23934 reports an ~8% increase between 1.9.4 and 1.10 for default binaries, and a slight decrease with -ldflags="-s -w". (#11799 is relevant to debug info size.)

I'm seeing a significant regression in binaries sizes between go1.10.1 and the current tip on linux/amd64. A simple hello world is 26% bigger than it was on 1.10:

$ cat test.go 
package main

import "fmt"

func main() {
	fmt.Println("hi!")
}

$ go version
go version go1.10.1 linux/amd64
$ go build test.go
$ ls -l test
-rwxr-xr-x 1 alberto alberto 2011612 Apr 27 11:14 test

$ gotip version
go version devel +a3bafcf8cc Thu Apr 26 18:26:06 2018 +0000 linux/amd64
$ gotip build test.go
$ ls -l test
-rwxr-xr-x 1 alberto alberto 2543972 Apr 27 11:15 test

It's not just the hello world. The go binary went from 11MB to 15MB. gofmt from 3.4MB to 4.3MB.

Is this expected?

ALTree · Apr 27, 2018

I'm seeing a significant regression in binaries sizes between go1.10.1 and the current tip on linux/amd64. A simple hello world is 26% bigger than it was on 1.10:

$ cat test.go 
package main

import "fmt"

func main() {
	fmt.Println("hi!")
}

$ go version
go version go1.10.1 linux/amd64
$ go build test.go
$ ls -l test
-rwxr-xr-x 1 alberto alberto 2011612 Apr 27 11:14 test

$ gotip version
go version devel +a3bafcf8cc Thu Apr 26 18:26:06 2018 +0000 linux/amd64
$ gotip build test.go
$ ls -l test
-rwxr-xr-x 1 alberto alberto 2543972 Apr 27 11:15 test

It's not just the hello world. The go binary went from 11MB to 15MB. gofmt from 3.4MB to 4.3MB.

Is this expected?

@ALTree

@ALTree have you tried the same tests with -ldflags="-w -s"? There have been many changes to debugging info recently, so perhaps it's just that the binaries contain more debug info.

mvdan · Apr 27, 2018

@ALTree have you tried the same tests with -ldflags="-w -s"? There have been many changes to debugging info recently, so perhaps it's just that the binaries contain more debug info.

Generally speaking we do not recommend stripping debug info (it's useful, after all), so that's only useful for figuring out why they grew, not a justification for the growth.

dominikh · Apr 27, 2018

Generally speaking we do not recommend stripping debug info (it's useful, after all), so that's only useful for figuring out why they grew, not a justification for the growth.

@mvdan

@mvdan Yes, and stripped binaries are the same size as in 1.10, but that's not the default...

But yeah, if this is an expected effect of the recent work on debug info then... ok, I guess? I just wanted to make sure this was intentional.

ALTree · Apr 27, 2018

@mvdan Yes, and stripped binaries are the same size as in 1.10, but that's not the default...

But yeah, if this is an expected effect of the recent work on debug info then... ok, I guess? I just wanted to make sure this was intentional.

I should have been explicit; I meant it only as a way to quickly figure out where the growth was coming from.

mvdan · Apr 27, 2018

I should have been explicit; I meant it only as a way to quickly figure out where the growth was coming from.

When I looked a few weeks ago, all of the increase (and some) was dwarf. I think the best fix is probably #11799.

josharian · Apr 27, 2018

When I looked a few weeks ago, all of the increase (and some) was dwarf. I think the best fix is probably #11799.

Change https://golang.org/cl/118276 mentions this issue: cmd/link: compress DWARF sections in ELF binaries

gopherbot · Jun 15, 2018

Change https://golang.org/cl/118276 mentions this issue: cmd/link: compress DWARF sections in ELF binaries

Change https://golang.org/cl/127075 mentions this issue: html: lazily populate Unescape tables

gopherbot · Jul 31, 2018

Change https://golang.org/cl/127075 mentions this issue: html: lazily populate Unescape tables

robpike added the accepted label Sep 15, 2014

rsc modified the milestones: Go1.9, Go1.8 Oct 6, 2016

bradfitz modified the milestones: Go1.10, Go1.9 Jun 8, 2017

rsc modified the milestones: Go1.10, Go1.11 Nov 22, 2017

bradfitz added the release-blocker label May 18, 2018

bradfitz removed the release-blocker label Jun 29, 2018

bradfitz modified the milestones: Go1.11, Unplanned Jun 29, 2018

golang/go

Join GitHub today

all: binaries too big and growing #6853

Comments

robpike commented Nov 30, 2013

This comment has been minimized.

robpike commented Nov 30, 2013

This comment has been minimized.

robpike commented Nov 30, 2013

This comment has been minimized.

gopherbot commented Dec 2, 2013

This comment has been minimized.

randall77 commented Dec 3, 2013

This comment has been minimized.

rsc commented Dec 4, 2013

This comment has been minimized.

rsc commented Dec 4, 2013

This comment has been minimized.

rsc commented Dec 4, 2013

This comment has been minimized.

zephyr commented Dec 7, 2013

This comment has been minimized.

minux commented Dec 7, 2013

This comment has been minimized.

leo-liu commented Dec 8, 2013

This comment has been minimized.

minux commented Dec 8, 2013

This comment has been minimized.

robpike commented Feb 19, 2014

This comment has been minimized.

robpike commented Feb 19, 2014

This comment has been minimized.

robpike commented Feb 19, 2014

This comment has been minimized.

rsc commented Feb 19, 2014

This comment has been minimized.

bradfitz commented Feb 19, 2014

This comment has been minimized.

robpike commented Feb 19, 2014

This comment has been minimized.

rsc commented Feb 19, 2014

This comment has been minimized.

rsc commented Feb 19, 2014

This comment has been minimized.

robpike commented Feb 19, 2014

This comment has been minimized.

gopherbot commented Mar 4, 2014

This comment has been minimized.

ianlancetaylor commented Mar 4, 2014

This comment has been minimized.

gopherbot commented Mar 4, 2014

This comment has been minimized.

ianlancetaylor commented Mar 4, 2014

This comment has been minimized.

gopherbot commented Mar 6, 2014

This comment has been minimized.

rsc commented Apr 3, 2014

This comment has been minimized.

rsc commented Apr 3, 2014

This comment has been minimized.

rsc commented Apr 3, 2014

This comment has been minimized.

rsc commented Sep 15, 2014

robpike added the accepted label Sep 15, 2014

This comment has been minimized.

crawshaw commented Oct 5, 2016

This comment has been minimized.

rsc commented Oct 6, 2016

rsc modified the milestones: Go1.9, Go1.8 Oct 6, 2016

This comment has been minimized.

gopherbot commented May 10, 2017

This comment has been minimized.

gopherbot commented May 10, 2017

This comment has been minimized.

gopherbot commented May 24, 2017

This comment has been minimized.

bradfitz commented Jun 8, 2017

bradfitz modified the milestones: Go1.10, Go1.9 Jun 8, 2017

This comment has been minimized.

gopherbot commented Aug 24, 2017

ALTree commented Apr 27, 2018 •

edited

ALTree commented Apr 27, 2018 •

edited