all: binaries too big and growing #6853

Open
robpike opened this Issue Nov 30, 2013 · 96 comments

Comments

Projects
None yet
@robpike
Contributor

robpike commented Nov 30, 2013

As an experiment, I build "hello, world" at the release points for go 1.0.
1.1, and 1.2. Here are the binary's sizes:

% ls -l x.1.?
-rwxr-xr-x  1 r  staff  1191952 Nov 30 10:25 x.1.0
-rwxr-xr-x  1 r  staff  1525936 Nov 30 10:20 x.1.1
-rwxr-xr-x  1 r  staff  2188576 Nov 30 10:18 x.1.2
% size x.1.?
__TEXT  __DATA  __OBJC  others  dec hex
880640  33682096    0   4112    34566848    20f72c0 x.1.0
1064960 94656   0   75952   1235568 12da70  x.1.1
1429504 147896  0   177440  1754840 1ac6d8  x.1.2
% 

A near-doubling of the binary size in two releases is a bug of a kind. I will hold on to
the files so they can be analyzed more, but am filing this issue to get the topic
registered. We need to develop a better understanding of the problem and how to address
it.

Marking this 1.3 (not maybe) because I consider it a priority.


A few months ago I exchanged mail with Russ about this topic regarding a different, much
larger binary. To avoid him having to redo the analysis, here is what he said at the
time:

====
i sent CL 13722046 to make the nm -S output a bit more useful.
for the toy binary i now get

  4a2280  1898528 D symtab
  26f3a0  1405936 D type.*
  671aa0  1058432 D pclntab
  3c6790   598056 D go.string.*
  4620c0    49600 D gcbss
  7a7c20    45496 B runtime.mheap
  46e280    21936 D gcdata
  7a29e0    21056 b bufferList
  1ed600    16480 T crypto/tls.(*Conn).clientHandshake
  79eb20    16064 b semtable
  1b3d90    14224 T net/http.init

that seems plausible to me. some notes:

symtab is the plan 9 symbol table. it in the binary but never referenced at run time. it
supports things like nm -S only. it needs to move into an unmapped section of the
binary, but it is only costing at most 8k at run time right now due to fragmentation and
it just wasn't worth the effort to try to move. the new linker will make this easier. of
course, moving it in the file doesn't shrink the file.

the thing named pclntab is a reencoding of the original pclntab and the parts of the
plan 9 symbol table that we did need at run time (mostly just a list of functions and
their names and addresses). as you can see, it is much smaller than the old form (the
symbol table dominates).

type.* is the reflect types and go.string.* is the static go string data. the *
indicates that i coalesced many symbols into one, to avoid useless individual names
bloating the symbol table. if we tried we could probably cut the reflect types by 2-4x.
it would mean packing the data a bit more compactly than an ordinary go data structure
would and then using unsafe to get it back out.

gcbss and gcdata are garbage collection bits for the bss and data segments. that's what
atom symbol did, and it's not clear whether it will last (probably not) and whether what
will replace it will be smaller. time will tell. i have a meeting with dmitriy, carl,
and keith next week to figure out what the plan is.

runtime.mheap, bufferList, and semtable are bss.

you're not seeing the gdb dwarf debug information here, because it's not a runtime
symbol.
 
g% otool -l $(which toy) | egrep '^  segname|filesize'
  segname __PAGEZERO
 filesize 0
  segname __TEXT
 filesize 7811072
  segname __DATA
 filesize 126560
  segname __LINKEDIT
 filesize 921772
  segname __DWARF
 filesize 2886943
g% 

there's another 3 MB. you can build with -ldflags -w to get rid of that at least.
if you read the full otool -l output you will find

Load command 6
     cmd LC_SYMTAB
 cmdsize 24
  symoff 10825728
   nsyms 22559
  stroff 11186924
 strsize 560576

looks like another 1 MB or so (560576+11186924-10825728 or 22559*16+560576) for the
mach-o symbol table.

when we do the new linker we can make recording this kind of information in a useful
form a priority.
@robpike

This comment has been minimized.

Show comment
Hide comment
@robpike

robpike Nov 30, 2013

Contributor

Comment 1:

Note: the binaries were build on amd64 10.7.5 (Lion), with  gcc -version
i686-apple-darwin11-llvm-gcc-4.2
For the record, I couldn't do this experiment on 10.9 with Xcode 5 because the older
releases wouldn't build due to gcc/clang skew.
Contributor

robpike commented Nov 30, 2013

Comment 1:

Note: the binaries were build on amd64 10.7.5 (Lion), with  gcc -version
i686-apple-darwin11-llvm-gcc-4.2
For the record, I couldn't do this experiment on 10.9 with Xcode 5 because the older
releases wouldn't build due to gcc/clang skew.
@robpike

This comment has been minimized.

Show comment
Hide comment
@robpike

robpike Nov 30, 2013

Contributor

Comment 2:

Labels changed: added priority-later, removed priority-triage.

Contributor

robpike commented Nov 30, 2013

Comment 2:

Labels changed: added priority-later, removed priority-triage.

@gopherbot

This comment has been minimized.

Show comment
Hide comment
@gopherbot

gopherbot Dec 2, 2013

Comment 3 by jlourenco27:

Just for added reference, this are the size on go 1.1.2 vs 1.2 with OS X 10.9 and Xcode
5 (darwin gcc llvm 5.0 x86_64):
$ ls -l *1.*
-rwxr-xr-x  1 j      staff  1525984  2 Dez 21:44 hello_1.1.2
-rwxr-xr-x  1 j      staff  2192672  2 Dez 21:40 hello_1.2
$ size *1.*
__TEXT  __DATA  __OBJC  others  dec hex
1064960 94720   0   76000   1235680 12dae0  hello_1.1.2
1433600 147896  0   177440  1758936 1ad6d8  hello_1.2

Comment 3 by jlourenco27:

Just for added reference, this are the size on go 1.1.2 vs 1.2 with OS X 10.9 and Xcode
5 (darwin gcc llvm 5.0 x86_64):
$ ls -l *1.*
-rwxr-xr-x  1 j      staff  1525984  2 Dez 21:44 hello_1.1.2
-rwxr-xr-x  1 j      staff  2192672  2 Dez 21:40 hello_1.2
$ size *1.*
__TEXT  __DATA  __OBJC  others  dec hex
1064960 94720   0   76000   1235680 12dae0  hello_1.1.2
1433600 147896  0   177440  1758936 1ad6d8  hello_1.2
@randall77

This comment has been minimized.

Show comment
Hide comment
@randall77

randall77 Dec 3, 2013

Contributor

Comment 4:

This issue was updated by revision f238049.

R=golang-dev, rsc
CC=golang-dev
https://golang.org/cl/35940047
Contributor

randall77 commented Dec 3, 2013

Comment 4:

This issue was updated by revision f238049.

R=golang-dev, rsc
CC=golang-dev
https://golang.org/cl/35940047
@rsc

This comment has been minimized.

Show comment
Hide comment
@rsc

rsc Dec 4, 2013

Contributor

Comment 5:

Labels changed: added release-go1.3.

Contributor

rsc commented Dec 4, 2013

Comment 5:

Labels changed: added release-go1.3.

@rsc

This comment has been minimized.

Show comment
Hide comment
@rsc

rsc Dec 4, 2013

Contributor

Comment 6:

Labels changed: removed go1.3.

Contributor

rsc commented Dec 4, 2013

Comment 6:

Labels changed: removed go1.3.

@rsc

This comment has been minimized.

Show comment
Hide comment
@rsc

rsc Dec 4, 2013

Contributor

Comment 7:

Labels changed: added repo-main.

Contributor

rsc commented Dec 4, 2013

Comment 7:

Labels changed: added repo-main.

@zephyr

This comment has been minimized.

Show comment
Hide comment
@zephyr

zephyr Dec 7, 2013

Comment 8:

In this context, please reevaluate if the golang binaries can be changed to work with
UPX (ultimate packer for executables).¹²
For a small amount of computing power, upx can reduce the size of a binary down to a
quarter of its original size. You can authenticate this using a existing example
›fixer‹ programm for golang binaries on linux/amd64.³
While this approach doesn't fix the root of the problem – only the symptoms – it
would be nice to have this possibility always on hand.
For technical background of this problem (PT_LOAD[0].p_offset==0), please look at the
UPX bugtracker⁴.
¹ http://upx.sourceforge.net/
² https://en.wikipedia.org/wiki/UPX
³ https://github.com/pwaller/goupxhttp://sourceforge.net/p/upx/bugs/195/

zephyr commented Dec 7, 2013

Comment 8:

In this context, please reevaluate if the golang binaries can be changed to work with
UPX (ultimate packer for executables).¹²
For a small amount of computing power, upx can reduce the size of a binary down to a
quarter of its original size. You can authenticate this using a existing example
›fixer‹ programm for golang binaries on linux/amd64.³
While this approach doesn't fix the root of the problem – only the symptoms – it
would be nice to have this possibility always on hand.
For technical background of this problem (PT_LOAD[0].p_offset==0), please look at the
UPX bugtracker⁴.
¹ http://upx.sourceforge.net/
² https://en.wikipedia.org/wiki/UPX
³ https://github.com/pwaller/goupxhttp://sourceforge.net/p/upx/bugs/195/
@minux

This comment has been minimized.

Show comment
Hide comment
@minux

minux Dec 7, 2013

Member

Comment 9:

re #8, I don't think it's Go's problem. upx should be made more flexible to handle this.
Member

minux commented Dec 7, 2013

Comment 9:

re #8, I don't think it's Go's problem. upx should be made more flexible to handle this.
@leo-liu

This comment has been minimized.

Show comment
Hide comment
@leo-liu

leo-liu Dec 8, 2013

Comment 10:

minux:
We may figure out why the binaries are growing fast first (lager runtime, optimization,
etc.), before we claim that it isn't Go's problem.

leo-liu commented Dec 8, 2013

Comment 10:

minux:
We may figure out why the binaries are growing fast first (lager runtime, optimization,
etc.), before we claim that it isn't Go's problem.
@minux

This comment has been minimized.

Show comment
Hide comment
@minux

minux Dec 8, 2013

Member

Comment 11:

re #10, my #9 reply is to #8, which is about an entirely different problem.
i'm not saying that the ever-growing binaries is not our problem, only that i
don't believe that upx not accepting our binaries is our problem.
it's clear that upx isn't able to handle all possible and correct ELF files (i.e.
if the kernel can execute our binaries just fine, it's upx's problem to not be
able to compress them).
Member

minux commented Dec 8, 2013

Comment 11:

re #10, my #9 reply is to #8, which is about an entirely different problem.
i'm not saying that the ever-growing binaries is not our problem, only that i
don't believe that upx not accepting our binaries is our problem.
it's clear that upx isn't able to handle all possible and correct ELF files (i.e.
if the kernel can execute our binaries just fine, it's upx's problem to not be
able to compress them).
@robpike

This comment has been minimized.

Show comment
Hide comment
@robpike

robpike Feb 19, 2014

Contributor

Comment 12:

More detail. The Plan 9 symbol table is about to be deleted. Here is a reference point,
adding one new entry to the list above:
$ ls -l *1.*
-rwxr-xr-x  1 j      staff  1525984  2 Dez 21:44 hello_1.1.2
-rwxr-xr-x  1 j      staff  2192672  2 Dez 21:40 hello_1.2
-rwxr-xr-x  1 j      staff  2474512 Feb 18 20:27 hello_1.2.x
$ size *1.*
__TEXT  __DATA  __OBJC  others  dec hex
1064960 94720   0   76000   1235680 12dae0  hello_1.1.2
1433600 147896  0   177440  1758936 1ad6d8  hello_1.2
1699840 160984  0   188944  2049768 1f46e8     hello_1.2.x
Text has grown substantially, as has data. At least some of this is due to new
annotations for the garbage collector.
Contributor

robpike commented Feb 19, 2014

Comment 12:

More detail. The Plan 9 symbol table is about to be deleted. Here is a reference point,
adding one new entry to the list above:
$ ls -l *1.*
-rwxr-xr-x  1 j      staff  1525984  2 Dez 21:44 hello_1.1.2
-rwxr-xr-x  1 j      staff  2192672  2 Dez 21:40 hello_1.2
-rwxr-xr-x  1 j      staff  2474512 Feb 18 20:27 hello_1.2.x
$ size *1.*
__TEXT  __DATA  __OBJC  others  dec hex
1064960 94720   0   76000   1235680 12dae0  hello_1.1.2
1433600 147896  0   177440  1758936 1ad6d8  hello_1.2
1699840 160984  0   188944  2049768 1f46e8     hello_1.2.x
Text has grown substantially, as has data. At least some of this is due to new
annotations for the garbage collector.
@robpike

This comment has been minimized.

Show comment
Hide comment
@robpike

robpike Feb 19, 2014

Contributor

Comment 13:

More detail. The Plan 9 symbol table is about to be deleted. Here is a reference point,
adding one new entry to the list above:
$ ls -l *1.*
-rwxr-xr-x  1 r  staff  1191952 Nov 30 10:25 x.1.0
-rwxr-xr-x  1 r  staff  1525936 Nov 30 10:20 x.1.1
-rwxr-xr-x  1 r  staff  2188576 Nov 30 10:18 x.1.2
-rwxr-xr-x  1 r  staff  2474512 Feb 18 20:27 hello_1.2.x
$ size *1.*
__TEXT  __DATA  __OBJC  others  dec hex
880640  33682096     0  4112    34566848     20f72c0    x.1.0
1064960 94656   0   75952   1235568 12da70  x.1.1
1429504 147896  0   177440  1754840 1ac6d8  x.1.2
1699840 160984  0   188944  2049768 1f46e8     hello_1.2.x
Text has grown substantially, as has data. At least some of this is due to new
annotations for the garbage collector.
Contributor

robpike commented Feb 19, 2014

Comment 13:

More detail. The Plan 9 symbol table is about to be deleted. Here is a reference point,
adding one new entry to the list above:
$ ls -l *1.*
-rwxr-xr-x  1 r  staff  1191952 Nov 30 10:25 x.1.0
-rwxr-xr-x  1 r  staff  1525936 Nov 30 10:20 x.1.1
-rwxr-xr-x  1 r  staff  2188576 Nov 30 10:18 x.1.2
-rwxr-xr-x  1 r  staff  2474512 Feb 18 20:27 hello_1.2.x
$ size *1.*
__TEXT  __DATA  __OBJC  others  dec hex
880640  33682096     0  4112    34566848     20f72c0    x.1.0
1064960 94656   0   75952   1235568 12da70  x.1.1
1429504 147896  0   177440  1754840 1ac6d8  x.1.2
1699840 160984  0   188944  2049768 1f46e8     hello_1.2.x
Text has grown substantially, as has data. At least some of this is due to new
annotations for the garbage collector.
@robpike

This comment has been minimized.

Show comment
Hide comment
@robpike

robpike Feb 19, 2014

Contributor

Comment 14:

More detail. The Plan 9 symbol table is about to be deleted. Here is a reference point,
adding one new entry to the list above:
$ ls -l *1.*
-rwxr-xr-x  1 r  staff  1191952 Nov 30 10:25 x.1.0
-rwxr-xr-x  1 r  staff  1525936 Nov 30 10:20 x.1.1
-rwxr-xr-x  1 r  staff  2188576 Nov 30 10:18 x.1.2
-rwxr-xr-x  1 r  staff  2474512 Feb 18 20:27 hello_1.2.x
$ size *1.*
__TEXT  __DATA  __OBJC  others  dec hex
880640  33682096     0  4112    34566848     20f72c0    x.1.0
1064960 94656   0   75952   1235568 12da70  x.1.1
1429504 147896  0   177440  1754840 1ac6d8  x.1.2
1699840 160984  0   188944  2049768 1f46e8     x.1.2.x
Text has grown substantially, as has data. At least some of this is due to new
annotations for the garbage collector.
Contributor

robpike commented Feb 19, 2014

Comment 14:

More detail. The Plan 9 symbol table is about to be deleted. Here is a reference point,
adding one new entry to the list above:
$ ls -l *1.*
-rwxr-xr-x  1 r  staff  1191952 Nov 30 10:25 x.1.0
-rwxr-xr-x  1 r  staff  1525936 Nov 30 10:20 x.1.1
-rwxr-xr-x  1 r  staff  2188576 Nov 30 10:18 x.1.2
-rwxr-xr-x  1 r  staff  2474512 Feb 18 20:27 hello_1.2.x
$ size *1.*
__TEXT  __DATA  __OBJC  others  dec hex
880640  33682096     0  4112    34566848     20f72c0    x.1.0
1064960 94656   0   75952   1235568 12da70  x.1.1
1429504 147896  0   177440  1754840 1ac6d8  x.1.2
1699840 160984  0   188944  2049768 1f46e8     x.1.2.x
Text has grown substantially, as has data. At least some of this is due to new
annotations for the garbage collector.
@rsc

This comment has been minimized.

Show comment
Hide comment
@rsc

rsc Feb 19, 2014

Contributor

Comment 15:

This issue was updated by revision 964f6d3.

Nothing reads the Plan 9 symbol table anymore.
The last holdout was 'go tool nm', but since being rewritten in Go
it uses the standard symbol table for the binary format
(ELF, Mach-O, PE) instead.
Removing the Plan 9 symbol table saves ~15% disk space
on most binaries.
Two supporting changes included in this CL:
debug/gosym: use Go 1.2 pclntab to synthesize func-only
symbol table when there is no Plan 9 symbol table
debug/elf, debug/macho, debug/pe: ignore final EOF from ReadAt
LGTM=r
R=r, bradfitz
CC=golang-codereviews
https://golang.org/cl/65740045
Contributor

rsc commented Feb 19, 2014

Comment 15:

This issue was updated by revision 964f6d3.

Nothing reads the Plan 9 symbol table anymore.
The last holdout was 'go tool nm', but since being rewritten in Go
it uses the standard symbol table for the binary format
(ELF, Mach-O, PE) instead.
Removing the Plan 9 symbol table saves ~15% disk space
on most binaries.
Two supporting changes included in this CL:
debug/gosym: use Go 1.2 pclntab to synthesize func-only
symbol table when there is no Plan 9 symbol table
debug/elf, debug/macho, debug/pe: ignore final EOF from ReadAt
LGTM=r
R=r, bradfitz
CC=golang-codereviews
https://golang.org/cl/65740045
@bradfitz

This comment has been minimized.

Show comment
Hide comment
@bradfitz

bradfitz Feb 19, 2014

Member

Comment 16:

After revision 737767dd81fd, I see a 25% reduction:
Before:
-rwxr-xr-x  1 bradfitz  staff  23556028 Feb 18 20:47 bin/camlistored
After:
-rwxr-xr-x  1 bradfitz  staff  17727420 Feb 18 20:48 bin/camlistored
Member

bradfitz commented Feb 19, 2014

Comment 16:

After revision 737767dd81fd, I see a 25% reduction:
Before:
-rwxr-xr-x  1 bradfitz  staff  23556028 Feb 18 20:47 bin/camlistored
After:
-rwxr-xr-x  1 bradfitz  staff  17727420 Feb 18 20:48 bin/camlistored
@robpike

This comment has been minimized.

Show comment
Hide comment
@robpike

robpike Feb 19, 2014

Contributor

Comment 17:

For my test case before/after deleting the Plan 9 symbol table:
% ls -l ...
-rwxr-xr-x  1 r  staff  2474512 Feb 18 20:27 hello_1.2.x
-rwxr-xr-x  1 r  staff  2150928 Feb 18 22:28 hello_1.2.y
% size ...
__TEXT  __DATA  __OBJC  others  dec hex
1699840 160984  0   188944  2049768 1f46e8     hello_1.2.x
1376256 160984  0   188944  1726184 1a56e8    hello_1.2.x
% 
So deleting the Plan 9 symbol table pretty close to exactly compensates for the GC
information. We're back at Go 1.2 levels, still far too large but it's a start.
Contributor

robpike commented Feb 19, 2014

Comment 17:

For my test case before/after deleting the Plan 9 symbol table:
% ls -l ...
-rwxr-xr-x  1 r  staff  2474512 Feb 18 20:27 hello_1.2.x
-rwxr-xr-x  1 r  staff  2150928 Feb 18 22:28 hello_1.2.y
% size ...
__TEXT  __DATA  __OBJC  others  dec hex
1699840 160984  0   188944  2049768 1f46e8     hello_1.2.x
1376256 160984  0   188944  1726184 1a56e8    hello_1.2.x
% 
So deleting the Plan 9 symbol table pretty close to exactly compensates for the GC
information. We're back at Go 1.2 levels, still far too large but it's a start.
@rsc

This comment has been minimized.

Show comment
Hide comment
@rsc

rsc Feb 19, 2014

Contributor

Comment 18:

This issue was updated by revision 2541cc8.

Every function now has a gcargs and gclocals symbol
holding associated garbage collection information.
Put them all in the same meta-symbol as the go.func data
and then drop individual entries from symbol table.
Removing gcargs and gclocals reduces the size of a
typical binary by 10%.
LGTM=r
R=r
CC=golang-codereviews
https://golang.org/cl/65870044
Contributor

rsc commented Feb 19, 2014

Comment 18:

This issue was updated by revision 2541cc8.

Every function now has a gcargs and gclocals symbol
holding associated garbage collection information.
Put them all in the same meta-symbol as the go.func data
and then drop individual entries from symbol table.
Removing gcargs and gclocals reduces the size of a
typical binary by 10%.
LGTM=r
R=r
CC=golang-codereviews
https://golang.org/cl/65870044
@rsc

This comment has been minimized.

Show comment
Hide comment
@rsc

rsc Feb 19, 2014

Contributor

Comment 19:

This issue was updated by revision ae38b03.

For an ephemeral binary - one created, run, and then deleted -
there is no need to write dwarf debug information, since the
binary will not be used with gdb. In this case, instruct the linker
not to spend time and disk space generating the debug information
by passing the -w flag to the linker.
Omitting dwarf information reduces the size of most binaries by 25%.
We may be more aggressive about this in the future.
LGTM=bradfitz, r
R=r, bradfitz
CC=golang-codereviews
https://golang.org/cl/65890043
Contributor

rsc commented Feb 19, 2014

Comment 19:

This issue was updated by revision ae38b03.

For an ephemeral binary - one created, run, and then deleted -
there is no need to write dwarf debug information, since the
binary will not be used with gdb. In this case, instruct the linker
not to spend time and disk space generating the debug information
by passing the -w flag to the linker.
Omitting dwarf information reduces the size of most binaries by 25%.
We may be more aggressive about this in the future.
LGTM=bradfitz, r
R=r, bradfitz
CC=golang-codereviews
https://golang.org/cl/65890043
@robpike

This comment has been minimized.

Show comment
Hide comment
@robpike

robpike Feb 19, 2014

Contributor

Comment 20:

After removing gcargs from the symbol table (stepping across CL 65870044)
% ls -l x.1.2.[yz]
-rwxr-xr-x  1 r  staff  2150928 Feb 18 22:28 hello_1.2.y
-rwxr-xr-x  1 r  staff  1932880 Feb 19 08:14 hello_1.2.z
% size x.1.2.[yz] 
__TEXT  __DATA  __OBJC  others  dec hex
1376256 160984  0   188944  1726184 1a56e8    hello_1.2.y
1376256 160984  0   110160  1647400 192328 hello_1.2.z
% 
It's now smaller than at 1.2 but still much bigger than 1.1, let alone 1.0.
Contributor

robpike commented Feb 19, 2014

Comment 20:

After removing gcargs from the symbol table (stepping across CL 65870044)
% ls -l x.1.2.[yz]
-rwxr-xr-x  1 r  staff  2150928 Feb 18 22:28 hello_1.2.y
-rwxr-xr-x  1 r  staff  1932880 Feb 19 08:14 hello_1.2.z
% size x.1.2.[yz] 
__TEXT  __DATA  __OBJC  others  dec hex
1376256 160984  0   188944  1726184 1a56e8    hello_1.2.y
1376256 160984  0   110160  1647400 192328 hello_1.2.z
% 
It's now smaller than at 1.2 but still much bigger than 1.1, let alone 1.0.
@gopherbot

This comment has been minimized.

Show comment
Hide comment
@gopherbot

gopherbot Mar 4, 2014

Comment 21:

I would like to take a look at compressing pclntab. Is it compressible?

Comment 21:

I would like to take a look at compressing pclntab. Is it compressible?
@ianlancetaylor

This comment has been minimized.

Show comment
Hide comment
@ianlancetaylor

ianlancetaylor Mar 4, 2014

Contributor

Comment 22:

The pclntab data should be reasonably compact, and note that fast access to the data is
important, since it is used for runtime.Callers and friends.  If you can find a
significant reduction in size that would be great, but small tweaks are probably not
desirable at this point.
Contributor

ianlancetaylor commented Mar 4, 2014

Comment 22:

The pclntab data should be reasonably compact, and note that fast access to the data is
important, since it is used for runtime.Callers and friends.  If you can find a
significant reduction in size that would be great, but small tweaks are probably not
desirable at this point.
@gopherbot

This comment has been minimized.

Show comment
Hide comment
@gopherbot

gopherbot Mar 4, 2014

Comment 23 by fuzxxl:

Is there any documentation for what the pclntab contains and for what constraints its
data structure must fullfill?

Comment 23 by fuzxxl:

Is there any documentation for what the pclntab contains and for what constraints its
data structure must fullfill?
@ianlancetaylor

This comment has been minimized.

Show comment
Hide comment
@ianlancetaylor

ianlancetaylor Mar 4, 2014

Contributor

Comment 24:

Let's not use this issue as a discussion list.  Please ask questions on golang-dev. 
Thanks.
Contributor

ianlancetaylor commented Mar 4, 2014

Comment 24:

Let's not use this issue as a discussion list.  Please ask questions on golang-dev. 
Thanks.
@gopherbot

This comment has been minimized.

Show comment
Hide comment
@gopherbot

gopherbot Mar 6, 2014

Comment 25 by allard.guy.m:

From the peanut gallery, AFAICT this breaks pprof interactive command 'list'.  At tip I
get, e.g.:
(pprof) list runner
Total: 6424 samples
objdump: syminit: Success
no filename found in main.runner<400c40>
Which works as expected with 1.2.1.

Comment 25 by allard.guy.m:

From the peanut gallery, AFAICT this breaks pprof interactive command 'list'.  At tip I
get, e.g.:
(pprof) list runner
Total: 6424 samples
objdump: syminit: Success
no filename found in main.runner<400c40>
Which works as expected with 1.2.1.
@rsc

This comment has been minimized.

Show comment
Hide comment
@rsc

rsc Apr 3, 2014

Contributor

Comment 26:

Let's not use this issue as a discussion list.  Please ask questions on golang-dev. 
Thanks.
pprof not working is issue #7452.

Labels changed: added restrict-addissuecomment-commit.

Contributor

rsc commented Apr 3, 2014

Comment 26:

Let's not use this issue as a discussion list.  Please ask questions on golang-dev. 
Thanks.
pprof not working is issue #7452.

Labels changed: added restrict-addissuecomment-commit.

@rsc

This comment has been minimized.

Show comment
Hide comment
@rsc

rsc Apr 3, 2014

Contributor

Comment 27:

This is as fixed as it is going to be for Go 1.3.
Right now at tip + CL 80370045 on darwin/amd64, compiling this program:
package main
import "fmt"
func main() {
    fmt.Println("hello, world")
}
I get 1830352 bytes for the binary. Assuming this is the same case for which Rob's
numbers are reported, by this metric Go 1.3 will roll back more than half the size
increase caused by Go 1.2 (relative to Go 1.1). Will leave further improvement for Go
1.4.

Labels changed: added release-go1.4, removed release-go1.3.

Contributor

rsc commented Apr 3, 2014

Comment 27:

This is as fixed as it is going to be for Go 1.3.
Right now at tip + CL 80370045 on darwin/amd64, compiling this program:
package main
import "fmt"
func main() {
    fmt.Println("hello, world")
}
I get 1830352 bytes for the binary. Assuming this is the same case for which Rob's
numbers are reported, by this metric Go 1.3 will roll back more than half the size
increase caused by Go 1.2 (relative to Go 1.1). Will leave further improvement for Go
1.4.

Labels changed: added release-go1.4, removed release-go1.3.

@rsc

This comment has been minimized.

Show comment
Hide comment
@rsc

rsc Apr 3, 2014

Contributor

Comment 28:

This issue was updated by revision a26c01a.

LGTM=khr
R=khr
CC=golang-codereviews
https://golang.org/cl/80370045
Contributor

rsc commented Apr 3, 2014

Comment 28:

This issue was updated by revision a26c01a.

LGTM=khr
R=khr
CC=golang-codereviews
https://golang.org/cl/80370045
@rsc

This comment has been minimized.

Show comment
Hide comment
@rsc

rsc Sep 15, 2014

Contributor

Comment 29:

Labels changed: added release-go1.5, removed release-go1.4.

Contributor

rsc commented Sep 15, 2014

Comment 29:

Labels changed: added release-go1.5, removed release-go1.4.

@adg adg locked and limited conversation to collaborators Dec 8, 2014

@bradfitz bradfitz modified the milestone: Go1.5 Dec 16, 2014

@rsc rsc removed the priority-later label Apr 10, 2015

@gopherbot

This comment has been minimized.

Show comment
Hide comment
@gopherbot

gopherbot Mar 29, 2016

CL https://golang.org/cl/21284 mentions this issue.

CL https://golang.org/cl/21284 mentions this issue.

@gopherbot

This comment has been minimized.

Show comment
Hide comment
@gopherbot

gopherbot Apr 1, 2016

CL https://golang.org/cl/21395 mentions this issue.

CL https://golang.org/cl/21395 mentions this issue.

@gopherbot

This comment has been minimized.

Show comment
Hide comment
@gopherbot

gopherbot Apr 1, 2016

CL https://golang.org/cl/21396 mentions this issue.

CL https://golang.org/cl/21396 mentions this issue.

@gopherbot

This comment has been minimized.

Show comment
Hide comment
@gopherbot

gopherbot Apr 6, 2016

CL https://golang.org/cl/21583 mentions this issue.

CL https://golang.org/cl/21583 mentions this issue.

@gopherbot

This comment has been minimized.

Show comment
Hide comment
@gopherbot

gopherbot Apr 11, 2016

CL https://golang.org/cl/21777 mentions this issue.

CL https://golang.org/cl/21777 mentions this issue.

@gopherbot

This comment has been minimized.

Show comment
Hide comment
@gopherbot

gopherbot Apr 11, 2016

CL https://golang.org/cl/21776 mentions this issue.

CL https://golang.org/cl/21776 mentions this issue.

@gopherbot

This comment has been minimized.

Show comment
Hide comment
@gopherbot

gopherbot Apr 22, 2016

CL https://golang.org/cl/22371 mentions this issue.

CL https://golang.org/cl/22371 mentions this issue.

@gopherbot

This comment has been minimized.

Show comment
Hide comment
@gopherbot

gopherbot Apr 22, 2016

CL https://golang.org/cl/22373 mentions this issue.

CL https://golang.org/cl/22373 mentions this issue.

@gopherbot

This comment has been minimized.

Show comment
Hide comment
@gopherbot

gopherbot Apr 24, 2016

CL https://golang.org/cl/22395 mentions this issue.

CL https://golang.org/cl/22395 mentions this issue.

@rsc

This comment has been minimized.

Show comment
Hide comment
@rsc

rsc May 17, 2016

Contributor

Great progress this release cycle. Kicking open issue to next cycle because there's still a bit more to do. But at least the "and growing" has been reversed for one cycle.

Contributor

rsc commented May 17, 2016

Great progress this release cycle. Kicking open issue to next cycle because there's still a bit more to do. But at least the "and growing" has been reversed for one cycle.

@rsc rsc modified the milestones: Go1.8, Go1.7 May 17, 2016

@quentinmit quentinmit added the NeedsFix label Oct 5, 2016

@quentinmit

This comment has been minimized.

Show comment
Hide comment
@quentinmit

quentinmit Oct 5, 2016

Contributor

@rsc It seems like we haven't done any work on this for 1.8. I don't think keeping an umbrella issue like this open is useful. Is there anything in particular you had in mind for this issue, or should we close it now?

Contributor

quentinmit commented Oct 5, 2016

@rsc It seems like we haven't done any work on this for 1.8. I don't think keeping an umbrella issue like this open is useful. Is there anything in particular you had in mind for this issue, or should we close it now?

@crawshaw

This comment has been minimized.

Show comment
Hide comment
@crawshaw

crawshaw Oct 5, 2016

Contributor

I have some more typeOff work I was hoping to do in 1.8 (the continuation of the work linked here done in 1.7), though I am busy in other projects right now. Let's at least leave this open until the window closes in a few weeks.

Contributor

crawshaw commented Oct 5, 2016

I have some more typeOff work I was hoping to do in 1.8 (the continuation of the work linked here done in 1.7), though I am busy in other projects right now. Let's at least leave this open until the window closes in a few weeks.

@rsc

This comment has been minimized.

Show comment
Hide comment
@rsc

rsc Oct 6, 2016

Contributor

We have very few umbrella issues, but this one is fine to keep. It's good to keep in mind and collect all the work toward this. The binaries are still much bigger than we hope they would be.

Contributor

rsc commented Oct 6, 2016

We have very few umbrella issues, but this one is fine to keep. It's good to keep in mind and collect all the work toward this. The binaries are still much bigger than we hope they would be.

@rsc rsc modified the milestones: Go1.9, Go1.8 Oct 6, 2016

@gopherbot

This comment has been minimized.

Show comment
Hide comment
@gopherbot

gopherbot May 10, 2017

CL https://golang.org/cl/43090 mentions this issue.

CL https://golang.org/cl/43090 mentions this issue.

@gopherbot

This comment has been minimized.

Show comment
Hide comment
@gopherbot

gopherbot May 10, 2017

CL https://golang.org/cl/43190 mentions this issue.

CL https://golang.org/cl/43190 mentions this issue.

@gopherbot

This comment has been minimized.

Show comment
Hide comment
@gopherbot

gopherbot May 24, 2017

CL https://golang.org/cl/44007 mentions this issue.

CL https://golang.org/cl/44007 mentions this issue.

@bradfitz

This comment has been minimized.

Show comment
Hide comment
@bradfitz

bradfitz Jun 8, 2017

Member

No notable wins (maybe a slight loss) at tip (to-be Go 1.9) compared to Go 1.8:

bradfitz@gdev:~$ for x in go1.4 go1.5 go1.6 go1.7 go1.8 go; do ls -l $x/bin/go; done
-rwxr-xr-x 1 bradfitz bradfitz 9571864 Feb 18  2015 go1.4/bin/go
-rwxr-xr-x 1 bradfitz bradfitz 11290832 Jan 13  2016 go1.5/bin/go
-rwxr-xr-x 1 bradfitz bradfitz 12534640 Jul 18  2016 go1.6/bin/go
-rwxr-xr-x 1 bradfitz bradfitz 9953979 Aug 15  2016 go1.7/bin/go
-rwxr-xr-x 1 bradfitz bradfitz 10068917 Feb 16 19:28 go1.8/bin/go
-rwxr-xr-x 1 bradfitz bradfitz 10346229 Jun  6 00:03 go/bin/go
bradfitz@gdev:~$ for x in go1.4 go1.5 go1.6 go1.7 go1.8 go; do ls -l $x/pkg/tool/linux_amd64/cgo; done
-rwxr-xr-x 1 bradfitz bradfitz 4560376 Feb 18  2015 go1.4/pkg/tool/linux_amd64/cgo
-rwxr-xr-x 1 bradfitz bradfitz 5111280 Jan 13  2016 go1.5/pkg/tool/linux_amd64/cgo
-rwxr-xr-x 1 bradfitz bradfitz 5279368 Jul 18  2016 go1.6/pkg/tool/linux_amd64/cgo
-rwxr-xr-x 1 bradfitz bradfitz 4114079 Aug 15  2016 go1.7/pkg/tool/linux_amd64/cgo
-rwxr-xr-x 1 bradfitz bradfitz 3914818 Feb 16 19:28 go1.8/pkg/tool/linux_amd64/cgo
-rwxr-xr-x 1 bradfitz bradfitz 4015485 Jun  6 00:03 go/pkg/tool/linux_amd64/cgo
bradfitz@gdev:~$ for x in go1.4 go1.5 go1.6 go1.7 go1.8 go; do ls -l $x/pkg/tool/linux_amd64/objdump; done
-rwxr-xr-x 1 bradfitz bradfitz 4408008 Feb 18  2015 go1.4/pkg/tool/linux_amd64/objdump
-rwxr-xr-x 1 bradfitz bradfitz 4889240 Jan 13  2016 go1.5/pkg/tool/linux_amd64/objdump
-rwxr-xr-x 1 bradfitz bradfitz 4988648 Jul 18  2016 go1.6/pkg/tool/linux_amd64/objdump
-rwxr-xr-x 1 bradfitz bradfitz 3622669 Aug 15  2016 go1.7/pkg/tool/linux_amd64/objdump
-rwxr-xr-x 1 bradfitz bradfitz 3717826 Feb 16 19:28 go1.8/pkg/tool/linux_amd64/objdump
-rwxr-xr-x 1 bradfitz bradfitz 3836105 Jun  6 00:03 go/pkg/tool/linux_amd64/objdump
bradfitz@gdev:~$ for x in go1.4 go1.5 go1.6 go1.7 go1.8 go; do ls -l $x/bin/gofmt; done
-rwxr-xr-x 1 bradfitz bradfitz 3594744 May 11  2016 go1.4/bin/gofmt
-rwxr-xr-x 1 bradfitz bradfitz 3944064 Jan 13  2016 go1.5/bin/gofmt
-rwxr-xr-x 1 bradfitz bradfitz 3895320 Jul 18  2016 go1.6/bin/gofmt
-rwxr-xr-x 1 bradfitz bradfitz 3036195 Aug 15  2016 go1.7/bin/gofmt
-rwxr-xr-x 1 bradfitz bradfitz 3481554 Feb 16 19:28 go1.8/bin/gofmt
-rwxr-xr-x 1 bradfitz bradfitz 3257512 Jun  6 00:03 go/bin/gofmt

I'm going to move this ongoing tracking bug to Go 1.10, since I don't see anything more happening for Go 1.9.

Member

bradfitz commented Jun 8, 2017

No notable wins (maybe a slight loss) at tip (to-be Go 1.9) compared to Go 1.8:

bradfitz@gdev:~$ for x in go1.4 go1.5 go1.6 go1.7 go1.8 go; do ls -l $x/bin/go; done
-rwxr-xr-x 1 bradfitz bradfitz 9571864 Feb 18  2015 go1.4/bin/go
-rwxr-xr-x 1 bradfitz bradfitz 11290832 Jan 13  2016 go1.5/bin/go
-rwxr-xr-x 1 bradfitz bradfitz 12534640 Jul 18  2016 go1.6/bin/go
-rwxr-xr-x 1 bradfitz bradfitz 9953979 Aug 15  2016 go1.7/bin/go
-rwxr-xr-x 1 bradfitz bradfitz 10068917 Feb 16 19:28 go1.8/bin/go
-rwxr-xr-x 1 bradfitz bradfitz 10346229 Jun  6 00:03 go/bin/go
bradfitz@gdev:~$ for x in go1.4 go1.5 go1.6 go1.7 go1.8 go; do ls -l $x/pkg/tool/linux_amd64/cgo; done
-rwxr-xr-x 1 bradfitz bradfitz 4560376 Feb 18  2015 go1.4/pkg/tool/linux_amd64/cgo
-rwxr-xr-x 1 bradfitz bradfitz 5111280 Jan 13  2016 go1.5/pkg/tool/linux_amd64/cgo
-rwxr-xr-x 1 bradfitz bradfitz 5279368 Jul 18  2016 go1.6/pkg/tool/linux_amd64/cgo
-rwxr-xr-x 1 bradfitz bradfitz 4114079 Aug 15  2016 go1.7/pkg/tool/linux_amd64/cgo
-rwxr-xr-x 1 bradfitz bradfitz 3914818 Feb 16 19:28 go1.8/pkg/tool/linux_amd64/cgo
-rwxr-xr-x 1 bradfitz bradfitz 4015485 Jun  6 00:03 go/pkg/tool/linux_amd64/cgo
bradfitz@gdev:~$ for x in go1.4 go1.5 go1.6 go1.7 go1.8 go; do ls -l $x/pkg/tool/linux_amd64/objdump; done
-rwxr-xr-x 1 bradfitz bradfitz 4408008 Feb 18  2015 go1.4/pkg/tool/linux_amd64/objdump
-rwxr-xr-x 1 bradfitz bradfitz 4889240 Jan 13  2016 go1.5/pkg/tool/linux_amd64/objdump
-rwxr-xr-x 1 bradfitz bradfitz 4988648 Jul 18  2016 go1.6/pkg/tool/linux_amd64/objdump
-rwxr-xr-x 1 bradfitz bradfitz 3622669 Aug 15  2016 go1.7/pkg/tool/linux_amd64/objdump
-rwxr-xr-x 1 bradfitz bradfitz 3717826 Feb 16 19:28 go1.8/pkg/tool/linux_amd64/objdump
-rwxr-xr-x 1 bradfitz bradfitz 3836105 Jun  6 00:03 go/pkg/tool/linux_amd64/objdump
bradfitz@gdev:~$ for x in go1.4 go1.5 go1.6 go1.7 go1.8 go; do ls -l $x/bin/gofmt; done
-rwxr-xr-x 1 bradfitz bradfitz 3594744 May 11  2016 go1.4/bin/gofmt
-rwxr-xr-x 1 bradfitz bradfitz 3944064 Jan 13  2016 go1.5/bin/gofmt
-rwxr-xr-x 1 bradfitz bradfitz 3895320 Jul 18  2016 go1.6/bin/gofmt
-rwxr-xr-x 1 bradfitz bradfitz 3036195 Aug 15  2016 go1.7/bin/gofmt
-rwxr-xr-x 1 bradfitz bradfitz 3481554 Feb 16 19:28 go1.8/bin/gofmt
-rwxr-xr-x 1 bradfitz bradfitz 3257512 Jun  6 00:03 go/bin/gofmt

I'm going to move this ongoing tracking bug to Go 1.10, since I don't see anything more happening for Go 1.9.

@bradfitz bradfitz modified the milestones: Go1.10, Go1.9 Jun 8, 2017

@gopherbot

This comment has been minimized.

Show comment
Hide comment
@gopherbot

gopherbot Aug 24, 2017

Change https://golang.org/cl/57130 mentions this issue: cmd/compile/internal/ssa: combine consecutive loads and stores on amd64

Change https://golang.org/cl/57130 mentions this issue: cmd/compile/internal/ssa: combine consecutive loads and stores on amd64

@mvdan

This comment has been minimized.

Show comment
Hide comment
@mvdan

mvdan Aug 28, 2017

Member

This issue reports slight increases between 1.8 and 1.9: #21653

Member

mvdan commented Aug 28, 2017

This issue reports slight increases between 1.8 and 1.9: #21653

@gopherbot

This comment has been minimized.

Show comment
Hide comment
@gopherbot

gopherbot Sep 9, 2017

Change https://golang.org/cl/61190 mentions this issue: cmd/compile: specialize map creation for small hint sizes

Change https://golang.org/cl/61190 mentions this issue: cmd/compile: specialize map creation for small hint sizes

@rsc rsc modified the milestones: Go1.10, Go1.11 Nov 22, 2017

@gopherbot

This comment has been minimized.

Show comment
Hide comment
@gopherbot

gopherbot Jan 17, 2018

Change https://golang.org/cl/88135 mentions this issue: cmd/compile: don't combine 64-bit loads/stores on amd64

Change https://golang.org/cl/88135 mentions this issue: cmd/compile: don't combine 64-bit loads/stores on amd64

@FiloSottile

This comment has been minimized.

Show comment
Hide comment
@FiloSottile

FiloSottile Feb 20, 2018

Member

#23934 reports an ~8% increase between 1.9.4 and 1.10 for default binaries, and a slight decrease with -ldflags="-s -w". (#11799 is relevant to debug info size.)

Member

FiloSottile commented Feb 20, 2018

#23934 reports an ~8% increase between 1.9.4 and 1.10 for default binaries, and a slight decrease with -ldflags="-s -w". (#11799 is relevant to debug info size.)

navytux referenced this issue Mar 7, 2018

encoding/binary: use an offset instead of slicing
While running make.bash, over 5% of all pointer writes
come from encoding/binary doing struct reads.

This change replaces slicing during such reads with an offset.
This avoids updating the slice pointer with every
struct field read or write.

This has no impact when the write barrier is off.
Running the benchmarks with GOGC=1, however,
shows significant improvement:

name          old time/op    new time/op    delta
ReadStruct-8    13.2µs ± 6%    10.1µs ± 5%  -23.24%  (p=0.000 n=10+10)

name          old speed      new speed      delta
ReadStruct-8  5.69MB/s ± 6%  7.40MB/s ± 5%  +30.18%  (p=0.000 n=10+10)

Change-Id: I22904263196bfeddc38abe8989428e263aee5253
Reviewed-on: https://go-review.googlesource.com/98757
Run-TryBot: Josh Bleecher Snyder <josharian@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
@ALTree

This comment has been minimized.

Show comment
Hide comment
@ALTree

ALTree Apr 27, 2018

Member

I'm seeing a significant regression in binaries sizes between go1.10.1 and the current tip on linux/amd64. A simple hello world is 26% bigger than it was on 1.10:

$ cat test.go 
package main

import "fmt"

func main() {
	fmt.Println("hi!")
}

$ go version
go version go1.10.1 linux/amd64
$ go build test.go
$ ls -l test
-rwxr-xr-x 1 alberto alberto 2011612 Apr 27 11:14 test

$ gotip version
go version devel +a3bafcf8cc Thu Apr 26 18:26:06 2018 +0000 linux/amd64
$ gotip build test.go
$ ls -l test
-rwxr-xr-x 1 alberto alberto 2543972 Apr 27 11:15 test

It's not just the hello world. The go binary went from 11MB to 15MB. gofmt from 3.4MB to 4.3MB.

Is this expected?

Member

ALTree commented Apr 27, 2018

I'm seeing a significant regression in binaries sizes between go1.10.1 and the current tip on linux/amd64. A simple hello world is 26% bigger than it was on 1.10:

$ cat test.go 
package main

import "fmt"

func main() {
	fmt.Println("hi!")
}

$ go version
go version go1.10.1 linux/amd64
$ go build test.go
$ ls -l test
-rwxr-xr-x 1 alberto alberto 2011612 Apr 27 11:14 test

$ gotip version
go version devel +a3bafcf8cc Thu Apr 26 18:26:06 2018 +0000 linux/amd64
$ gotip build test.go
$ ls -l test
-rwxr-xr-x 1 alberto alberto 2543972 Apr 27 11:15 test

It's not just the hello world. The go binary went from 11MB to 15MB. gofmt from 3.4MB to 4.3MB.

Is this expected?

@mvdan

This comment has been minimized.

Show comment
Hide comment
@mvdan

mvdan Apr 27, 2018

Member

@ALTree have you tried the same tests with -ldflags="-w -s"? There have been many changes to debugging info recently, so perhaps it's just that the binaries contain more debug info.

Member

mvdan commented Apr 27, 2018

@ALTree have you tried the same tests with -ldflags="-w -s"? There have been many changes to debugging info recently, so perhaps it's just that the binaries contain more debug info.

@dominikh

This comment has been minimized.

Show comment
Hide comment
@dominikh

dominikh Apr 27, 2018

Member

Generally speaking we do not recommend stripping debug info (it's useful, after all), so that's only useful for figuring out why they grew, not a justification for the growth.

Member

dominikh commented Apr 27, 2018

Generally speaking we do not recommend stripping debug info (it's useful, after all), so that's only useful for figuring out why they grew, not a justification for the growth.

@ALTree

This comment has been minimized.

Show comment
Hide comment
@ALTree

ALTree Apr 27, 2018

Member

@mvdan Yes, and stripped binaries are the same size as in 1.10, but that's not the default...

But yeah, if this is an expected effect of the recent work on debug info then... ok, I guess? I just wanted to make sure this was intentional.

Member

ALTree commented Apr 27, 2018

@mvdan Yes, and stripped binaries are the same size as in 1.10, but that's not the default...

But yeah, if this is an expected effect of the recent work on debug info then... ok, I guess? I just wanted to make sure this was intentional.

@mvdan

This comment has been minimized.

Show comment
Hide comment
@mvdan

mvdan Apr 27, 2018

Member

I should have been explicit; I meant it only as a way to quickly figure out where the growth was coming from.

Member

mvdan commented Apr 27, 2018

I should have been explicit; I meant it only as a way to quickly figure out where the growth was coming from.

@josharian

This comment has been minimized.

Show comment
Hide comment
@josharian

josharian Apr 27, 2018

Contributor

When I looked a few weeks ago, all of the increase (and some) was dwarf. I think the best fix is probably #11799.

Contributor

josharian commented Apr 27, 2018

When I looked a few weeks ago, all of the increase (and some) was dwarf. I think the best fix is probably #11799.

@gopherbot

This comment has been minimized.

Show comment
Hide comment
@gopherbot

gopherbot Jun 15, 2018

Change https://golang.org/cl/118276 mentions this issue: cmd/link: compress DWARF sections in ELF binaries

Change https://golang.org/cl/118276 mentions this issue: cmd/link: compress DWARF sections in ELF binaries

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.