New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/compile: significant binary size increase in go1.11 #27266

Open
warmchang opened this Issue Aug 27, 2018 · 12 comments

Comments

Projects
None yet
8 participants
@warmchang

warmchang commented Aug 27, 2018

What version of Go are you using (go version)?

 ⚡ root@k8s  /home/ubuntu  go version
go version go1.11 linux/amd64

Does this issue reproduce with the latest release?

Yep, upgrade the golang to version 1.11 and build the software.

What operating system and processor architecture are you using (go env)?

 ⚡ root@k8s  /home/ubuntu  go env
GOARCH="amd64"
GOBIN="/usr/local/go/bin"
GOCACHE="/root/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/paasdata/gopath"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build147710718=/tmp/go-build -gno-record-gcc-switches"

What did you do?

  1. For example, let's build hyperkube (kubernetes v1.11.1) with go1.10.3
 ⚡ root@k8s  /paasdata/gopath/src/k8s.io/kubernetes-1.11.1  go version
go version go1.10.3 linux/amd64
 ⚡ root@k8s  /paasdata/gopath/src/k8s.io/kubernetes-1.11.1  make hyperkube
+++ [0827 14:21:57] Building go targets for linux/amd64:
    ./vendor/k8s.io/code-generator/cmd/deepcopy-gen
+++ [0827 14:22:08] Building go targets for linux/amd64:
    ./vendor/k8s.io/code-generator/cmd/defaulter-gen
+++ [0827 14:22:16] Building go targets for linux/amd64:
    ./vendor/k8s.io/code-generator/cmd/conversion-gen
+++ [0827 14:22:24] Building go targets for linux/amd64:
    ./vendor/k8s.io/code-generator/cmd/openapi-gen
+++ [0827 14:22:35] Building go targets for linux/amd64:
    ./vendor/github.com/jteeuwen/go-bindata/go-bindata
+++ [0827 14:22:37] Building go targets for linux/amd64:
    cmd/hyperkube
 ⚡ root@k8s  /paasdata/gopath/src/k8s.io/kubernetes-1.11.1 
  1. Check the generated binary files, the hyperkube is 217M
 ⚡ root@k8s  /paasdata/gopath/src/k8s.io/kubernetes-1.11.1  ll -h _output/local/bin/linux/amd64
total 255M
-rwxr-xr-x 1 root root 7.4M Aug 27 14:22 conversion-gen
-rwxr-xr-x 1 root root 7.4M Aug 27 14:21 deepcopy-gen
-rwxr-xr-x 1 root root 7.4M Aug 27 14:22 defaulter-gen
-rwxr-xr-x 1 root root 2.8M Aug 27 14:22 go-bindata
-rwxr-xr-x 1 root root 217M Aug 27 14:26 hyperkube
-rwxr-xr-x 1 root root  14M Aug 27 14:22 openapi-gen
 ⚡ root@k8s  /paasdata/gopath/src/k8s.io/kubernetes-1.11.1 
  1. Change the golang to 1.11, build hyperkube (kubernetes v1.11.1) again
 ⚡ root@k8s  /paasdata/gopath/src/k8s.io/kubernetes-1.11.1  make clean
+++ [0827 14:33:26] Verifying Prerequisites....
+++ [0827 14:33:26] Removing _output directory
Removing pkg/generated/openapi/zz_generated.openapi.go ..
Removing test/e2e/generated/bindata.go ..
 ⚡ root@k8s  /paasdata/gopath/src/k8s.io/kubernetes-1.11.1  rm -rf /usr/local/go
 ⚡ root@k8s  /paasdata/gopath/src/k8s.io/kubernetes-1.11.1  ln -s /usr/local/go1.10.3 /usr/local/go
 ⚡ root@k8s  /paasdata/gopath/src/k8s.io/kubernetes-1.11.1  go version
go version go1.10.3 linux/amd64
 ⚡ root@k8s  /paasdata/gopath/src/k8s.io/kubernetes-1.11.1  go version
go version go1.11 linux/amd64
 ⚡ root@k8s  /paasdata/gopath/src/k8s.io/kubernetes-1.11.1  make hyperkube
+++ [0827 14:33:53] Building go targets for linux/amd64:
    ./vendor/k8s.io/code-generator/cmd/deepcopy-gen
+++ [0827 14:34:05] Building go targets for linux/amd64:
    ./vendor/k8s.io/code-generator/cmd/defaulter-gen
+++ [0827 14:34:12] Building go targets for linux/amd64:
    ./vendor/k8s.io/code-generator/cmd/conversion-gen
+++ [0827 14:34:21] Building go targets for linux/amd64:
    ./vendor/k8s.io/code-generator/cmd/openapi-gen
+++ [0827 14:34:32] Building go targets for linux/amd64:
    ./vendor/github.com/jteeuwen/go-bindata/go-bindata
+++ [0827 14:34:34] Building go targets for linux/amd64:
    cmd/hyperkube
 ⚡ root@k8s  /paasdata/gopath/src/k8s.io/kubernetes-1.11.1 
  1. Check the new files, the hyperkube increased to 275M, the size increased 26.7% compared to versions 1.10.3
 ⚡ root@k8s  /paasdata/gopath/src/k8s.io/kubernetes-1.11.1  ll -h _output/local/bin/linux/amd64
total 313M
-rwxr-xr-x 1 root root 7.4M Aug 27 14:34 conversion-gen
-rwxr-xr-x 1 root root 7.4M Aug 27 14:33 deepcopy-gen
-rwxr-xr-x 1 root root 7.3M Aug 27 14:34 defaulter-gen
-rwxr-xr-x 1 root root 2.7M Aug 27 14:34 go-bindata
-rwxr-xr-x 1 root root 275M Aug 27 14:37 hyperkube
-rwxr-xr-x 1 root root  13M Aug 27 14:34 openapi-gen
 ⚡ root@k8s  /paasdata/gopath/src/k8s.io/kubernetes-1.11.1 

What did you expect to see?

The compiler version upgrade has little effect on the size of the generated binary.

What did you see instead?

The size increased 26.7% compared to 1.10.x versions.

@warmchang

This comment has been minimized.

warmchang commented Aug 27, 2018

And go1.11 consumes more resources, there are some issue (#26186 & #26132 ) tracker this.

Use go1.10.3 to compile normally, but switching to go1.11 will give an error.

 ⚡ root@k8s  /paasdata/gopath/src/k8s.io/kubernetes-1.11.1  free -h
              total        used        free      shared  buff/cache   available
Mem:           7.6G        966M        5.8G        5.7M        922M        6.4G
Swap:          1.0G        444M        579M
 ⚡ root@k8s  /paasdata/gopath/src/k8s.io/kubernetes-1.11.1  make all
+++ [0827 14:50:27] Building go targets for linux/amd64:
    ./vendor/k8s.io/code-generator/cmd/deepcopy-gen
+++ [0827 14:50:37] Building go targets for linux/amd64:
    ./vendor/k8s.io/code-generator/cmd/defaulter-gen
+++ [0827 14:50:44] Building go targets for linux/amd64:
    ./vendor/k8s.io/code-generator/cmd/conversion-gen
+++ [0827 14:50:53] Building go targets for linux/amd64:
    ./vendor/k8s.io/code-generator/cmd/openapi-gen
+++ [0827 14:51:03] Building go targets for linux/amd64:
    ./vendor/github.com/jteeuwen/go-bindata/go-bindata
+++ [0827 14:51:05] Building go targets for linux/amd64:
    cmd/kube-proxy
    cmd/kube-apiserver
    cmd/kube-controller-manager
    cmd/cloud-controller-manager
    cmd/kubelet
    cmd/kubeadm
    cmd/hyperkube
    cmd/kube-scheduler
    vendor/k8s.io/kube-aggregator
    vendor/k8s.io/apiextensions-apiserver
    cluster/gce/gci/mounter
    cmd/kubectl
    cmd/gendocs
    cmd/genkubedocs
    cmd/genman
    cmd/genyaml
    cmd/genswaggertypedocs
    cmd/linkcheck
    vendor/github.com/onsi/ginkgo/ginkgo
    test/e2e/e2e.test
    cmd/kubemark
    vendor/github.com/onsi/ginkgo/ginkgo
    test/e2e_node/e2e_node.test
/usr/local/go/pkg/tool/linux_amd64/link: signal: killed
# k8s.io/kubernetes/cmd/hyperkube
/usr/local/go/pkg/tool/linux_amd64/link: running gcc failed: signal: killed

!!! [0827 14:56:34] Call tree:
!!! [0827 14:56:34]  1: /paasdata/gopath/src/k8s.io/kubernetes-1.11.1/hack/lib/golang.sh:610 kube::golang::build_binaries_for_platform(...)
!!! [0827 14:56:34]  2: hack/make-rules/build.sh:27 kube::golang::build_binaries(...)
!!! [0827 14:56:34] Call tree:
!!! [0827 14:56:34]  1: hack/make-rules/build.sh:27 kube::golang::build_binaries(...)
!!! [0827 14:56:34] Call tree:
!!! [0827 14:56:34]  1: hack/make-rules/build.sh:27 kube::golang::build_binaries(...)
make: *** [all] Error 1
 ✘ ⚡ root@k8s  /paasdata/gopath/src/k8s.io/kubernetes-1.11.1 

@josharian josharian changed the title from The binaries generated by go1.11 are much larger than those compiled from previous versions (1.10 or 1.9) to cmd/compile: significant binary size increase in go1.11 Aug 27, 2018

@josharian

This comment has been minimized.

Contributor

josharian commented Aug 27, 2018

There's an umbrella bug for binary size: #6853. I'm going to leave this one open now, though, to make sure we understand this specific regression. In particular, is it due to DWARF size increases (#26074)? Can you run sizecmp on your binaries? Thanks.

@josharian josharian added this to the Go1.12 milestone Aug 27, 2018

@warmchang

This comment has been minimized.

warmchang commented Aug 28, 2018

Hi @josharian the sizecmp dependon otool, which is the tools for OSX, and can't run on my linux env:

 ⚡ root@k8s  /paasdata/gopath/src/k8s.io/kubernetes-1.11.1  ./sizecmp hyperkube-go1.11 hyperkube-go1.10.3
2018/08/28 10:21:39 otool -l hyperkube-go1.11: exec: "otool": executable file not found in $PATH

Use bloaty to analyse the binaries:

@warmchang

This comment has been minimized.

warmchang commented Aug 28, 2018

The size increase is not just DWARF information, I rebuild the file with flag GOLDFLAGS="-s -w" (trip DWARF, debuginfo, symbol).

As shown in the following data, the size increased about 16%.

 ⚡ root@k8s  /paasdata/gopath/src/k8s.io/kubernetes-1.11.1  ll -h ./hyperkube-go1.1*-w
-rwxr-xr-x 1 root root 132M Aug 28 10:38 ./hyperkube-go1.10.3-s-w
-rwxr-xr-x 1 root root 153M Aug 28 10:32 ./hyperkube-go1.11-s-w
 ⚡ root@k8s  /paasdata/gopath/src/k8s.io/kubernetes-1.11.1 
@mvrhov

This comment has been minimized.

mvrhov commented Sep 18, 2018

Our internal project also compiled with -s -w: (The debug binaries are a tad bit smaller in 1.11 tan those from 1.10, but that's due to DWARF compression) 1.11 also ~14% larger images

  • 1.10 13.607.104 bytes
  • 1.11 15.875.680 bytes
@networkimprov

This comment has been minimized.

networkimprov commented Sep 22, 2018

From #27784 (comment):

"bloaty suggests the extra size is coming from __gopclntab"

@warmchang

This comment has been minimized.

warmchang commented Sep 25, 2018

The bloaty analyse result by "make all WHAT=cmd/hyperkube":


 ⚡ root@k8s  /paasdata/gopath/src/k8s.io/kubernetes-1.11.1  bloaty ./hyperkube-go1.10.3
     VM SIZE                       FILE SIZE
 --------------                 --------------
  41.0%  53.9Mi .text            53.9Mi  24.8%
   0.0%       0 .debug_info      41.3Mi  19.1%
  30.5%  40.1Mi .rodata          40.1Mi  18.5%
  25.5%  33.5Mi .gopclntab       33.5Mi  15.4%
   0.0%       0 .strtab          15.1Mi   7.0%
   0.0%       0 .debug_pubtypes  8.65Mi   4.0%
   0.0%       0 .debug_frame     6.26Mi   2.9%
   0.0%       0 .debug_line      4.95Mi   2.3%
   0.0%       0 .symtab          4.29Mi   2.0%
   2.6%  3.40Mi .noptrdata       3.40Mi   1.6%
   0.0%       0 .debug_pubnames  2.99Mi   1.4%
   0.0%       0 .debug_ranges    1.92Mi   0.9%
   0.2%   253Ki .typelink         253Ki   0.1%
   0.1%   186Ki .bss                  0   0.0%
   0.1%   139Ki .data             139Ki   0.1%
   0.0%  47.2Ki .itablink        47.2Ki   0.0%
   0.0%  16.4Ki [31 Others]      22.5Ki   0.0%
   0.0%       0 .debug_loc       19.0Ki   0.0%
   0.0%  17.5Ki .noptrbss             0   0.0%
   0.0%       0 .debug_abbrev    9.21Ki   0.0%
   0.0%       0 .debug_str       6.61Ki   0.0%
 100.0%   131Mi TOTAL             216Mi 100.0%
 ⚡ root@k8s  /paasdata/gopath/src/k8s.io/kubernetes-1.11.1  bloaty ./hyperkube-go1.11
     VM SIZE                       FILE SIZE
 --------------                 --------------
  35.8%  54.6Mi .text            54.6Mi  19.9%
  30.8%  47.0Mi .gopclntab       47.0Mi  17.1%
  30.7%  46.9Mi .rodata          46.9Mi  17.1%
   0.0%       0 .debug_info      42.3Mi  15.4%
   0.0%       0 .debug_loc       25.7Mi   9.4%
   0.0%       0 .strtab          15.1Mi   5.5%
   0.0%       0 .debug_line      8.95Mi   3.3%
   0.0%       0 .debug_pubtypes  8.47Mi   3.1%
   0.0%       0 .debug_ranges    8.19Mi   3.0%
   0.0%       0 .debug_frame     6.30Mi   2.3%
   0.0%       0 .symtab          4.31Mi   1.6%
   2.2%  3.40Mi .noptrdata       3.40Mi   1.2%
   0.0%       0 .debug_pubnames  3.04Mi   1.1%
   0.2%   251Ki .typelink         251Ki   0.1%
   0.1%   186Ki .bss                  0   0.0%
   0.1%   139Ki .data             139Ki   0.0%
   0.0%  47.2Ki .itablink        47.2Ki   0.0%
   0.0%  16.4Ki [31 Others]      22.3Ki   0.0%
   0.0%  17.9Ki .noptrbss             0   0.0%
   0.0%       0 .debug_abbrev    9.13Ki   0.0%
   0.0%       0 .debug_str       6.63Ki   0.0%
 100.0%   152Mi TOTAL             274Mi 100.0%
 ⚡ root@k8s  /paasdata/gopath/src/k8s.io/kubernetes-1.11.1 

@warmchang

This comment has been minimized.

warmchang commented Sep 25, 2018

The bloaty analyse result by 'make all WHAT=cmd/hyperkube GOLDFLAGS="-s -w"' (trip DWARF, debuginfo, symbol):


 ⚡ root@k8s  /paasdata/gopath/src/k8s.io/kubernetes-1.11.1  bloaty ./hyperkube-go1.10.3-s-w
     VM SIZE                     FILE SIZE
 --------------               --------------
  41.0%  53.9Mi .text          53.9Mi  41.0%
  30.5%  40.1Mi .rodata        40.1Mi  30.5%
  25.5%  33.5Mi .gopclntab     33.5Mi  25.5%
   2.6%  3.40Mi .noptrdata     3.40Mi   2.6%
   0.2%   253Ki .typelink       253Ki   0.2%
   0.1%   186Ki .bss                0   0.0%
   0.1%   139Ki .data           139Ki   0.1%
   0.0%  47.2Ki .itablink      47.2Ki   0.0%
   0.0%  17.5Ki .noptrbss           0   0.0%
   0.0%  3.91Ki .dynsym        3.91Ki   0.0%
   0.0%  3.57Ki .eh_frame      3.57Ki   0.0%
   0.0%  3.05Ki .dynstr        3.05Ki   0.0%
   0.0%     624 [ELF Headers]  2.80Ki   0.0%
   0.0%     866 [18 Others]    1.26Ki   0.0%
   0.0%  1.12Ki .rela.plt      1.12Ki   0.0%
   0.0%     988 .gnu.hash         988   0.0%
   0.0%       0 [Unmapped]        983   0.0%
   0.0%     784 .plt              784   0.0%
   0.0%     700 .eh_frame_hdr     700   0.0%
   0.0%     496 .dynamic          496   0.0%
   0.0%     408 .got.plt          408   0.0%
 100.0%   131Mi TOTAL           131Mi 100.0%
 ⚡ root@k8s  /paasdata/gopath/src/k8s.io/kubernetes-1.11.1  bloaty ./hyperkube-go1.11-s-w
     VM SIZE                     FILE SIZE
 --------------               --------------
  35.8%  54.6Mi .text          54.6Mi  35.8%
  30.8%  47.0Mi .gopclntab     47.0Mi  30.8%
  30.7%  46.9Mi .rodata        46.9Mi  30.8%
   2.2%  3.40Mi .noptrdata     3.40Mi   2.2%
   0.2%   251Ki .typelink       251Ki   0.2%
   0.1%   186Ki .bss                0   0.0%
   0.1%   139Ki .data           139Ki   0.1%
   0.0%  47.2Ki .itablink      47.2Ki   0.0%
   0.0%  17.9Ki .noptrbss           0   0.0%
   0.0%  3.91Ki .dynsym        3.91Ki   0.0%
   0.0%  3.57Ki .eh_frame      3.57Ki   0.0%
   0.0%  3.05Ki .dynstr        3.05Ki   0.0%
   0.0%     624 [ELF Headers]  2.80Ki   0.0%
   0.0%     875 [18 Others]    1.27Ki   0.0%
   0.0%  1.12Ki .rela.plt      1.12Ki   0.0%
   0.0%     988 .gnu.hash         988   0.0%
   0.0%     784 .plt              784   0.0%
   0.0%       0 [Unmapped]        767   0.0%
   0.0%     700 .eh_frame_hdr     700   0.0%
   0.0%     496 .dynamic          496   0.0%
   0.0%     408 .got.plt          408   0.0%
 100.0%   152Mi TOTAL           152Mi 100.0%
 ⚡ root@k8s  /paasdata/gopath/src/k8s.io/kubernetes-1.11.1 

@odeke-em

This comment has been minimized.

Member

odeke-em commented Oct 18, 2018

I'll also kindly rope in some more folks who might be interested in this bug @randall77 @ianlancetaylor @bradfitz @cherrymui @heschik

@bradfitz

This comment has been minimized.

Member

bradfitz commented Oct 18, 2018

@heschik

This comment has been minimized.

Contributor

heschik commented Oct 18, 2018

I haven't looked into this deeply at all, but growth in .gopclntab would seem to implicate https://golang.org/issue/24543 (safepoints everywhere). The design doc there says to expect growth of ~10% unless optimizations are applied.

@aclements

This comment has been minimized.

Member

aclements commented Oct 23, 2018

The growth in .gopclntab is almost certainly mostly from safepoints everywhere, but that's only 13.5MB of growth, or 6.25% of overall growth in the unstripped binary and 10% growth in the stripped binary.

Here's a section-by-section comparison from benchstat of the bloaty output:

For unstripped binaries:

name             old MB     new MB     delta
.debug_pubtypes  8.65 ± 0%  8.47 ± 0%    -2.08%
.noptrdata       3.40 ± 0%  3.40 ± 0%    +0.00%
.strtab          15.1 ± 0%  15.1 ± 0%    +0.00%
.symtab          4.29 ± 0%  4.31 ± 0%    +0.47%
.debug_frame     6.26 ± 0%  6.30 ± 0%    +0.64%
.text            53.9 ± 0%  54.6 ± 0%    +1.30%
.debug_pubnames  2.99 ± 0%  3.04 ± 0%    +1.67%
.debug_info      41.3 ± 0%  42.3 ± 0%    +2.42%
.rodata          40.1 ± 0%  46.9 ± 0%   +16.96%
TOTAL             216 ± 0%   274 ± 0%   +26.85%
.gopclntab       33.5 ± 0%  47.0 ± 0%   +40.30%
.debug_line      4.95 ± 0%  8.95 ± 0%   +80.81%
.debug_ranges    1.92 ± 0%  8.19 ± 0%  +326.56%

For .gopclntab, it would be reasonable to strip the safepoints everywhere data in -dwarf=off mode. That's only used for debugger call injection right now, which isn't much use without DWARF data.

@rdsubhas rdsubhas referenced this issue Nov 8, 2018

Merged

Use golang 1.11 with modules #109

11 of 11 tasks complete
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment