cmd/compile: significant binary size increase in go1.11

[warmchang]

What version of Go are you using (go version)?

 ⚡ root@k8s  /home/ubuntu  go version
go version go1.11 linux/amd64

Does this issue reproduce with the latest release?

Yep, upgrade the golang to version 1.11 and build the software.

What operating system and processor architecture are you using (go env)?

 ⚡ root@k8s  /home/ubuntu  go env
GOARCH="amd64"
GOBIN="/usr/local/go/bin"
GOCACHE="/root/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/paasdata/gopath"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build147710718=/tmp/go-build -gno-record-gcc-switches"

What did you do?

1. For example, let's build hyperkube (kubernetes v1.11.1) with go1.10.3

 ⚡ root@k8s  /paasdata/gopath/src/k8s.io/kubernetes-1.11.1  go version
go version go1.10.3 linux/amd64
 ⚡ root@k8s  /paasdata/gopath/src/k8s.io/kubernetes-1.11.1  make hyperkube
+++ [0827 14:21:57] Building go targets for linux/amd64:
    ./vendor/k8s.io/code-generator/cmd/deepcopy-gen
+++ [0827 14:22:08] Building go targets for linux/amd64:
    ./vendor/k8s.io/code-generator/cmd/defaulter-gen
+++ [0827 14:22:16] Building go targets for linux/amd64:
    ./vendor/k8s.io/code-generator/cmd/conversion-gen
+++ [0827 14:22:24] Building go targets for linux/amd64:
    ./vendor/k8s.io/code-generator/cmd/openapi-gen
+++ [0827 14:22:35] Building go targets for linux/amd64:
    ./vendor/github.com/jteeuwen/go-bindata/go-bindata
+++ [0827 14:22:37] Building go targets for linux/amd64:
    cmd/hyperkube
 ⚡ root@k8s  /paasdata/gopath/src/k8s.io/kubernetes-1.11.1 

2. Check the generated binary files, the hyperkube is 217M

 ⚡ root@k8s  /paasdata/gopath/src/k8s.io/kubernetes-1.11.1  ll -h _output/local/bin/linux/amd64
total 255M
-rwxr-xr-x 1 root root 7.4M Aug 27 14:22 conversion-gen
-rwxr-xr-x 1 root root 7.4M Aug 27 14:21 deepcopy-gen
-rwxr-xr-x 1 root root 7.4M Aug 27 14:22 defaulter-gen
-rwxr-xr-x 1 root root 2.8M Aug 27 14:22 go-bindata
-rwxr-xr-x 1 root root 217M Aug 27 14:26 hyperkube
-rwxr-xr-x 1 root root  14M Aug 27 14:22 openapi-gen
 ⚡ root@k8s  /paasdata/gopath/src/k8s.io/kubernetes-1.11.1 

3. Change the golang to 1.11, build hyperkube (kubernetes v1.11.1) again

 ⚡ root@k8s  /paasdata/gopath/src/k8s.io/kubernetes-1.11.1  make clean
+++ [0827 14:33:26] Verifying Prerequisites....
+++ [0827 14:33:26] Removing _output directory
Removing pkg/generated/openapi/zz_generated.openapi.go ..
Removing test/e2e/generated/bindata.go ..
 ⚡ root@k8s  /paasdata/gopath/src/k8s.io/kubernetes-1.11.1  rm -rf /usr/local/go
 ⚡ root@k8s  /paasdata/gopath/src/k8s.io/kubernetes-1.11.1  ln -s /usr/local/go1.10.3 /usr/local/go
 ⚡ root@k8s  /paasdata/gopath/src/k8s.io/kubernetes-1.11.1  go version
go version go1.10.3 linux/amd64
 ⚡ root@k8s  /paasdata/gopath/src/k8s.io/kubernetes-1.11.1  go version
go version go1.11 linux/amd64
 ⚡ root@k8s  /paasdata/gopath/src/k8s.io/kubernetes-1.11.1  make hyperkube
+++ [0827 14:33:53] Building go targets for linux/amd64:
    ./vendor/k8s.io/code-generator/cmd/deepcopy-gen
+++ [0827 14:34:05] Building go targets for linux/amd64:
    ./vendor/k8s.io/code-generator/cmd/defaulter-gen
+++ [0827 14:34:12] Building go targets for linux/amd64:
    ./vendor/k8s.io/code-generator/cmd/conversion-gen
+++ [0827 14:34:21] Building go targets for linux/amd64:
    ./vendor/k8s.io/code-generator/cmd/openapi-gen
+++ [0827 14:34:32] Building go targets for linux/amd64:
    ./vendor/github.com/jteeuwen/go-bindata/go-bindata
+++ [0827 14:34:34] Building go targets for linux/amd64:
    cmd/hyperkube
 ⚡ root@k8s  /paasdata/gopath/src/k8s.io/kubernetes-1.11.1 

4. Check the new files, the hyperkube increased to 275M, the size increased 26.7% compared to versions 1.10.3

 ⚡ root@k8s  /paasdata/gopath/src/k8s.io/kubernetes-1.11.1  ll -h _output/local/bin/linux/amd64
total 313M
-rwxr-xr-x 1 root root 7.4M Aug 27 14:34 conversion-gen
-rwxr-xr-x 1 root root 7.4M Aug 27 14:33 deepcopy-gen
-rwxr-xr-x 1 root root 7.3M Aug 27 14:34 defaulter-gen
-rwxr-xr-x 1 root root 2.7M Aug 27 14:34 go-bindata
-rwxr-xr-x 1 root root 275M Aug 27 14:37 hyperkube
-rwxr-xr-x 1 root root  13M Aug 27 14:34 openapi-gen
 ⚡ root@k8s  /paasdata/gopath/src/k8s.io/kubernetes-1.11.1 

What did you expect to see?

The compiler version upgrade has little effect on the size of the generated binary.

What did you see instead?

The size increased 26.7% compared to 1.10.x versions.

[warmchang]

And go1.11 consumes more resources, there are some issue (#26186 & #26132 ) tracker this.

Use go1.10.3 to compile normally, but switching to go1.11 will give an error.

 ⚡ root@k8s  /paasdata/gopath/src/k8s.io/kubernetes-1.11.1  free -h
              total        used        free      shared  buff/cache   available
Mem:           7.6G        966M        5.8G        5.7M        922M        6.4G
Swap:          1.0G        444M        579M
 ⚡ root@k8s  /paasdata/gopath/src/k8s.io/kubernetes-1.11.1  make all
+++ [0827 14:50:27] Building go targets for linux/amd64:
    ./vendor/k8s.io/code-generator/cmd/deepcopy-gen
+++ [0827 14:50:37] Building go targets for linux/amd64:
    ./vendor/k8s.io/code-generator/cmd/defaulter-gen
+++ [0827 14:50:44] Building go targets for linux/amd64:
    ./vendor/k8s.io/code-generator/cmd/conversion-gen
+++ [0827 14:50:53] Building go targets for linux/amd64:
    ./vendor/k8s.io/code-generator/cmd/openapi-gen
+++ [0827 14:51:03] Building go targets for linux/amd64:
    ./vendor/github.com/jteeuwen/go-bindata/go-bindata
+++ [0827 14:51:05] Building go targets for linux/amd64:
    cmd/kube-proxy
    cmd/kube-apiserver
    cmd/kube-controller-manager
    cmd/cloud-controller-manager
    cmd/kubelet
    cmd/kubeadm
    cmd/hyperkube
    cmd/kube-scheduler
    vendor/k8s.io/kube-aggregator
    vendor/k8s.io/apiextensions-apiserver
    cluster/gce/gci/mounter
    cmd/kubectl
    cmd/gendocs
    cmd/genkubedocs
    cmd/genman
    cmd/genyaml
    cmd/genswaggertypedocs
    cmd/linkcheck
    vendor/github.com/onsi/ginkgo/ginkgo
    test/e2e/e2e.test
    cmd/kubemark
    vendor/github.com/onsi/ginkgo/ginkgo
    test/e2e_node/e2e_node.test
/usr/local/go/pkg/tool/linux_amd64/link: signal: killed
# k8s.io/kubernetes/cmd/hyperkube
/usr/local/go/pkg/tool/linux_amd64/link: running gcc failed: signal: killed

!!! [0827 14:56:34] Call tree:
!!! [0827 14:56:34]  1: /paasdata/gopath/src/k8s.io/kubernetes-1.11.1/hack/lib/golang.sh:610 kube::golang::build_binaries_for_platform(...)
!!! [0827 14:56:34]  2: hack/make-rules/build.sh:27 kube::golang::build_binaries(...)
!!! [0827 14:56:34] Call tree:
!!! [0827 14:56:34]  1: hack/make-rules/build.sh:27 kube::golang::build_binaries(...)
!!! [0827 14:56:34] Call tree:
!!! [0827 14:56:34]  1: hack/make-rules/build.sh:27 kube::golang::build_binaries(...)
make: *** [all] Error 1
 ✘ ⚡ root@k8s  /paasdata/gopath/src/k8s.io/kubernetes-1.11.1 

[josharian]

There's an umbrella bug for binary size: #6853. I'm going to leave this one open now, though, to make sure we understand this specific regression. In particular, is it due to DWARF size increases (#26074)? Can you run sizecmp on your binaries? Thanks.

[warmchang]

Hi @josharian the sizecmp dependon otool, which is the tools for OSX, and can't run on my linux env:

 ⚡ root@k8s  /paasdata/gopath/src/k8s.io/kubernetes-1.11.1  ./sizecmp hyperkube-go1.11 hyperkube-go1.10.3
2018/08/28 10:21:39 otool -l hyperkube-go1.11: exec: "otool": executable file not found in $PATH

Use bloaty to analyse the binaries:

• Normal build: cmd/compile: significant binary size increase in go1.11 #27266 (comment)
• Trip build: cmd/compile: significant binary size increase in go1.11 #27266 (comment)

[warmchang]

The size increase is not just DWARF information, I rebuild the file with flag GOLDFLAGS="-s -w" (trip DWARF, debuginfo, symbol).

As shown in the following data, the size increased about 16%.

 ⚡ root@k8s  /paasdata/gopath/src/k8s.io/kubernetes-1.11.1  ll -h ./hyperkube-go1.1*-w
-rwxr-xr-x 1 root root 132M Aug 28 10:38 ./hyperkube-go1.10.3-s-w
-rwxr-xr-x 1 root root 153M Aug 28 10:32 ./hyperkube-go1.11-s-w
 ⚡ root@k8s  /paasdata/gopath/src/k8s.io/kubernetes-1.11.1 

[mvrhov]

Our internal project also compiled with -s -w: (The debug binaries are a tad bit smaller in 1.11 tan those from 1.10, but that's due to DWARF compression) 1.11 also ~14% larger images

• 1.10 13.607.104 bytes
• 1.11 15.875.680 bytes

[networkimprov]

From #27784 (comment):

"bloaty suggests the extra size is coming from __gopclntab"

[warmchang]

The bloaty analyse result by "make all WHAT=cmd/hyperkube":

 ⚡ root@k8s  /paasdata/gopath/src/k8s.io/kubernetes-1.11.1  bloaty ./hyperkube-go1.10.3
     VM SIZE                       FILE SIZE
 --------------                 --------------
  41.0%  53.9Mi .text            53.9Mi  24.8%
   0.0%       0 .debug_info      41.3Mi  19.1%
  30.5%  40.1Mi .rodata          40.1Mi  18.5%
  25.5%  33.5Mi .gopclntab       33.5Mi  15.4%
   0.0%       0 .strtab          15.1Mi   7.0%
   0.0%       0 .debug_pubtypes  8.65Mi   4.0%
   0.0%       0 .debug_frame     6.26Mi   2.9%
   0.0%       0 .debug_line      4.95Mi   2.3%
   0.0%       0 .symtab          4.29Mi   2.0%
   2.6%  3.40Mi .noptrdata       3.40Mi   1.6%
   0.0%       0 .debug_pubnames  2.99Mi   1.4%
   0.0%       0 .debug_ranges    1.92Mi   0.9%
   0.2%   253Ki .typelink         253Ki   0.1%
   0.1%   186Ki .bss                  0   0.0%
   0.1%   139Ki .data             139Ki   0.1%
   0.0%  47.2Ki .itablink        47.2Ki   0.0%
   0.0%  16.4Ki [31 Others]      22.5Ki   0.0%
   0.0%       0 .debug_loc       19.0Ki   0.0%
   0.0%  17.5Ki .noptrbss             0   0.0%
   0.0%       0 .debug_abbrev    9.21Ki   0.0%
   0.0%       0 .debug_str       6.61Ki   0.0%
 100.0%   131Mi TOTAL             216Mi 100.0%
 ⚡ root@k8s  /paasdata/gopath/src/k8s.io/kubernetes-1.11.1  bloaty ./hyperkube-go1.11
     VM SIZE                       FILE SIZE
 --------------                 --------------
  35.8%  54.6Mi .text            54.6Mi  19.9%
  30.8%  47.0Mi .gopclntab       47.0Mi  17.1%
  30.7%  46.9Mi .rodata          46.9Mi  17.1%
   0.0%       0 .debug_info      42.3Mi  15.4%
   0.0%       0 .debug_loc       25.7Mi   9.4%
   0.0%       0 .strtab          15.1Mi   5.5%
   0.0%       0 .debug_line      8.95Mi   3.3%
   0.0%       0 .debug_pubtypes  8.47Mi   3.1%
   0.0%       0 .debug_ranges    8.19Mi   3.0%
   0.0%       0 .debug_frame     6.30Mi   2.3%
   0.0%       0 .symtab          4.31Mi   1.6%
   2.2%  3.40Mi .noptrdata       3.40Mi   1.2%
   0.0%       0 .debug_pubnames  3.04Mi   1.1%
   0.2%   251Ki .typelink         251Ki   0.1%
   0.1%   186Ki .bss                  0   0.0%
   0.1%   139Ki .data             139Ki   0.0%
   0.0%  47.2Ki .itablink        47.2Ki   0.0%
   0.0%  16.4Ki [31 Others]      22.3Ki   0.0%
   0.0%  17.9Ki .noptrbss             0   0.0%
   0.0%       0 .debug_abbrev    9.13Ki   0.0%
   0.0%       0 .debug_str       6.63Ki   0.0%
 100.0%   152Mi TOTAL             274Mi 100.0%
 ⚡ root@k8s  /paasdata/gopath/src/k8s.io/kubernetes-1.11.1 

[warmchang]

The bloaty analyse result by 'make all WHAT=cmd/hyperkube GOLDFLAGS="-s -w"' (trip DWARF, debuginfo, symbol):

 ⚡ root@k8s  /paasdata/gopath/src/k8s.io/kubernetes-1.11.1  bloaty ./hyperkube-go1.10.3-s-w
     VM SIZE                     FILE SIZE
 --------------               --------------
  41.0%  53.9Mi .text          53.9Mi  41.0%
  30.5%  40.1Mi .rodata        40.1Mi  30.5%
  25.5%  33.5Mi .gopclntab     33.5Mi  25.5%
   2.6%  3.40Mi .noptrdata     3.40Mi   2.6%
   0.2%   253Ki .typelink       253Ki   0.2%
   0.1%   186Ki .bss                0   0.0%
   0.1%   139Ki .data           139Ki   0.1%
   0.0%  47.2Ki .itablink      47.2Ki   0.0%
   0.0%  17.5Ki .noptrbss           0   0.0%
   0.0%  3.91Ki .dynsym        3.91Ki   0.0%
   0.0%  3.57Ki .eh_frame      3.57Ki   0.0%
   0.0%  3.05Ki .dynstr        3.05Ki   0.0%
   0.0%     624 [ELF Headers]  2.80Ki   0.0%
   0.0%     866 [18 Others]    1.26Ki   0.0%
   0.0%  1.12Ki .rela.plt      1.12Ki   0.0%
   0.0%     988 .gnu.hash         988   0.0%
   0.0%       0 [Unmapped]        983   0.0%
   0.0%     784 .plt              784   0.0%
   0.0%     700 .eh_frame_hdr     700   0.0%
   0.0%     496 .dynamic          496   0.0%
   0.0%     408 .got.plt          408   0.0%
 100.0%   131Mi TOTAL           131Mi 100.0%
 ⚡ root@k8s  /paasdata/gopath/src/k8s.io/kubernetes-1.11.1  bloaty ./hyperkube-go1.11-s-w
     VM SIZE                     FILE SIZE
 --------------               --------------
  35.8%  54.6Mi .text          54.6Mi  35.8%
  30.8%  47.0Mi .gopclntab     47.0Mi  30.8%
  30.7%  46.9Mi .rodata        46.9Mi  30.8%
   2.2%  3.40Mi .noptrdata     3.40Mi   2.2%
   0.2%   251Ki .typelink       251Ki   0.2%
   0.1%   186Ki .bss                0   0.0%
   0.1%   139Ki .data           139Ki   0.1%
   0.0%  47.2Ki .itablink      47.2Ki   0.0%
   0.0%  17.9Ki .noptrbss           0   0.0%
   0.0%  3.91Ki .dynsym        3.91Ki   0.0%
   0.0%  3.57Ki .eh_frame      3.57Ki   0.0%
   0.0%  3.05Ki .dynstr        3.05Ki   0.0%
   0.0%     624 [ELF Headers]  2.80Ki   0.0%
   0.0%     875 [18 Others]    1.27Ki   0.0%
   0.0%  1.12Ki .rela.plt      1.12Ki   0.0%
   0.0%     988 .gnu.hash         988   0.0%
   0.0%     784 .plt              784   0.0%
   0.0%       0 [Unmapped]        767   0.0%
   0.0%     700 .eh_frame_hdr     700   0.0%
   0.0%     496 .dynamic          496   0.0%
   0.0%     408 .got.plt          408   0.0%
 100.0%   152Mi TOTAL           152Mi 100.0%
 ⚡ root@k8s  /paasdata/gopath/src/k8s.io/kubernetes-1.11.1 

[odeke-em]

I'll also kindly rope in some more folks who might be interested in this bug @randall77 @ianlancetaylor @bradfitz @cherrymui @heschik

[bradfitz]

/cc @dr2chase @aclements

[heschi]

I haven't looked into this deeply at all, but growth in .gopclntab would seem to implicate https://golang.org/issue/24543 (safepoints everywhere). The design doc there says to expect growth of ~10% unless optimizations are applied.