crypto/x509: certificate validation errors with go1.7.4 binaries + macOS Sierra

[cblecker]

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version go1.7.4 darwin/amd64

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/opt/gopath"
GORACE=""
GOROOT="/usr/local/opt/go/libexec"
GOTOOLDIR="/usr/local/opt/go/libexec/pkg/tool/darwin_amd64"
CC="clang"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/lw/b5k2lvx53j98z_c0z8cyk_bcgjcp8_/T/go-build124753553=/tmp/go-build -gno-record-gcc-switches -fno-common"
CXX="clang++"
CGO_ENABLED="1"

What did you do?

On Jan 11, the Google Cloud SDK team included a new version of kubectl with the google-cloud-sdk. This new version is compiled with go1.7.4, rather than the previous release that was compiled with go1.7.1.

This combination of kubectl compiled under go1.7.4 + macOS Sierra seems to hit #18203, causing the certificates to report as invalid. I've seen mentions of it helm/helm#1749 and https://kubernetes.slack.com/archives/kubernetes-users/p1484587693022364, as well as encountered it myself. These issues seem to go away when using the version of kubectl compiled with Homebrew that is currently patching in the fix (Homebrew/homebrew-core#8628).

I'm not sure if there is a plan to release another 1.7.x go version before 1.8 drops next month, but I wanted to let you know that this bug is impacting users out in the wild.

What did you expect to see?

Darwin binaries compiled with go1.7.4 and run on macOS Sierra seem to hit the bug described in #18203. This causes problems when validating TLS certificates.

What did you see instead?

kubectl (go binary to control Kubernetes clusters from the command line) specificly reports Unable to connect to the server: Post https://accounts.google.com/o/oauth2/token: x509: certificate signed by unknown authority. This error is eliminated when using a version of kubectl that was compiled with 3357daa patched in.

References

#18141
#18203
helm/helm#1749
Homebrew/homebrew-core#8628

[bradfitz]

I think your bug report is misleading. You say:

CGO_ENABLED="1"

But from what I understand, this bug is specifically about CGO_ENABLED=0 binaries.

Can you confirm that the kubectl binary in question was built without cgo support?

[cblecker]

My local go environment is CGO_ENABLED, but the binary in question is pre-compiled and distributed by the Google Cloud SDK team. I'm not sure if there is a way to tell from the binary itself if CGO was used in it's compilation.

[bradfitz]

Use go tool nm on the kubectl binary. If it was built without cgo, you'll see the execSecurityRoots symbol in the binary:

$ go tool nm hello.cgo  | grep execSecurityRoots
$ go tool nm hello.cgo0  | grep execSecurityRoots
   56330 T crypto/x509.execSecurityRoots

[cblecker]

That was easy!

$ google-cloud-sdk/bin/kubectl version --client
Client Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.1", GitCommit:"82450d03cb057bab0950214ef122b67c83fb11df", GitTreeState:"clean", BuildDate:"2016-12-14T00:57:05Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"darwin/amd64"}
$ go tool nm google-cloud-sdk/bin/kubectl | grep execSecurityRoots
  dde4a0 T crypto/x509.execSecurityRoots
  dea6e0 T crypto/x509.execSecurityRoots.func1
  dea790 T crypto/x509.execSecurityRoots.func2
 1473ed0 R crypto/x509.execSecurityRoots.func2.f

[cblecker]

This is odd.. with my local go environment, I recompiled kubectl from source, and I'm still getting execSecurityRoots in go tool nm:

$ make generated_files 
+++ [0117 11:18:00] Building the toolchain targets:
    k8s.io/kubernetes/hack/cmd/teststale
    k8s.io/kubernetes/vendor/github.com/jteeuwen/go-bindata/go-bindata
+++ [0117 11:18:01] Generating bindata:
    test/e2e/framework/gobindata_util.go
+++ [0117 11:18:02] Building go targets for darwin/amd64:
    cmd/libs/go2idl/deepcopy-gen
+++ [0117 11:18:08] Building the toolchain targets:
    k8s.io/kubernetes/hack/cmd/teststale
    k8s.io/kubernetes/vendor/github.com/jteeuwen/go-bindata/go-bindata
+++ [0117 11:18:08] Generating bindata:
    test/e2e/framework/gobindata_util.go
+++ [0117 11:18:08] Building go targets for darwin/amd64:
    cmd/libs/go2idl/defaulter-gen
+++ [0117 11:18:13] Building the toolchain targets:
    k8s.io/kubernetes/hack/cmd/teststale
    k8s.io/kubernetes/vendor/github.com/jteeuwen/go-bindata/go-bindata
+++ [0117 11:18:13] Generating bindata:
    test/e2e/framework/gobindata_util.go
+++ [0117 11:18:14] Building go targets for darwin/amd64:
    cmd/libs/go2idl/conversion-gen
+++ [0117 11:18:20] Building the toolchain targets:
    k8s.io/kubernetes/hack/cmd/teststale
    k8s.io/kubernetes/vendor/github.com/jteeuwen/go-bindata/go-bindata
+++ [0117 11:18:20] Generating bindata:
    test/e2e/framework/gobindata_util.go
+++ [0117 11:18:21] Building go targets for darwin/amd64:
    cmd/libs/go2idl/openapi-gen
$ make kubectl
+++ [0117 11:19:01] Building the toolchain targets:
    k8s.io/kubernetes/hack/cmd/teststale
    k8s.io/kubernetes/vendor/github.com/jteeuwen/go-bindata/go-bindata
+++ [0117 11:19:01] Generating bindata:
    test/e2e/framework/gobindata_util.go
+++ [0117 11:19:01] Building go targets for darwin/amd64:
    cmd/kubectl
$ _output/bin/kubectl version --client
Client Version: version.Info{Major:"1", Minor:"5+", GitVersion:"v1.5.1-dirty", GitCommit:"82450d03cb057bab0950214ef122b67c83fb11df", GitTreeState:"dirty", BuildDate:"2017-01-17T19:19:01Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"darwin/amd64"}
$ go tool nm  _output/bin/kubectl | grep execSecurityRoots
  dde4a0 T crypto/x509.execSecurityRoots
  deafc0 T crypto/x509.execSecurityRoots.func1
 14753a8 R crypto/x509.execSecurityRoots.func1.f
$ go env
GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/opt/gopath"
GORACE=""
GOROOT="/usr/local/opt/go/libexec"
GOTOOLDIR="/usr/local/opt/go/libexec/pkg/tool/darwin_amd64"
CC="clang"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/lw/b5k2lvx53j98z_c0z8cyk_bcgjcp8_/T/go-build362149565=/tmp/go-build -gno-record-gcc-switches -fno-common"
CXX="clang++"
CGO_ENABLED="1"
$ 

[bradfitz]

@cblecker, but what is the Makefile doing? It's probably disabling cgo.

[cblecker]

Yeah, had to dig in a bunch of layers, but kubectl is handled as a static binary in the kubernetes project, and is compiled with CGO_ENABLED=0 here:
https://github.com/kubernetes/kubernetes/blob/master/hack/lib/golang.sh#L534

[titanous]

I have also run into the same problem with CGO_ENABLED=0 and my own (non-Kubernetes) binaries with the same Go/macOS versions.

[gopherbot]

CL https://golang.org/cl/35634 mentions this issue.

[bradfitz]

Submitted to 1.7 branch.