Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

crypto/x509: trust setting not inherited on Darwin #32891

tommyknows opened this issue Jul 2, 2019 · 8 comments

crypto/x509: trust setting not inherited on Darwin #32891

tommyknows opened this issue Jul 2, 2019 · 8 comments


Copy link

@tommyknows tommyknows commented Jul 2, 2019

What version of Go are you using (go version)?

$ go version
go version go1.12.5 darwin/amd64

But I'm trying to build it from source

Does this issue reproduce with the latest release?

Trying to build it from Master leads to same results

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/fm/tzdsx49104v20lr4m55_g43w0000gn/T/go-build735870497=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

Trying to build Go from source (tried different versions, official releases).
For that, I cloned the git repository and run the all.bash script in the src folder.

What did you expect to see?

The build succeeding

What did you see instead?

./all.bash Output

Building Go cmd/dist using /usr/local/go.
Building Go toolchain1 using /usr/local/go.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
Building Go toolchain2 using go_bootstrap and Go toolchain1.
Building Go toolchain3 using go_bootstrap and Go toolchain2.
Building packages and commands for darwin/amd64.

Testing packages.

ok archive/tar 0.093s
ok archive/zip 0.088s
ok bufio 0.113s
ok bytes 0.690s
ok compress/bzip2 0.128s
ok compress/flate 0.956s
ok compress/gzip 0.034s
ok compress/lzw 0.026s
ok compress/zlib 0.037s
ok container/heap 0.029s
ok container/list 0.024s
ok container/ring 0.028s
ok context 1.027s
ok crypto 0.017s
ok crypto/aes 0.074s
ok crypto/cipher 0.046s
ok crypto/des 0.042s
ok crypto/dsa 0.019s
ok crypto/ecdsa 0.443s
ok crypto/ed25519 0.565s
ok crypto/elliptic 0.082s
ok crypto/hmac 0.017s
ok crypto/internal/subtle 0.016s
ok crypto/md5 0.020s
ok crypto/rand 0.056s
ok crypto/rc4 0.094s
ok crypto/rsa 0.160s
ok crypto/sha1 0.060s
ok crypto/sha256 0.020s
ok crypto/sha512 0.019s
ok crypto/subtle 0.028s
ok crypto/tls 1.153s
--- FAIL: TestSystemRoots (0.79s)
root_darwin_test.go:35: cgo sys roots: 413.977623ms
root_darwin_test.go:36: non-cgo sys roots: 314.683156ms
root_darwin_test.go:77: certificate only present in non-cgo pool: CN=[redacted cert info] (verify error: x509: certificate signed by unknown authority)
root_darwin_test.go:79: signed certificate only present in non-cgo pool (acceptable): [redacted cert info]
root_darwin_test.go:77: certificate only present in non-cgo pool: CN=[redacted cert info] (verify error: x509: certificate signed by unknown authority)
root_darwin_test.go:79: signed certificate only present in non-cgo pool (acceptable): CN=Developer ID Certification Authority,OU=Apple Certification Authority,O=Apple Inc.,C=US
FAIL crypto/x509 1.394s
ok database/sql 0.587s
ok database/sql/driver 0.015s
ok debug/dwarf 0.036s
ok debug/elf 0.043s
ok debug/gosym 0.017s
ok debug/macho 0.022s
ok debug/pe 0.022s
ok debug/plan9obj 0.017s
ok encoding/ascii85 0.023s
ok encoding/asn1 0.019s
ok encoding/base32 0.031s
ok encoding/base64 0.028s
ok encoding/binary 0.016s
ok encoding/csv 0.017s
ok encoding/gob 0.039s
ok encoding/hex 0.015s
ok encoding/json 0.153s
ok encoding/pem 0.036s
ok encoding/xml 0.034s
ok errors 0.013s
ok expvar 0.028s
ok flag 0.012s
ok fmt 0.151s
ok go/ast 0.015s
ok go/build 0.419s
ok go/constant 0.013s
ok go/doc 0.070s
ok go/format 0.015s
ok go/importer 0.137s
ok go/internal/gccgoimporter 0.028s
ok go/internal/gcimporter 0.532s
ok go/internal/srcimporter 1.370s
ok go/parser 0.042s
ok go/printer 0.315s
ok go/scanner 0.013s
ok go/token 0.028s
ok go/types 1.390s
ok hash 0.013s
ok hash/adler32 0.015s
ok hash/crc32 0.022s
ok hash/crc64 0.014s
ok hash/fnv 0.011s
ok html 0.028s
ok html/template 0.049s
ok image 0.091s
ok image/color 0.027s
ok image/draw 0.069s
ok image/gif 0.359s
ok image/jpeg 0.305s
ok image/png 0.042s
ok index/suffixarray 0.360s
ok internal/cpu 0.035s
ok internal/fmtsort 0.014s
ok internal/oserror 0.012s
ok internal/poll 0.056s
ok internal/reflectlite 0.011s
ok internal/singleflight 0.020s
ok internal/trace 0.048s
ok internal/xcoff 0.030s
ok io 0.033s
ok io/ioutil 0.016s
ok log 0.013s
ok log/syslog 1.241s
ok math 0.020s
ok math/big 0.362s
ok math/bits 0.087s
ok math/cmplx 0.014s
ok math/rand 0.186s
ok mime 0.014s
ok mime/multipart 0.418s
ok mime/quotedprintable 0.020s
ok net 2.119s
ok net/http 2.904s
ok net/http/cgi 0.825s
ok net/http/cookiejar 0.036s
ok net/http/fcgi 0.028s
ok net/http/httptest 0.050s
ok net/http/httptrace 0.028s
ok net/http/httputil 0.071s
ok net/http/internal 0.012s
ok net/http/pprof 2.037s
ok net/internal/socktest 0.014s
ok net/mail 0.019s
ok net/rpc 0.091s
ok net/rpc/jsonrpc 0.031s
ok net/smtp 0.039s
ok net/textproto 0.016s
ok net/url 0.022s
ok os 0.901s
ok os/exec 1.522s
ok os/signal 2.893s
ok os/user 0.014s
ok path 0.011s
ok path/filepath 0.057s
ok plugin 0.011s
ok reflect 0.134s
ok regexp 0.083s
ok regexp/syntax 1.800s
ok runtime 24.725s
ok runtime/debug 0.079s
ok runtime/internal/atomic 0.017s
ok runtime/internal/math 0.010s
ok runtime/internal/sys 0.014s
ok runtime/pprof 9.488s
ok runtime/pprof/internal/profile 0.012s
ok runtime/trace 0.610s
ok sort 0.155s
ok strconv 1.791s
ok strings 0.236s
ok sync 0.394s
ok sync/atomic 0.032s
ok syscall 0.096s
ok testing 0.049s
ok testing/quick 0.056s
ok text/scanner 0.013s
ok text/tabwriter 0.013s
ok text/template 0.052s
ok text/template/parse 0.019s
ok time 2.648s
ok unicode 0.011s
ok unicode/utf16 0.015s
ok unicode/utf8 0.019s
ok cmd/addr2line 2.288s
ok cmd/api 0.035s
ok cmd/asm/internal/asm 0.968s
ok cmd/asm/internal/lex 0.008s
ok cmd/compile 0.013s
ok cmd/compile/internal/gc 19.163s
ok cmd/compile/internal/ssa 0.327s
ok cmd/compile/internal/syntax 0.020s
ok cmd/compile/internal/test 0.007s [no tests to run]
ok cmd/compile/internal/types 0.007s
ok cmd/cover 2.561s
ok cmd/doc 0.068s
ok cmd/fix 0.039s
ok cmd/go 53.396s
ok cmd/go/internal/auth 0.026s
ok cmd/go/internal/cache 0.822s
ok cmd/go/internal/dirhash 0.015s
ok cmd/go/internal/generate 0.037s
ok cmd/go/internal/get 0.085s
ok cmd/go/internal/imports 0.031s
ok cmd/go/internal/load 0.016s
ok cmd/go/internal/lockedfile 0.026s
ok cmd/go/internal/lockedfile/internal/filelock 0.046s
ok cmd/go/internal/modconv 0.047s
ok cmd/go/internal/modfetch 0.152s
ok cmd/go/internal/modfetch/codehost 0.012s
ok cmd/go/internal/modfile 0.018s
ok cmd/go/internal/modload 0.040s
ok cmd/go/internal/module 0.011s
ok cmd/go/internal/mvs 0.013s
ok cmd/go/internal/note 0.027s
ok cmd/go/internal/par 0.029s
ok cmd/go/internal/renameio 0.876s
ok cmd/go/internal/search 0.031s
ok cmd/go/internal/semver 0.033s
ok cmd/go/internal/sumweb 0.043s
ok cmd/go/internal/tlog 2.020s
ok cmd/go/internal/txtar 0.012s
ok cmd/go/internal/web 0.029s
ok cmd/go/internal/work 0.024s
ok cmd/gofmt 0.074s
ok cmd/internal/buildid 0.251s
ok cmd/internal/dwarf 0.011s
ok cmd/internal/edit 0.010s
ok cmd/internal/goobj 1.351s
ok cmd/internal/obj 0.012s
ok cmd/internal/obj/arm64 0.050s
ok cmd/internal/obj/x86 5.211s
ok cmd/internal/objabi 0.014s
ok cmd/internal/src 0.011s
ok cmd/internal/test2json 0.161s
ok cmd/link 5.084s
ok cmd/link/internal/ld 7.583s
ok cmd/link/internal/sym 0.012s
ok cmd/nm 3.880s
ok cmd/objdump 3.713s
ok cmd/pack 1.528s
ok cmd/trace 0.032s
ok cmd/vet 5.105s
2019/07/02 10:43:23 Failed: exit status 1

I can successfully build it if I run the command as sudo, but this makes sense as root should not have the certificates 🙂

What I am wondering however, do I need a patched version of the go binary to be able to build the patched version?

@FiloSottile requested to be tagged.

Copy link

@bcmills bcmills commented Jul 2, 2019

CC @agl

@bcmills bcmills added this to the Go1.13 milestone Jul 2, 2019
@bcmills bcmills added the OS-Darwin label Jul 2, 2019
Copy link

@tommyknows tommyknows commented Jul 8, 2019

Ping @agl / @FiloSottile
I'd be happy to help if you could point me into the right direction.

@FiloSottile FiloSottile changed the title crypto/x509 trust setting not inherited on Darwin crypto/x509: trust setting not inherited on Darwin Jul 8, 2019
Copy link

@FiloSottile FiloSottile commented Jul 8, 2019

Please provide the full debug output. Instead of ./all.bash use ./make.bash, and then GODEBUG=x509roots=1 ../bin/go test -v -run TestSystemRoots crypto/x509.

Copy link

@tommyknows tommyknows commented Jul 8, 2019

Seems like I can't even run the tests like this:

bash-3.2$ ./make.bash
Building Go cmd/dist using /usr/local/go.
Building Go toolchain1 using /usr/local/go.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
Building Go toolchain2 using go_bootstrap and Go toolchain1.
Building Go toolchain3 using go_bootstrap and Go toolchain2.
Building packages and commands for darwin/amd64.
Installed Go for darwin/amd64 in /Users/ramon/Documents/go/go-src
Installed commands in /Users/ramon/Documents/go/go-src/bin
bash-3.2$ GODEBUG=x509roots=1 ../bin/go test -v -run TestSystemRoots crypto/x509
# crypto/x509 [crypto/x509.test]
/var/folders/fm/tzdsx49104v20lr4m55_g43w0000gn/T/go-build347514254/b050/_testinginit.go:8:2: undefined: testing.Init
FAIL    crypto/x509 [build failed]

While ./all.bash works "fine" (in the sense of at least building)

@andybons andybons modified the milestones: Go1.13, Go1.14 Jul 8, 2019
Copy link

@FiloSottile FiloSottile commented Jul 9, 2019

That's a really weird failure and seems unrelated. Do you have GOROOT set, maybe?

Copy link

@tommyknows tommyknows commented Jul 9, 2019

yes, that was the case.

Sadly, I cleaned up my certificates and removed the ones that were signed by an unknown CA 🤦‍♂

I tried adding a certificate back in, but I couldn't get the cert to show up in the log output.
I don't exactly know what is needed for the certificate to show up in the SystemRoots, and if I can just create such a certificate with openssl.

Now, if you know how I can get a new certificate into that list, I'm willing to add some untrusted certificates to my System Keychain 🙂

If not, then this issue can be closed.

Sorry for the noise, stupid me deleting those certificates 😅

Copy link

@FiloSottile FiloSottile commented Jul 10, 2019

I'm afraid we don't have enough information to debug this, and I can't provide you instructions because at this point mismatches would be due to unexpected states of the keychain, which I wouldn't know how to reproduce.

No worries about it, and thanks for reporting it in the first place.

Copy link

@gopherbot gopherbot commented Apr 14, 2020

Change mentions this issue: crypto/x509: use Security.framework without cgo for roots on macOS

gopherbot pushed a commit that referenced this issue May 7, 2020
| Hello, if you are reading this and run macOS, please test this code: |
|                                                                      |
| $ GO111MODULE=on go get                   |
| $ gotip download                                              |
| $ GODEBUG=x509roots=1 gotip test crypto/x509 -v -run TestSystemRoots |

We currently have two code paths to extract system roots on macOS: one
uses cgo to invoke a maze of Security.framework APIs; the other is a
horrible fallback that runs "/usr/bin/security verify-cert" on every
root that has custom policies to check if it's trusted for SSL.

The fallback is not only terrifying because it shells out to a binary,
but also because it lets in certificates that are not trusted roots but
are signed by trusted roots, and because it applies some filters (EKUs
and expiration) only to roots with custom policies, as the others are
not passed to verify-cert. The other code path, of course, requires cgo,
so can't be used when cross-compiling and involves a large ball of C.

It's all a mess, and it broke oh-so-many times (#14514, #16532, #19436,
 #20990, #21416, #24437, #24652, #25649, #26073, #27958, #28025, #28092,
 #29497, #30471, #30672, #30763, #30889, #32891, #38215, #38365, ...).

Since macOS does not have a stable syscall ABI, we already dynamically
link and invoke libSystem.dylib regardless of cgo availability (#17490).

How that works is that functions in package syscall (like syscall.Open)
take the address of assembly trampolines (like libc_open_trampoline)
that jump to symbols imported with cgo_import_dynamic (like libc_open),
and pass them along with arguments to syscall.syscall (which is
implemented as runtime.syscall_syscall). syscall_syscall informs the
scheduler and profiler, and then uses asmcgocall to switch to a system
stack and invoke runtime.syscall. The latter is an assembly trampoline
that unpacks the Go ABI arguments passed to syscall.syscall, finally
calls the remote function, and puts the return value on the Go stack.
(This last bit is the part that cgo compiles from a C wrapper.)

We can do something similar to link and invoke Security.framework!

The one difference is that runtime.syscall and friends check errors
based on the errno convention, which Security doesn't follow, so I added
runtime.syscallNoErr which just skips interpreting the return value.
We only need a variant with six arguments because the calling convention
is register-based, and extra arguments simply zero out some registers.

That's plumbed through as crypto/x509/internal/macOS.syscall. The rest
of that package is a set of wrappers for Security.framework and Core
Foundation functions, like syscall is for libSystem. In theory, as long
as macOS respects ABI backwards compatibility (a.k.a. as long as
binaries built for a previous OS version keep running) this should be
stable, as the final result is not different from what a C compiler
would make. (One exception might be dictionary key strings, which we
make our own copy of instead of using the dynamic symbol. If they change
the value of those strings things might break. But why would they.)

Finally, I rewrote the crypto/x509 cgo logic in Go using those wrappers.
It works! I tried to make it match 1:1 the old logic, so that
root_darwin_amd64.go can be reviewed by comparing it to
root_cgo_darwin_amd64.go. The only difference is that we do proper error
handling now, and assume that if there is no error the return values are
there, while before we'd just check for nil pointers and move on.

I kept the cgo logic to help with review and testing, but we should
delete it once we are confident the new code works.

The nocgo logic is gone and we shall never speak of it again.

Fixes #32604
Fixes #19561
Fixes #38365
Awakens Cthulhu

Change-Id: Id850962bad667f71e3af594bdfebbbb1edfbcbb4
Reviewed-by: Katie Hockman <>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
5 participants
You can’t perform that action at this time.