Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

crypto/x509: TestSystemRoots flaky & slow in non-cgo case #18203

Closed
phacker opened this issue Dec 6, 2016 · 24 comments
Closed

crypto/x509: TestSystemRoots flaky & slow in non-cgo case #18203

phacker opened this issue Dec 6, 2016 · 24 comments
Assignees
Milestone

Comments

@phacker
Copy link

@phacker phacker commented Dec 6, 2016

Please answer these questions before submitting your issue. Thanks!
Using MacBook Pro macOS 10.12.1

What version of Go are you using (go version)?

go1.7.4

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH=""
GORACE=""
GOROOT="/usr/local/go"
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
CC="clang"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/pb/vn9226r53sscpljzr0g3yfw40000gn/T/go-build062460274=/tmp/go-build -gno-record-gcc-switches -fno-common"
CXX="clang++"
CGO_ENABLED="1"

What did you do?

Compiling tip (80acfe9)
Using 1.7.4 standard install at /usr/local/go
$ printenv | grep ^GO
GOROOT_BOOTSTRAP=/usr/local/go
$ cd golang/go/src
$ ./all.bash
Pats-MBP:src path$ ./all.bash

Building Go bootstrap tool.

cmd/dist

Building Go toolchain using /usr/local/go.

bootstrap/cmd/internal/dwarf
bootstrap/cmd/internal/sys
bootstrap/cmd/asm/internal/flags
bootstrap/cmd/internal/bio
bootstrap/cmd/compile/internal/syntax
bootstrap/cmd/internal/gcprog
bootstrap/cmd/internal/obj
bootstrap/math/big
bootstrap/debug/pe
bootstrap/cmd/internal/obj/arm
bootstrap/cmd/internal/obj/arm64
bootstrap/cmd/internal/obj/mips
bootstrap/cmd/internal/obj/ppc64
bootstrap/cmd/internal/obj/s390x
bootstrap/cmd/internal/obj/x86
bootstrap/cmd/asm/internal/lex
bootstrap/cmd/link/internal/ld
bootstrap/cmd/asm/internal/arch
bootstrap/cmd/compile/internal/ssa
bootstrap/cmd/asm/internal/asm
bootstrap/cmd/asm
bootstrap/cmd/link/internal/arm
bootstrap/cmd/link/internal/amd64
bootstrap/cmd/link/internal/arm64
bootstrap/cmd/link/internal/mips
bootstrap/cmd/link/internal/mips64
bootstrap/cmd/link/internal/ppc64
bootstrap/cmd/link/internal/s390x
bootstrap/cmd/link/internal/x86
bootstrap/cmd/link
bootstrap/cmd/compile/internal/gc
bootstrap/cmd/compile/internal/amd64
bootstrap/cmd/compile/internal/arm
bootstrap/cmd/compile/internal/arm64
bootstrap/cmd/compile/internal/mips
bootstrap/cmd/compile/internal/mips64
bootstrap/cmd/compile/internal/ppc64
bootstrap/cmd/compile/internal/s390x
bootstrap/cmd/compile/internal/x86
bootstrap/cmd/compile

Building go_bootstrap for host, darwin/amd64.

runtime/internal/sys
runtime/internal/atomic
runtime
errors
sync/atomic
unicode/utf8
unicode/utf16
unicode
internal/syscall/windows/sysdll
math
internal/race
encoding
sync
syscall
io
internal/singleflight
hash
hash/adler32
bytes
strings
strconv
bufio
path
crypto
encoding/base64
reflect
crypto/sha1
internal/syscall/windows/registry
internal/syscall/windows
time
os
os/signal
encoding/binary
fmt
sort
path/filepath
container/heap
regexp/syntax
io/ioutil
context
debug/dwarf
encoding/json
log
go/token
net/url
compress/flate
text/template/parse
flag
os/exec
go/scanner
go/ast
regexp
compress/zlib
text/template
debug/macho
debug/elf
go/parser
go/doc
go/build
cmd/go

Building packages and commands for darwin/amd64.

runtime/internal/sys
runtime/internal/atomic
runtime
errors
unicode
internal/race
sync/atomic
unicode/utf8
math
sync
container/list
io
syscall
hash
hash/adler32
hash/crc32
strconv
bytes
strings
bufio
path
reflect
text/tabwriter
container/ring
crypto
crypto/subtle
crypto/internal/cipherhw
crypto/cipher
math/rand
crypto/sha512
crypto/aes
crypto/hmac
time
crypto/md5
crypto/rc4
crypto/sha1
crypto/sha256
encoding/base64
internal/nettrace
internal/singleflight
vendor/golang_org/x/crypto/poly1305
vendor/golang_org/x/crypto/curve25519
database/sql/internal
encoding
encoding/ascii85
encoding/base32
unicode/utf16
vendor/golang_org/x/text/transform
hash/crc64
hash/fnv
html
image/color
os
image
image/color/palette
internal/syscall/windows
vendor/golang_org/x/net/route
internal/syscall/windows/registry
internal/syscall/windows/sysdll
runtime/trace
image/internal/imageutil
math/cmplx
image/draw
image/jpeg
os/signal
runtime/race
cmd/compile/internal/test
cmd/vet/internal/whitelist
fmt
encoding/binary
sort
path/filepath
regexp/syntax
cmd/internal/sys
compress/bzip2
io/ioutil
container/heap
crypto/des
encoding/pem
vendor/golang_org/x/crypto/chacha20poly1305/internal/chacha20
runtime/debug
vendor/golang_org/x/crypto/chacha20poly1305
cmd/internal/dwarf
flag
log
debug/dwarf
compress/flate
debug/gosym
cmd/internal/obj
debug/plan9obj
cmd/vendor/golang.org/x/arch/arm/armasm
debug/macho
debug/pe
cmd/vendor/golang.org/x/arch/ppc64/ppc64asm
cmd/vendor/golang.org/x/arch/x86/x86asm
cmd/internal/goobj
compress/zlib
debug/elf
regexp
archive/tar
archive/zip
compress/gzip
compress/lzw
context
math/big
cmd/internal/objfile
encoding/hex
go/token
os/exec
go/scanner
go/ast
cmd/addr2line
database/sql/driver
database/sql
go/parser
go/printer
encoding/csv
encoding/gob
crypto/dsa
crypto/elliptic
encoding/asn1
crypto/rand
cmd/cgo
crypto/rsa
crypto/ecdsa
crypto/x509/pkix
encoding/json
encoding/xml
vendor/golang_org/x/net/http2/hpack
vendor/golang_org/x/net/idna
vendor/golang_org/x/text/unicode/norm
vendor/golang_org/x/text/width
mime
mime/quotedprintable
net/http/internal
net/url
text/template/parse
go/constant
go/format
text/scanner
image/gif
go/types
image/png
text/template
index/suffixarray
runtime/cgo
internal/pprof/profile
go/doc
html/template
go/build
net
testing
internal/trace
internal/testenv
runtime/pprof/internal/protopprof
net/internal/socktest
runtime/pprof
os/user
plugin
go/internal/gccgoimporter
go/internal/gcimporter
testing/internal/testdeps
testing/iotest
testing/quick
cmd/api
go/importer
cmd/internal/obj/arm
cmd/internal/obj/arm64
cmd/internal/obj/mips
cmd/internal/obj/ppc64
cmd/internal/obj/s390x
cmd/internal/obj/x86
cmd/asm/internal/flags
cmd/asm/internal/lex
cmd/internal/bio
cmd/compile/internal/syntax
cmd/internal/gcprog
crypto/x509
vendor/golang_org/x/net/lex/httplex
net/textproto
mime/multipart
log/syslog
net/mail
cmd/asm/internal/arch
cmd/compile/internal/ssa
cmd/asm/internal/asm
cmd/internal/browser
cmd/cover
cmd/asm
cmd/dist
cmd/doc
crypto/tls
cmd/fix
cmd/gofmt
net/http/httptrace
net/http
net/smtp
cmd/link/internal/ld
cmd/nm
cmd/objdump
cmd/pack
cmd/pprof/internal/plugin
cmd/pprof/internal/report
cmd/pprof/internal/svg
cmd/pprof/internal/tempfile
cmd/pprof/internal/commands
cmd/pprof/internal/driver
cmd/pprof/internal/symbolizer
cmd/pprof/internal/symbolz
cmd/vet/internal/cfg
cmd/vet
cmd/link/internal/amd64
expvar
net/http/cgi
net/http/cookiejar
net/http/httptest
net/http/fcgi
net/http/httputil
net/http/pprof
net/rpc
cmd/go
net/rpc/jsonrpc
cmd/link/internal/arm
cmd/link/internal/arm64
cmd/link/internal/mips
cmd/link/internal/mips64
cmd/link/internal/ppc64
cmd/link/internal/s390x
cmd/link/internal/x86
cmd/pprof/internal/fetch
cmd/link
cmd/pprof
cmd/trace
cmd/compile/internal/gc
cmd/compile/internal/amd64
cmd/compile/internal/arm
cmd/compile/internal/arm64
cmd/compile/internal/mips
cmd/compile/internal/mips64
cmd/compile/internal/ppc64
cmd/compile/internal/s390x
cmd/compile/internal/x86
cmd/compile

Testing packages.

ok archive/tar 0.027s
ok archive/zip 1.271s
ok bufio 0.060s
ok bytes 0.081s
ok compress/bzip2 0.058s
ok compress/flate 0.681s
ok compress/gzip 0.017s
ok compress/lzw 0.011s
ok compress/zlib 0.020s
ok container/heap 0.011s
ok container/list 0.010s
ok container/ring 0.022s
ok context 1.040s
ok crypto/aes 0.033s
ok crypto/cipher 0.028s
ok crypto/des 0.015s
ok crypto/dsa 0.011s
ok crypto/ecdsa 0.066s
ok crypto/elliptic 0.032s
ok crypto/hmac 0.006s
ok crypto/md5 0.014s
ok crypto/rand 0.028s
ok crypto/rc4 0.124s
ok crypto/rsa 0.086s
ok crypto/sha1 0.028s
ok crypto/sha256 0.009s
ok crypto/sha512 0.012s
ok crypto/subtle 0.010s
ok crypto/tls 1.229s
--- FAIL: TestSystemRoots (1.46s)
root_darwin_test.go:32: got 172 roots
root_darwin_test.go:32: got 132 roots
root_darwin_test.go:34: want at least 150 system roots, have 132
FAIL
FAIL crypto/x509 2.215s
ok database/sql 0.087s
ok database/sql/driver 0.008s
ok debug/dwarf 0.014s
ok debug/elf 0.029s
ok debug/gosym 0.243s
ok debug/macho 0.012s
ok debug/pe 0.012s
ok debug/plan9obj 0.010s
ok encoding/ascii85 0.010s
ok encoding/asn1 0.011s
ok encoding/base32 0.010s
ok encoding/base64 0.009s
ok encoding/binary 0.010s
ok encoding/csv 0.011s
ok encoding/gob 0.043s
ok encoding/hex 0.011s
ok encoding/json 0.385s
ok encoding/pem 0.019s
ok encoding/xml 0.016s
ok errors 0.009s
ok expvar 0.012s
ok flag 0.010s
ok fmt 0.103s
ok go/ast 0.013s
ok go/build 0.091s
ok go/constant 0.026s
ok go/doc 0.045s
ok go/format 0.013s
ok go/internal/gccgoimporter 0.010s
ok go/internal/gcimporter 0.517s
ok go/parser 0.035s
ok go/printer 0.336s
ok go/scanner 0.010s
ok go/token 0.026s
ok go/types 0.673s
ok hash/adler32 0.012s
ok hash/crc32 0.012s
ok hash/crc64 0.009s
ok hash/fnv 0.011s
ok html 0.008s
ok html/template 0.069s
ok image 0.111s
ok image/color 0.058s
ok image/draw 0.062s
ok image/gif 0.074s
ok image/jpeg 0.125s
ok image/png 0.035s
ok index/suffixarray 0.015s
ok internal/pprof/profile 0.009s
ok internal/singleflight 0.019s
ok internal/trace 0.394s
ok io 0.031s
ok io/ioutil 0.011s
ok log 0.012s
ok log/syslog 1.253s
ok math 0.011s
ok math/big 1.534s
ok math/cmplx 0.008s
ok math/rand 0.054s
ok mime 0.012s
ok mime/multipart 0.240s
ok mime/quotedprintable 0.088s
ok net 4.106s
ok net/http 5.470s
ok net/http/cgi 0.508s
ok net/http/cookiejar 0.015s
ok net/http/fcgi 0.013s
ok net/http/httptest 0.015s
ok net/http/httptrace 0.012s
ok net/http/httputil 0.033s
ok net/http/internal 0.008s
ok net/internal/socktest 0.010s
ok net/mail 0.011s
ok net/rpc 0.023s
ok net/rpc/jsonrpc 0.015s
ok net/smtp 0.020s
ok net/textproto 0.014s
ok net/url 0.016s
ok os 0.524s
ok os/exec 0.434s
ok os/signal 4.307s
ok os/user 0.009s
ok path 0.008s
ok path/filepath 0.041s
ok reflect 0.049s
ok regexp 0.098s
ok regexp/syntax 0.226s
ok runtime 16.145s
ok runtime/debug 0.010s
ok runtime/internal/atomic 0.103s
ok runtime/internal/sys 0.008s
ok runtime/pprof 1.594s
ok runtime/pprof/internal/protopprof 0.011s
ok runtime/trace 1.615s
ok sort 0.063s
ok strconv 0.500s
ok strings 0.105s
ok sync 0.214s
ok sync/atomic 0.029s
ok syscall 0.079s
ok testing 1.469s
ok testing/quick 0.091s
ok text/scanner 0.012s
ok text/tabwriter 0.211s
ok text/template 0.408s
ok text/template/parse 0.013s
ok time 2.434s
ok unicode 0.011s
ok unicode/utf16 0.009s
ok unicode/utf8 0.010s
ok vendor/golang_org/x/crypto/chacha20poly1305 0.266s
ok vendor/golang_org/x/crypto/chacha20poly1305/internal/chacha20 0.007s
ok vendor/golang_org/x/crypto/curve25519 0.023s
ok vendor/golang_org/x/crypto/poly1305 0.009s
ok vendor/golang_org/x/net/http2/hpack 0.010s
ok vendor/golang_org/x/net/idna 0.011s
ok vendor/golang_org/x/net/lex/httplex 0.009s
ok vendor/golang_org/x/net/route 0.022s
ok cmd/addr2line 0.531s
ok cmd/api 0.013s
ok cmd/asm/internal/asm 0.165s
ok cmd/asm/internal/lex 0.012s
ok cmd/compile 4.633s
ok cmd/compile/internal/gc 8.041s
ok cmd/compile/internal/ssa 0.120s
ok cmd/compile/internal/syntax 0.013s
ok cmd/compile/internal/test 0.009s [no tests to run]
ok cmd/cover 0.875s
ok cmd/doc 0.026s
ok cmd/fix 0.015s
ok cmd/go 36.364s
ok cmd/gofmt 0.034s
ok cmd/internal/goobj 0.008s
ok cmd/internal/obj 0.011s
ok cmd/internal/obj/arm64 0.008s
ok cmd/internal/obj/x86 0.010s
ok cmd/link 0.010s
ok cmd/nm 0.791s
ok cmd/objdump 1.189s
ok cmd/pack 1.486s
ok cmd/trace 0.014s
ok cmd/vendor/golang.org/x/arch/arm/armasm 0.011s
ok cmd/vendor/golang.org/x/arch/ppc64/ppc64asm 0.010s
ok cmd/vendor/golang.org/x/arch/x86/x86asm 0.209s
ok cmd/vet 1.750s
ok cmd/vet/internal/cfg 0.009s
2016/12/05 17:19:25 Failed: exit status 1

If possible, provide a recipe for reproducing the error.
A complete runnable program is good.
A link on play.golang.org is best.

What did you expect to see?

Success

What did you see instead?

ok crypto/subtle 0.012s
ok crypto/tls 1.389s
--- FAIL: TestSystemRoots (1.28s)
root_darwin_test.go:32: got 172 roots
root_darwin_test.go:32: got 149 roots
root_darwin_test.go:34: want at least 150 system roots, have 149
FAIL
FAIL crypto/x509 2.081s

@quentinmit
Copy link
Contributor

@quentinmit quentinmit commented Dec 6, 2016

Do you have a number of CAs disabled in Keychain Access? I guess we can slightly lower the threshold.

@pathacker
Copy link

@pathacker pathacker commented Dec 6, 2016

@bradfitz bradfitz changed the title crypto/x509 TestSystemRoots crypto/x509: TestSystemRoots Dec 6, 2016
@bradfitz bradfitz added this to the Go1.8 milestone Dec 6, 2016
@bradfitz bradfitz added the Testing label Dec 6, 2016
@bradfitz
Copy link
Contributor

@bradfitz bradfitz commented Dec 6, 2016

@quentinmit, leaving for you to lower the threshold.

@cookieo9
Copy link
Contributor

@cookieo9 cookieo9 commented Dec 6, 2016

I have a similar issue, my case:

root_darwin_test.go:34: want at least 150 system roots, have 145

In my case I've added certificates (to support delve, a debugger), although not necessarily root certificates. I've certainly not removed them. My machine was upgraded to Sierra (10.12) from Mavericks (10.10) so that might explain the difference there.

Has Apple invalidated some certificates recently? It might have something to do with: https://support.apple.com/en-us/HT202858

@quentinmit quentinmit added the NeedsFix label Dec 6, 2016
@quentinmit quentinmit changed the title crypto/x509: TestSystemRoots crypto/x509: TestSystemRoots finds 0<n<150 certificates Dec 6, 2016
@bradfitz bradfitz assigned bradfitz and unassigned quentinmit Dec 7, 2016
@gopherbot
Copy link

@gopherbot gopherbot commented Dec 7, 2016

CL https://golang.org/cl/34019 mentions this issue.

@gopherbot gopherbot closed this in be8a6fd Dec 7, 2016
@quentinmit
Copy link
Contributor

@quentinmit quentinmit commented Dec 7, 2016

@bradfitz @rsc
I'm worried that this is not as simple as lowering the threshold, which is why I didn't just send a CL to do that. My Sierra machine reports 169 roots and my El Capitan machine reports 175 roots.

The error report is about the second number printed by TestSystemRoots, which is the number of root CAs reported by the nocgo codepath. That codepath only looks at certificates provided by Apple, so the only way that number should differ should be if the user marks a cert as untrusted. The users here have reported that they didn't disable any certs, so that leaves me scratching my head.

@quentinmit quentinmit reopened this Dec 7, 2016
@phacker
Copy link
Author

@phacker phacker commented Dec 7, 2016

@phacker
Copy link
Author

@phacker phacker commented Dec 7, 2016

@quentinmit
Copy link
Contributor

@quentinmit quentinmit commented Dec 7, 2016

@phacker
I'm just referring to the number of roots that Go logs, when you do

go test -v -run TestSystemRoots crypto/x509

Your log above says it found 132 roots. Actually, that's very weird - you also pasted a log that says it found 149 roots in the "What did you see instead?" section.

To eliminate any problems with Go, let's go back to the underlying data. It's stored in /System/Library/Keychains/SystemRootCertificates.keychain, so you should be able to count Apple's roots with:

security find-certificate -a -p /System/Library/Keychains/SystemRootCertificates.keychain | grep BEGIN | wc -l

This does not take into account disabled certificates. On my El Capitan machine, I get 189 certificates in the keychain, but when Go checks if they're trusted it ends up seeing a consistent 175 roots. If you run the "go test" command above, do you get a different number of roots each time?

@phacker
Copy link
Author

@phacker phacker commented Dec 7, 2016

@quentinmit
Copy link
Contributor

@quentinmit quentinmit commented Dec 7, 2016

So your machine has 173 roots of unknown trust, and you're seeing a random number of trusted roots less than that every time you run the test.

Well, that's fun. :)

Can you apply this patch and rerun the test to see if we get any useful debugging output?

diff --git a/src/crypto/x509/root_darwin.go b/src/crypto/x509/root_darwin.go
index 59b303d..dff1593 100644
--- a/src/crypto/x509/root_darwin.go
+++ b/src/crypto/x509/root_darwin.go
@@ -93,7 +93,8 @@ func verifyCertWithSystem(block *pem.Block, add func(*Certificate)) {
                cmd = exec.Command("/usr/bin/security", "verify-cert", "-c", "/dev/stdin", "-l")
                cmd.Stdin = bytes.NewReader(data)
        }
-       if cmd.Run() == nil {
+       output, err := cmd.CombinedOutput()
+       if err == nil {
                // Non-zero exit means untrusted
                cert, err := ParseCertificate(block.Bytes)
                if err != nil {
@@ -101,7 +102,9 @@ func verifyCertWithSystem(block *pem.Block, add func(*Certificate)) {
                }
 
                add(cert)
+               return
        }
+       fmt.Fprintf(os.Stderr, "verify-cert failed: %v\n%s\n", err, output)
 }
@quentinmit quentinmit removed the Testing label Dec 7, 2016
@phacker
Copy link
Author

@phacker phacker commented Dec 7, 2016

@bradfitz bradfitz assigned quentinmit and unassigned bradfitz Dec 8, 2016
@bradfitz
Copy link
Contributor

@bradfitz bradfitz commented Dec 8, 2016

@quentinmit, you planning on doing something here?

@quentinmit
Copy link
Contributor

@quentinmit quentinmit commented Dec 8, 2016

@bradfitz Good question. I don't know what the next step is on this bug; Apple's error message isn't exactly helpful. We can move to 1.8Maybe though.

@quentinmit quentinmit modified the milestones: Go1.8Maybe, Go1.8 Dec 8, 2016
@quentinmit
Copy link
Contributor

@quentinmit quentinmit commented Dec 12, 2016

@phacker We have another thing we'd like you to try, since you're able to reproduce this. Can you change needsTmpFile() in src/crypto/x509/root_darwin.go to unconditionally "return true" and see if you can reproduce the bug?

I'm afraid I've been completely unable to reproduce this on every Mac I have access to, both Sierra and El Capitan.

@pathacker
Copy link

@pathacker pathacker commented Dec 12, 2016

@bradfitz
Copy link
Contributor

@bradfitz bradfitz commented Dec 12, 2016

@quentinmit, yeah, sounds like the piping to stdin from a bytes.Reader is flaky. Is Apple's verify-cert code open source? I don't know which parts are & aren't. I'm curious if there's a bug in how they read from stdin.

@quentinmit
Copy link
Contributor

@quentinmit quentinmit commented Dec 12, 2016

@gopherbot
Copy link

@gopherbot gopherbot commented Dec 15, 2016

CL https://golang.org/cl/34389 mentions this issue.

@bradfitz bradfitz modified the milestones: Go1.8, Go1.8Maybe Dec 16, 2016
@bradfitz bradfitz assigned bradfitz and unassigned quentinmit Dec 16, 2016
@bradfitz bradfitz changed the title crypto/x509: TestSystemRoots finds 0<n<150 certificates crypto/x509: TestSystemRoots flaky & slow in non-cgo case Dec 16, 2016
@bradfitz
Copy link
Contributor

@bradfitz bradfitz commented Dec 16, 2016

See https://golang.org/cl/34389 for a new strategy that doesn't take 3.5 seconds to load certs.

/cc @mitchellh (could you test?)

@cblecker
Copy link

@cblecker cblecker commented Jan 17, 2017

Hi @bradfitz --
Sorry for commenting on a closed issue, but I'm not sure where it would be more appropriate to (I'm not really involved in the golang community). I've seen this issue pop up another place. On Jan 11, the Google Cloud SDK team included a new version of kubectl with the google-cloud-sdk. This new version is compiled with go1.7.4, rather than the previous release that was compiled with go1.7.1.

This combination of kubectl compiled under go1.7.4 + macOS Sierra hits this bug, causing timeouts. I've seen mentions of it helm/helm#1749 and https://kubernetes.slack.com/archives/kubernetes-users/p1484587693022364, as well as encountered it myself. These issues seem to go away when using the version of kubectl compiled with Homebrew that is currently patching in the fix (Homebrew/homebrew-core#8628).

I'm not sure if there is a plan to release another 1.7.x go version before 1.8 drops next month, but I wanted to let you know that this bug is impacting users out in the wild.

If there is a better way or place to report this, then please let me know (and again my apologies).

@minux
Copy link
Member

@minux minux commented Jan 17, 2017

@bradfitz
Copy link
Contributor

@bradfitz bradfitz commented Jan 17, 2017

@cblecker, ugh. Please file a new bug and reference this one. (Comments on closed issues are where action items go to die.) No promises that we'll do a 1.7.5, but if we do, we'll consider it.

@gopherbot
Copy link

@gopherbot gopherbot commented Jan 24, 2017

CL https://golang.org/cl/35634 mentions this issue.

gopherbot pushed a commit that referenced this issue Jan 25, 2017
…in root cert discovery

Backporting Go 1.8's fix to #18203
Fixes #18688

---

Piping into security verify-cert only worked on macOS Sierra, and was
flaky for unknown reasons. Users reported that the number of trusted
root certs stopped randomly jumping around once they switched to using
verify-cert against files on disk instead of /dev/stdin.

But even using "security verify-cert" on 150-200 certs took too
long. It took 3.5 seconds on my machine. More than 4 goroutines
hitting verify-cert didn't help much, and soon started to hurt
instead.

New strategy, from comments in the code:

// 1. Run "security trust-settings-export" and "security
//    trust-settings-export -d" to discover the set of certs with some
//    user-tweaked trusy policy. We're too lazy to parse the XML (at
//    least at this stage of Go 1.8) to understand what the trust
//    policy actually is. We just learn that there is _some_ policy.
//
// 2. Run "security find-certificate" to dump the list of system root
//    CAs in PEM format.
//
// 3. For each dumped cert, conditionally verify it with "security
//    verify-cert" if that cert was in the set discovered in Step 1.
//    Without the Step 1 optimization, running "security verify-cert"
//    150-200 times takes 3.5 seconds. With the optimization, the
//    whole process takes about 180 milliseconds with 1 untrusted root
//    CA. (Compared to 110ms in the cgo path)

Change-Id: I79737d9f2cb9b020ba297a326d4d31d68c7e9fee
Reviewed-on: https://go-review.googlesource.com/35634
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
@golang golang locked and limited conversation to collaborators Jan 24, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
9 participants
You can’t perform that action at this time.