Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

crypto/x509: macos 10.14 SIGSEGV in crypto/x509._Cfunc_FetchPEMRoots #28092

Closed
akamensky opened this issue Oct 9, 2018 · 37 comments
Closed

crypto/x509: macos 10.14 SIGSEGV in crypto/x509._Cfunc_FetchPEMRoots #28092

akamensky opened this issue Oct 9, 2018 · 37 comments
Assignees
Milestone

Comments

@akamensky
Copy link

@akamensky akamensky commented Oct 9, 2018

What version of Go are you using (go version)?

go version go1.11.1 darwin/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/username/Library/Caches/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/username/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/j6/vhm757px2zz3jgvxy1w3gfzr0000gn/T/go-build447402874=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

Just running (and debugging) the code that was working perfectly fine on 1.10 (since 1.11 release a lot of weird issues like debugger cannot kill process sometimes, and runaway CPU usage randomly etc etc)

What did you expect to see?

No crashing

What did you see instead?

fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8c72e3e pc=0x8c57013]

runtime stack:
runtime.throw(0x1bea596, 0x2a)
	/usr/local/go/src/runtime/panic.go:608 +0x72
runtime.sigpanic()
	/usr/local/go/src/runtime/signal_unix.go:374 +0x288

goroutine 130 [syscall]:
runtime.cgocall(0x1001ba0, 0xc00071aa30, 0x1be9723)
	/usr/local/go/src/runtime/cgocall.go:128 +0x5e fp=0xc00071a9f0 sp=0xc00071a9b8 pc=0x1004afe
crypto/x509._Cfunc_FetchPEMRoots(0xc000193ef8, 0xc000193f20, 0xc000000000)
	_cgo_gotypes.go:108 +0x4d fp=0xc00071aa30 sp=0xc00071a9f0 pc=0x128c7cd
crypto/x509.loadSystemRoots.func1(0xc000193ef8, 0xc000193f20, 0x0)
	/usr/local/go/src/crypto/x509/root_cgo_darwin.go:172 +0x1b8 fp=0xc00071aac0 sp=0xc00071aa30 pc=0x1292148
crypto/x509.loadSystemRoots(0x0, 0x0, 0x0)
	/usr/local/go/src/crypto/x509/root_cgo_darwin.go:172 +0xbf fp=0xc00071abb0 sp=0xc00071aac0 pc=0x128c98f
crypto/x509.initSystemRoots()
	/usr/local/go/src/crypto/x509/root.go:21 +0x37 fp=0xc00071abf0 sp=0xc00071abb0 pc=0x1272227
sync.(*Once).Do(0x25992d8, 0x1bff2c8)
	/usr/local/go/src/sync/once.go:44 +0xd1 fp=0xc00071ac28 sp=0xc00071abf0 pc=0x107e511
crypto/x509.systemRootsPool(0x0)
	/usr/local/go/src/crypto/x509/root.go:16 +0x42 fp=0xc00071ac48 sp=0xc00071ac28 pc=0x12721d2
crypto/x509.(*Certificate).Verify(0xc00015e100, 0xc000632b60, 0x10, 0xc0007931a0, 0x0, 0xbee74173c72b9df8, 0xc24227121, 0x257b7a0, 0x0, 0x0, ...)
	/usr/local/go/src/crypto/x509/verify.go:746 +0xb61 fp=0xc00071b030 sp=0xc00071ac48 pc=0x1278901
crypto/tls.(*clientHandshakeState).doFullHandshake(0xc00071be58, 0x0, 0x0)
	/usr/local/go/src/crypto/tls/handshake_client.go:340 +0x28a1 fp=0xc00071b988 sp=0xc00071b030 pc=0x12bd2f1
crypto/tls.(*clientHandshakeState).handshake(0xc00071be58, 0x0, 0x0)
	/usr/local/go/src/crypto/tls/handshake_client.go:247 +0xa01 fp=0xc00071bcd0 sp=0xc00071b988 pc=0x12ba291
crypto/tls.(*Conn).clientHandshake(0xc0000a3180, 0x0, 0x0)
	/usr/local/go/src/crypto/tls/handshake_client.go:168 +0x7ab fp=0xc00071bf10 sp=0xc00071bcd0 pc=0x12b96cb
crypto/tls.(*Conn).Handshake(0xc0000a3180, 0x0, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:1272 +0x14d fp=0xc00071bf78 sp=0xc00071bf10 pc=0x12b7a7d
net/http.(*persistConn).addTLS.func2(0x0, 0xc0000a3180, 0xc00036b810, 0xc0004cdbc0)
	/usr/local/go/src/net/http/transport.go:1171 +0x61 fp=0xc00071bfc0 sp=0xc00071bf78 pc=0x13ffac1
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc00071bfc8 sp=0xc00071bfc0 pc=0x105f961
created by net/http.(*persistConn).addTLS
	/usr/local/go/src/net/http/transport.go:1167 +0x21d

goroutine 1 [IO wait]:
internal/poll.runtime_pollWait(0x27a9e58, 0x72, 0x8)
	/usr/local/go/src/runtime/netpoll.go:173 +0x5e
internal/poll.(*pollDesc).wait(0xc00053b698, 0x72, 0x2527800, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x96
internal/poll.(*pollDesc).waitRead(0xc00053b698, 0xffffffffffffff00, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x45
internal/poll.(*FD).Accept(0xc00053b680, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:384 +0x383
net.(*netFD).accept(0xc00053b680, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/fd_unix.go:238 +0x7d
net.(*TCPListener).accept(0xc0004c22f8, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/tcpsock_posix.go:139 +0x56
net.(*TCPListener).AcceptTCP(0xc0004c22f8, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/tcpsock.go:247 +0x79
net/http.tcpKeepAliveListener.Accept(0xc0004c22f8, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:3232 +0x53
net/http.(*Server).Serve(0xc000335c70, 0x1ecc120, 0xc0004c22f8, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:2826 +0x350
net/http.(*Server).ListenAndServe(0xc000335c70, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:2764 +0x1ed
net/http.ListenAndServe(0x1bd1719, 0xe, 0x1ec6b60, 0xc00033ed00, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:3004 +0xd6
main.main()
	/Users/username/go/src/bitbucket.org/{{redacted}}/main.go:39 +0x43f

goroutine 7 [chan receive]:
github.com/go-redis/redis/internal/pool.(*ConnPool).reaper(0xc00013a160, 0xdf8475800)
	/Users/username/go/src/github.com/go-redis/redis/internal/pool/pool.go:449 +0xb4
created by github.com/go-redis/redis/internal/pool.NewConnPool
	/Users/username/go/src/github.com/go-redis/redis/internal/pool/pool.go:99 +0x2a3

goroutine 80 [chan receive]:
github.com/go-redis/redis/internal/pool.(*ConnPool).reaper(0xc000377a20, 0xdf8475800)
	/Users/username/go/src/github.com/go-redis/redis/internal/pool/pool.go:449 +0xb4
created by github.com/go-redis/redis/internal/pool.NewConnPool
	/Users/username/go/src/github.com/go-redis/redis/internal/pool/pool.go:99 +0x2a3

goroutine 51 [IO wait]:
internal/poll.runtime_pollWait(0x27a9d88, 0x72, 0x23)
	/usr/local/go/src/runtime/netpoll.go:173 +0x5e
internal/poll.(*pollDesc).wait(0xc00053b718, 0x72, 0x1ec7000, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x96
internal/poll.(*pollDesc).waitRead(0xc00053b718, 0xc000177000, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x45
internal/poll.(*FD).Read(0xc00053b700, 0xc000177000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x36a
net.(*netFD).Read(0xc00053b700, 0xc000177000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/fd_unix.go:202 +0x7a
net.(*conn).Read(0xc0004c2308, 0xc000177000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:177 +0xb0
net/http.(*connReader).Read(0xc0001b5140, 0xc000177000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:786 +0x2c4
bufio.(*Reader).fill(0xc0000a8840)
	/usr/local/go/src/bufio/bufio.go:100 +0x2d2
bufio.(*Reader).ReadSlice(0xc0000a8840, 0xc0006a550a, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/bufio/bufio.go:341 +0x3d6
bufio.(*Reader).ReadLine(0xc0000a8840, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/bufio/bufio.go:370 +0x91
net/textproto.(*Reader).readLineSlice(0xc0006f2de0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/textproto/reader.go:55 +0xaf
net/textproto.(*Reader).ReadLine(0xc0006f2de0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/textproto/reader.go:36 +0x6f
net/http.readRequest(0xc0000a8840, 0x101000, 0xc0008cd100, 0x0, 0x0)
	/usr/local/go/src/net/http/request.go:958 +0xb0
net/http.(*conn).readRequest(0xc0001ec460, 0x1ecc5e0, 0xc000216300, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:966 +0x4b0
net/http.(*conn).serve(0xc0001ec460, 0x1ecc5e0, 0xc000216300)
	/usr/local/go/src/net/http/server.go:1788 +0xa31
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:2851 +0x7b0

goroutine 52 [IO wait]:
internal/poll.runtime_pollWait(0x27a9cb8, 0x72, 0x23)
	/usr/local/go/src/runtime/netpoll.go:173 +0x5e
internal/poll.(*pollDesc).wait(0xc00053b798, 0x72, 0x1ec7000, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x96
internal/poll.(*pollDesc).waitRead(0xc00053b798, 0xc000644000, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x45
internal/poll.(*FD).Read(0xc00053b780, 0xc000644000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x36a
net.(*netFD).Read(0xc00053b780, 0xc000644000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/fd_unix.go:202 +0x7a
net.(*conn).Read(0xc0004c2310, 0xc000644000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:177 +0xb0
net/http.(*connReader).Read(0xc0006369c0, 0xc000644000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:786 +0x2c4
bufio.(*Reader).fill(0xc000081200)
	/usr/local/go/src/bufio/bufio.go:100 +0x2d2
bufio.(*Reader).ReadSlice(0xc000081200, 0xc00068550a, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/bufio/bufio.go:341 +0x3d6
bufio.(*Reader).ReadLine(0xc000081200, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/bufio/bufio.go:370 +0x91
net/textproto.(*Reader).readLineSlice(0xc0006369f0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/textproto/reader.go:55 +0xaf
net/textproto.(*Reader).ReadLine(0xc0006369f0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/textproto/reader.go:36 +0x6f
net/http.readRequest(0xc000081200, 0x101000, 0xc00074a500, 0x0, 0x0)
	/usr/local/go/src/net/http/request.go:958 +0xb0
net/http.(*conn).readRequest(0xc0001ec5a0, 0x1ecc5e0, 0xc000339f80, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:966 +0x4b0
net/http.(*conn).serve(0xc0001ec5a0, 0x1ecc5e0, 0xc000339f80)
	/usr/local/go/src/net/http/server.go:1788 +0xa31
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:2851 +0x7b0

goroutine 37 [chan receive]:
github.com/go-redis/redis/internal/pool.(*ConnPool).reaper(0xc000912000, 0xdf8475800)
	/Users/username/go/src/github.com/go-redis/redis/internal/pool/pool.go:449 +0xb4
created by github.com/go-redis/redis/internal/pool.NewConnPool
	/Users/username/go/src/github.com/go-redis/redis/internal/pool/pool.go:99 +0x2a3

goroutine 57 [chan receive]:
github.com/go-redis/redis/internal/pool.(*ConnPool).reaper(0xc000376f20, 0xdf8475800)
	/Users/username/go/src/github.com/go-redis/redis/internal/pool/pool.go:449 +0xb4
created by github.com/go-redis/redis/internal/pool.NewConnPool
	/Users/username/go/src/github.com/go-redis/redis/internal/pool/pool.go:99 +0x2a3

goroutine 74 [IO wait]:
internal/poll.runtime_pollWait(0x27a9638, 0x72, 0x23)
	/usr/local/go/src/runtime/netpoll.go:173 +0x5e
internal/poll.(*pollDesc).wait(0xc00053b318, 0x72, 0x1ec7000, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x96
internal/poll.(*pollDesc).waitRead(0xc00053b318, 0xc0007ac000, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x45
internal/poll.(*FD).Read(0xc00053b300, 0xc0007ac000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x36a
net.(*netFD).Read(0xc00053b300, 0xc0007ac000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/fd_unix.go:202 +0x7a
net.(*conn).Read(0xc00033b8a0, 0xc0007ac000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:177 +0xb0
net/http.(*connReader).Read(0xc0007aa360, 0xc0007ac000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:786 +0x2c4
bufio.(*Reader).fill(0xc00076e7e0)
	/usr/local/go/src/bufio/bufio.go:100 +0x2d2
bufio.(*Reader).ReadSlice(0xc00076e7e0, 0xc00068950a, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/bufio/bufio.go:341 +0x3d6
bufio.(*Reader).ReadLine(0xc00076e7e0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/bufio/bufio.go:370 +0x91
net/textproto.(*Reader).readLineSlice(0xc000703fb0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/textproto/reader.go:55 +0xaf
net/textproto.(*Reader).ReadLine(0xc000703fb0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/textproto/reader.go:36 +0x6f
net/http.readRequest(0xc00076e7e0, 0x101000, 0xc0008dec00, 0x0, 0x0)
	/usr/local/go/src/net/http/request.go:958 +0xb0
net/http.(*conn).readRequest(0xc0001badc0, 0x1ecc5e0, 0xc000774580, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:966 +0x4b0
net/http.(*conn).serve(0xc0001badc0, 0x1ecc5e0, 0xc000774580)
	/usr/local/go/src/net/http/server.go:1788 +0xa31
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:2851 +0x7b0

goroutine 72 [IO wait]:
internal/poll.runtime_pollWait(0x27a97d8, 0x72, 0x23)
	/usr/local/go/src/runtime/netpoll.go:173 +0x5e
internal/poll.(*pollDesc).wait(0xc00053b218, 0x72, 0x1ec7000, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x96
internal/poll.(*pollDesc).waitRead(0xc00053b218, 0xc0007ba000, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x45
internal/poll.(*FD).Read(0xc00053b200, 0xc0007ba000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x36a
net.(*netFD).Read(0xc00053b200, 0xc0007ba000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/fd_unix.go:202 +0x7a
net.(*conn).Read(0xc00033b890, 0xc0007ba000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:177 +0xb0
net/http.(*connReader).Read(0xc0007aa840, 0xc0007ba000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:786 +0x2c4
bufio.(*Reader).fill(0xc00076e960)
	/usr/local/go/src/bufio/bufio.go:100 +0x2d2
bufio.(*Reader).ReadSlice(0xc00076e960, 0xc0008dd50a, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/bufio/bufio.go:341 +0x3d6
bufio.(*Reader).ReadLine(0xc00076e960, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/bufio/bufio.go:370 +0x91
net/textproto.(*Reader).readLineSlice(0xc00086dad0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/textproto/reader.go:55 +0xaf
net/textproto.(*Reader).ReadLine(0xc00086dad0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/textproto/reader.go:36 +0x6f
net/http.readRequest(0xc00076e960, 0x101000, 0xc0008ccc00, 0x0, 0x0)
	/usr/local/go/src/net/http/request.go:958 +0xb0
net/http.(*conn).readRequest(0xc0001ba8c0, 0x1ecc5e0, 0xc000774800, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:966 +0x4b0
net/http.(*conn).serve(0xc0001ba8c0, 0x1ecc5e0, 0xc000774800)
	/usr/local/go/src/net/http/server.go:1788 +0xa31
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:2851 +0x7b0

goroutine 73 [IO wait]:
internal/poll.runtime_pollWait(0x27a9708, 0x72, 0x23)
	/usr/local/go/src/runtime/netpoll.go:173 +0x5e
internal/poll.(*pollDesc).wait(0xc00053b298, 0x72, 0x1ec7000, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x96
internal/poll.(*pollDesc).waitRead(0xc00053b298, 0xc0007bc000, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x45
internal/poll.(*FD).Read(0xc00053b280, 0xc0007bc000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x36a
net.(*netFD).Read(0xc00053b280, 0xc0007bc000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/fd_unix.go:202 +0x7a
net.(*conn).Read(0xc00033b898, 0xc0007bc000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:177 +0xb0
net/http.(*connReader).Read(0xc0007aaab0, 0xc0007bc000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:786 +0x2c4
bufio.(*Reader).fill(0xc00076ea20)
	/usr/local/go/src/bufio/bufio.go:100 +0x2d2
bufio.(*Reader).ReadSlice(0xc00076ea20, 0xc0007c750a, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/bufio/bufio.go:341 +0x3d6
bufio.(*Reader).ReadLine(0xc00076ea20, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/bufio/bufio.go:370 +0x91
net/textproto.(*Reader).readLineSlice(0xc00086cd50, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/textproto/reader.go:55 +0xaf
net/textproto.(*Reader).ReadLine(0xc00086cd50, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/textproto/reader.go:36 +0x6f
net/http.readRequest(0xc00076ea20, 0x101000, 0xc00074b400, 0x0, 0x0)
	/usr/local/go/src/net/http/request.go:958 +0xb0
net/http.(*conn).readRequest(0xc0001bab40, 0x1ecc5e0, 0xc000774a00, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:966 +0x4b0
net/http.(*conn).serve(0xc0001bab40, 0x1ecc5e0, 0xc000774a00)
	/usr/local/go/src/net/http/server.go:1788 +0xa31
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:2851 +0x7b0

goroutine 71 [IO wait]:
internal/poll.runtime_pollWait(0x27a98a8, 0x72, 0x23)
	/usr/local/go/src/runtime/netpoll.go:173 +0x5e
internal/poll.(*pollDesc).wait(0xc00053b198, 0x72, 0x1ec7000, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x96
internal/poll.(*pollDesc).waitRead(0xc00053b198, 0xc0007ae000, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x45
internal/poll.(*FD).Read(0xc00053b180, 0xc0007ae000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x36a
net.(*netFD).Read(0xc00053b180, 0xc0007ae000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/fd_unix.go:202 +0x7a
net.(*conn).Read(0xc00033b888, 0xc0007ae000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:177 +0xb0
net/http.(*connReader).Read(0xc0007aa5d0, 0xc0007ae000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:786 +0x2c4
bufio.(*Reader).fill(0xc00076e8a0)
	/usr/local/go/src/bufio/bufio.go:100 +0x2d2
bufio.(*Reader).ReadSlice(0xc00076e8a0, 0xc0007b950a, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/bufio/bufio.go:341 +0x3d6
bufio.(*Reader).ReadLine(0xc00076e8a0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/bufio/bufio.go:370 +0x91
net/textproto.(*Reader).readLineSlice(0xc000890030, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/textproto/reader.go:55 +0xaf
net/textproto.(*Reader).ReadLine(0xc000890030, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/textproto/reader.go:36 +0x6f
net/http.readRequest(0xc00076e8a0, 0x101000, 0xc0008de400, 0x0, 0x0)
	/usr/local/go/src/net/http/request.go:958 +0xb0
net/http.(*conn).readRequest(0xc0001ba640, 0x1ecc5e0, 0xc0007746c0, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:966 +0x4b0
net/http.(*conn).serve(0xc0001ba640, 0x1ecc5e0, 0xc0007746c0)
	/usr/local/go/src/net/http/server.go:1788 +0xa31
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:2851 +0x7b0

goroutine 97 [select]:
net/http.(*Transport).getConn(0x256f3c0, 0xc0009524b0, 0xc000331d00, 0xc000680780, 0x5, 0xc000632b60, 0x14, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/transport.go:1004 +0xa32
net/http.(*Transport).roundTrip(0x256f3c0, 0xc0008de900, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/transport.go:451 +0xd0f
net/http.(*Transport).RoundTrip(0x256f3c0, 0xc0008de900, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/roundtrip.go:17 +0x5e
net/http.send(0xc0008de900, 0x1ec5a00, 0x256f3c0, 0x0, 0x0, 0x0, 0x0, 0x1c00850, 0x0, 0x0)
	/usr/local/go/src/net/http/client.go:250 +0x4ae
net/http.(*Client).send(0x257b460, 0xc0008de900, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/client.go:174 +0x1e2
net/http.(*Client).do(0x257b460, 0xc0008de900, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/client.go:641 +0x10af
net/http.(*Client).Do(0x257b460, 0xc0008de900, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/client.go:509 +0x5e
github.com/aws/aws-sdk-go/aws/corehandlers.sendFollowRedirects(0xc000045400, 0x0, 0x0, 0x0)
	/Users/username/go/src/github.com/aws/aws-sdk-go/aws/corehandlers/handlers.go:120 +0x72
github.com/aws/aws-sdk-go/aws/corehandlers.glob..func3(0xc000045400)
	/Users/username/go/src/github.com/aws/aws-sdk-go/aws/corehandlers/handlers.go:112 +0x246
github.com/aws/aws-sdk-go/aws/request.(*HandlerList).Run(0xc0000455a8, 0xc000045400)
	/Users/username/go/src/github.com/aws/aws-sdk-go/aws/request/handlers.go:213 +0xf2
github.com/aws/aws-sdk-go/aws/request.(*Request).Send(0xc000045400, 0x0, 0x0)
	/Users/username/go/src/github.com/aws/aws-sdk-go/aws/request/request.go:493 +0x553
github.com/aws/aws-sdk-go/service/s3.(*S3).GetBucketLocation(0xc0004c2530, 0xc0004c2538, 0x0, 0x0, 0x0)
	/Users/username/go/src/github.com/aws/aws-sdk-go/service/s3/api.go:2141 +0x92
bitbucket.org/{{redacted}}(0xc000199940, 0x0)
	/Users/username/go/src/bitbucket.org/{{redacted}}:199 +0x10b
bitbucket.org/{{redacted}}(0xc000199940, 0x0, 0x0)
	/Users/username/go/src/bitbucket.org/{{redacted}}:109 +0x4e
bitbucket.org/{{redacted}}(0xc000199940, 0x0, 0x0)
	/Users/username/go/src/bitbucket.org/{{redacted}}:94 +0x37
bitbucket.org/{{redacted}}(0xc000874900)
	/Users/username/go/src/bitbucket.org/{{redacted}}:60 +0xa1
bitbucket.org/{{redacted}}(0xc000874900)
	/Users/username/go/src/bitbucket.org/{{redacted}}:54 +0x49
created by bitbucket.org/{{redacted}}
	/Users/username/go/src/bitbucket.org/{{redacted}}:45 +0x3af

goroutine 108 [chan receive]:
net/http.(*persistConn).addTLS(0xc0008fb560, 0xc000632b60, 0x10, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/transport.go:1177 +0x246
net/http.(*Transport).dialConn(0x256f3c0, 0x1ecc620, 0xc00009c008, 0xc000331d00, 0xc000680780, 0x5, 0xc000632b60, 0x14, 0x0, 0x0, ...)
	/usr/local/go/src/net/http/transport.go:1322 +0xf31
net/http.(*Transport).getConn.func4(0x256f3c0, 0x1ecc620, 0xc00009c008, 0xc0009524e0, 0xc00034ac60)
	/usr/local/go/src/net/http/transport.go:999 +0x8d
created by net/http.(*Transport).getConn
	/usr/local/go/src/net/http/transport.go:998 +0x83d
fatal error: unexpected signal during runtime execution
panic during panic
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8c96b98 pc=0x8c167e7]

runtime stack:
runtime.throw(0x1bea596, 0x2a)
	/usr/local/go/src/runtime/panic.go:608 +0x72
runtime.sigpanic()
	/usr/local/go/src/runtime/signal_unix.go:374 +0x288
fatal error: unexpected signal during runtime execution
stack trace unavailable
@ianlancetaylor ianlancetaylor changed the title macos 10.14 frequent but seemingly random crash with SIGSEGV runtime: macos 10.14 frequent but seemingly random crash with SIGSEGV Oct 9, 2018
@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Oct 9, 2018

You say that the crash is random. Is it always in crypto/x509.loadSystemRoots, or do you have different stack traces?

CC @randall77

@akamensky
Copy link
Author

@akamensky akamensky commented Oct 9, 2018

@ianlancetaylor when I said random means that it happens at random, not that it happens with random functions. The same code works fine until I just get this stack trace on the process that has been running for quite a bit, all with same compiled binary.

I did not compare stack traces so far. I will catch a couple more and report back on this if they all crash in same function or not.

@akamensky
Copy link
Author

@akamensky akamensky commented Oct 23, 2018

Sorry for the delay, confirmed that it breaks in same place for me, always same stack trace. This happens when running anything that loads SSL/TLS related things, and as noted before it does not happen consistently, but rather at random.

For now I am falling back to 1.10.4 which works fine (I guess something that has not been backported to 1.10 branch is causing this?)

@aclements
Copy link
Member

@aclements aclements commented Dec 18, 2018

This traceback looks to me like the C code invoked by crypto/x509 had a bad memory reference. Re-titling to reflect that this is most likely an issue in crypto/x509, not in the runtime.

@aclements aclements changed the title runtime: macos 10.14 frequent but seemingly random crash with SIGSEGV crypto/x509: macos 10.14 SIGSEGV in crypto/x509._Cfunc_FetchPEMRoots Dec 18, 2018
@FiloSottile
Copy link
Member

@FiloSottile FiloSottile commented Jan 2, 2019

@akamensky Can you check if this is still an issue at tip with the new macOS roots code?

@akamensky
Copy link
Author

@akamensky akamensky commented Jan 28, 2019

@FiloSottile I currently have no way to verify whether this issue is resolved or not. Meanwhile after doing completely clean install of 1.11.4 (and now 1.11.5) with removing entire $GOPATH/* and then rebuilding everything I had not seen this issue for awhile. Which possibly suggests it may have been caused by using different go compiler version / different standard library version to compile one of dependencies and then linking it with binary(?)

Since I have not seen this issue for awhile, I think okay to close (I guess can reopen if starts appearing again?)

@FiloSottile
Copy link
Member

@FiloSottile FiloSottile commented Jan 28, 2019

Thank you, indeed reopen if it happens again!

@bitfield
Copy link

@bitfield bitfield commented Apr 16, 2019

I'm seeing something very similar:

fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0x1a3108 pc=0x7fff823244dd]

runtime stack:
runtime.throw(0x14ebcb4, 0x2a)
        /usr/local/Cellar/go/1.12.4/libexec/src/runtime/panic.go:617 +0x72
runtime.sigpanic()
        /usr/local/Cellar/go/1.12.4/libexec/src/runtime/signal_unix.go:374 +0x4a9

goroutine 39 [syscall]:
runtime.cgocall(0x1001e00, 0xc000153470, 0xc0000258e8)
        /usr/local/Cellar/go/1.12.4/libexec/src/runtime/cgocall.go:128 +0x5b fp=0xc000153440 sp=0xc000153408 pc=0x1004b8b
crypto/x509._Cfunc_FetchPEMRoots(0xc0000258e0, 0xc0000258e8, 0xc000065700, 0x0)
        _cgo_gotypes.go:110 +0x4d fp=0xc000153470 sp=0xc000153440 pc=0x124003d
crypto/x509.loadSystemRoots.func1(0xc0000258e0, 0xc0000258e8, 0x20300000000000)
        /usr/local/Cellar/go/1.12.4/libexec/src/crypto/x509/root_cgo_darwin.go:281 +0x12d fp=0xc0001534b0 sp=0xc000153470 pc=0x124416d
crypto/x509.loadSystemRoots(0x0, 0x0, 0x0)
        /usr/local/Cellar/go/1.12.4/libexec/src/crypto/x509/root_cgo_darwin.go:281 +0xec fp=0xc000153580 sp=0xc0001534b0 pc=0x12401ac
crypto/x509.initSystemRoots()
        /usr/local/Cellar/go/1.12.4/libexec/src/crypto/x509/root.go:21 +0x26 fp=0xc0001535a8 sp=0xc000153580 pc=0x1233466
sync.(*Once).Do(0x18bee98, 0x14f8c08)
        /usr/local/Cellar/go/1.12.4/libexec/src/sync/once.go:44 +0xb3 fp=0xc0001535d8 sp=0xc0001535a8 pc=0x106df93
crypto/x509.systemRootsPool(...)
        /usr/local/Cellar/go/1.12.4/libexec/src/crypto/x509/root.go:16
crypto/x509.(*Certificate).Verify(0xc0001a4000, 0xc000022620, 0x13, 0xc000162f90, 0x0, 0xbf2594e467f366d8, 0x640295f, 0x18a2340, 0x0, 0x0, ...)
        /usr/local/Cellar/go/1.12.4/libexec/src/crypto/x509/verify.go:744 +0x6ca fp=0xc0001536f0 sp=0xc0001535d8 pc=0x123714a
crypto/tls.(*Conn).verifyServerCertificate(0xc0000be380, 0xc0001780a0, 0x3, 0x3, 0xead, 0x0)
        /usr/local/Cellar/go/1.12.4/libexec/src/crypto/tls/handshake_client.go:838 +0x26a fp=0xc000153888 sp=0xc0001536f0 pc=0x12690fa
crypto/tls.(*clientHandshakeState).doFullHandshake(0xc000153dc8, 0xc00009e540, 0x60)
        /usr/local/Cellar/go/1.12.4/libexec/src/crypto/tls/handshake_client.go:454 +0x1869 fp=0xc000153bc0 sp=0xc000153888 pc=0x1267249
crypto/tls.(*clientHandshakeState).handshake(0xc000153dc8, 0xc000180120, 0x0)
        /usr/local/Cellar/go/1.12.4/libexec/src/crypto/tls/handshake_client.go:399 +0x3fb fp=0xc000153cc8 sp=0xc000153bc0 pc=0x126560b
crypto/tls.(*Conn).clientHandshake(0xc0000be380, 0x0, 0x0)
        /usr/local/Cellar/go/1.12.4/libexec/src/crypto/tls/handshake_client.go:208 +0x2cd fp=0xc000153f38 sp=0xc000153cc8 pc=0x1263c5d
crypto/tls.(*Conn).Handshake(0xc0000be380, 0x0, 0x0)
        /usr/local/Cellar/go/1.12.4/libexec/src/crypto/tls/conn.go:1343 +0xef fp=0xc000153f78 sp=0xc000153f38 pc=0x126226f
net/http.(*persistConn).addTLS.func2(0x0, 0xc0000be380, 0xc0001822d0, 0xc00017a660)
        /usr/local/Cellar/go/1.12.4/libexec/src/net/http/transport.go:1190 +0x42 fp=0xc000153fc0 sp=0xc000153f78 pc=0x1315752
runtime.goexit()
        /usr/local/Cellar/go/1.12.4/libexec/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc000153fc8 sp=0xc000153fc0 pc=0x1058ec1
created by net/http.(*persistConn).addTLS
        /usr/local/Cellar/go/1.12.4/libexec/src/net/http/transport.go:1186 +0x1ab

goroutine 1 [select]:
net/http.(*Transport).getConn(0x1897fe0, 0xc000162630, 0x0, 0xc000020700, 0x5, 0xc000022620, 0x17, 0x0, 0x0, 0x0, ...)
        /usr/local/Cellar/go/1.12.4/libexec/src/net/http/transport.go:1020 +0x63d
net/http.(*Transport).roundTrip(0x1897fe0, 0xc00016a700, 0xc000162570, 0xc000024160, 0xc000024154)
        /usr/local/Cellar/go/1.12.4/libexec/src/net/http/transport.go:467 +0x6ef
net/http.(*Transport).RoundTrip(0x1897fe0, 0xc00016a700, 0x1897fe0, 0xbf2594e6e1d59b18, 0x2542e412d)
        /usr/local/Cellar/go/1.12.4/libexec/src/net/http/roundtrip.go:17 +0x35
net/http.send(0xc00016a600, 0x157b800, 0x1897fe0, 0xbf2594e6e1d59b18, 0x2542e412d, 0x18a2340, 0xc000010070, 0xbf2594e6e1d59b18, 0x1, 0x0)
        /usr/local/Cellar/go/1.12.4/libexec/src/net/http/client.go:250 +0x461
net/http.(*Client).send(0xc000162420, 0xc00016a600, 0xbf2594e6e1d59b18, 0x2542e412d, 0x18a2340, 0xc000010070, 0x0, 0x1, 0x10188ff)
        /usr/local/Cellar/go/1.12.4/libexec/src/net/http/client.go:174 +0xfb
net/http.(*Client).do(0xc000162420, 0xc00016a600, 0x0, 0x0, 0x0)
        /usr/local/Cellar/go/1.12.4/libexec/src/net/http/client.go:641 +0x279
net/http.(*Client).Do(0xc000162420, 0xc00016a600, 0x0, 0x0, 0x1)
        /usr/local/Cellar/go/1.12.4/libexec/src/net/http/client.go:509 +0x35
github.com/bitfield/uptimerobot/pkg.(*Client).MakeAPICall(0x18a1fe0, 0x14dec4d, 0x11, 0xc00016e240, 0xc000157b30, 0x0, 0x0)
        /Users/john/git/bitfield/uptimerobot/pkg/uptimerobot.go:339 +0x6e9
github.com/bitfield/uptimerobot/pkg.(*Client).GetAccountDetails(0x18a1fe0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x2, 0xc00000e140)
        /Users/john/git/bitfield/uptimerobot/pkg/uptimerobot.go:193 +0xf1
github.com/bitfield/uptimerobot/cmd.glob..func1(0x1899820, 0x18bee08, 0x0, 0x0)
        /Users/john/git/bitfield/uptimerobot/cmd/account.go:15 +0x66
github.com/spf13/cobra.(*Command).execute(0x1899820, 0x18bee08, 0x0, 0x0, 0x1899820, 0x18bee08)
        /Users/john/go/pkg/mod/github.com/spf13/cobra@v0.0.3/command.go:766 +0x2ae
github.com/spf13/cobra.(*Command).ExecuteC(0x1898ea0, 0xc000157f68, 0x13fbe1e, 0x1898ea0)
        /Users/john/go/pkg/mod/github.com/spf13/cobra@v0.0.3/command.go:852 +0x2ec
github.com/spf13/cobra.(*Command).Execute(...)
        /Users/john/go/pkg/mod/github.com/spf13/cobra@v0.0.3/command.go:800
github.com/bitfield/uptimerobot/cmd.Execute()
        /Users/john/git/bitfield/uptimerobot/cmd/root.go:28 +0x32
main.main()
        /Users/john/git/bitfield/uptimerobot/main.go:8 +0x20

goroutine 5 [select]:
net/http.setRequestCancel.func3(0x0, 0xc000162570, 0xc000178000, 0xc000024160, 0xc000168120)
        /usr/local/Cellar/go/1.12.4/libexec/src/net/http/client.go:321 +0xcf
created by net/http.setRequestCancel
        /usr/local/Cellar/go/1.12.4/libexec/src/net/http/client.go:320 +0x251

goroutine 6 [chan receive]:
net/http.(*persistConn).addTLS(0xc000180000, 0xc000022620, 0x13, 0x0, 0xc000022634, 0x3)
        /usr/local/Cellar/go/1.12.4/libexec/src/net/http/transport.go:1196 +0x1d4
net/http.(*Transport).dialConn(0x1897fe0, 0x1585140, 0xc0000d2000, 0x0, 0xc000020700, 0x5, 0xc000022620, 0x17, 0x0, 0x0, ...)
        /usr/local/Cellar/go/1.12.4/libexec/src/net/http/transport.go:1269 +0x1cde
net/http.(*Transport).getConn.func4(0x1897fe0, 0x1585140, 0xc0000d2000, 0xc000162660, 0xc000168180)
        /usr/local/Cellar/go/1.12.4/libexec/src/net/http/transport.go:1015 +0xa6
created by net/http.(*Transport).getConn
        /usr/local/Cellar/go/1.12.4/libexec/src/net/http/transport.go:1014 +0x455

This happens intermittently when running any Go program that makes network requests (including go mod download!). I noticed it happening on various 1.11 versions and now on 1.12.1 and 1.12.4.

@aclements
Copy link
Member

@aclements aclements commented Apr 16, 2019

Re-opening.

@bitfield, could you provide a little more detail? For example, go version and go env. Thanks!

@aclements aclements reopened this Apr 16, 2019
@bitfield
Copy link

@bitfield bitfield commented Apr 19, 2019

go version
go version go1.12.4 darwin/amd64

go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/john/Library/Caches/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/john/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/Cellar/go/1.12.4/libexec"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.12.4/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/john/git/paidy/terraform-aws-ecs-deployment/test/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/q3/ybqqxyh92yd0yc865zqk0vpm0000gn/T/go-build202550920=/tmp/go-build -gno-record-gcc-switches -fno-common"

Here's another one:

fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0x5418c8 pc=0x7fff823244dd]

runtime stack:
runtime.throw(0x14c66b4, 0x2a)
        /usr/local/Cellar/go/1.12.4/libexec/src/runtime/panic.go:617 +0x72
runtime.sigpanic()
        /usr/local/Cellar/go/1.12.4/libexec/src/runtime/signal_unix.go:374 +0x4a9

goroutine 55 [syscall]:
runtime.cgocall(0x1001840, 0xc000137470, 0xc00032c028)
        /usr/local/Cellar/go/1.12.4/libexec/src/runtime/cgocall.go:128 +0x5b fp=0xc000137440 sp=0xc000137408 pc=0x1004c4b
crypto/x509._Cfunc_FetchPEMRoots(0xc00032c020, 0xc00032c028, 0xc00032a100, 0x0)
        _cgo_gotypes.go:110 +0x4d fp=0xc000137470 sp=0xc000137440 pc=0x11cd37d
crypto/x509.loadSystemRoots.func1(0xc00032c020, 0xc00032c028, 0x1018c7f)
        /usr/local/Cellar/go/1.12.4/libexec/src/crypto/x509/root_cgo_darwin.go:281 +0x12d fp=0xc0001374b0 sp=0xc000137470 pc=0x11d14ad
crypto/x509.loadSystemRoots(0x0, 0x0, 0x0)
        /usr/local/Cellar/go/1.12.4/libexec/src/crypto/x509/root_cgo_darwin.go:281 +0xec fp=0xc000137580 sp=0xc0001374b0 pc=0x11cd4ec
crypto/x509.initSystemRoots()
        /usr/local/Cellar/go/1.12.4/libexec/src/crypto/x509/root.go:21 +0x26 fp=0xc0001375a8 sp=0xc000137580 pc=0x11c2006
sync.(*Once).Do(0x18a6880, 0x14d12a8)
        /usr/local/Cellar/go/1.12.4/libexec/src/sync/once.go:44 +0xb3 fp=0xc0001375d8 sp=0xc0001375a8 pc=0x1065e23
crypto/x509.systemRootsPool(...)
        /usr/local/Cellar/go/1.12.4/libexec/src/crypto/x509/root.go:16
crypto/x509.(*Certificate).Verify(0xc000318000, 0xc000246180, 0x20, 0xc000230f00, 0x0, 0xbf26926a6b079060, 0x2c8c60a752, 0x1889e80, 0x0, 0x0, ...)
        /usr/local/Cellar/go/1.12.4/libexec/src/crypto/x509/verify.go:744 +0x6ca fp=0xc0001376f0 sp=0xc0001375d8 pc=0x11c5cea
crypto/tls.(*Conn).verifyServerCertificate(0xc000236380, 0xc0000b72c0, 0x4, 0x4, 0x1324, 0x0)
        /usr/local/Cellar/go/1.12.4/libexec/src/crypto/tls/handshake_client.go:838 +0x26a fp=0xc000137888 sp=0xc0001376f0 pc=0x11f4faa
crypto/tls.(*clientHandshakeState).doFullHandshake(0xc000137dc8, 0xc000290540, 0x57)
        /usr/local/Cellar/go/1.12.4/libexec/src/crypto/tls/handshake_client.go:454 +0x1869 fp=0xc000137bc0 sp=0xc000137888 pc=0x11f30f9
crypto/tls.(*clientHandshakeState).handshake(0xc000137dc8, 0xc0002f0240, 0x0)
        /usr/local/Cellar/go/1.12.4/libexec/src/crypto/tls/handshake_client.go:399 +0x3fb fp=0xc000137cc8 sp=0xc000137bc0 pc=0x11f14bb
crypto/tls.(*Conn).clientHandshake(0xc000236380, 0x0, 0x0)
        /usr/local/Cellar/go/1.12.4/libexec/src/crypto/tls/handshake_client.go:208 +0x2cd fp=0xc000137f38 sp=0xc000137cc8 pc=0x11efb0d
crypto/tls.(*Conn).Handshake(0xc000236380, 0x0, 0x0)
        /usr/local/Cellar/go/1.12.4/libexec/src/crypto/tls/conn.go:1343 +0xef fp=0xc000137f78 sp=0xc000137f38 pc=0x11ee1df
net/http.(*persistConn).addTLS.func2(0x0, 0xc000236380, 0xc00023e230, 0xc0000b69c0)
        /usr/local/Cellar/go/1.12.4/libexec/src/net/http/transport.go:1190 +0x42 fp=0xc000137fc0 sp=0xc000137f78 pc=0x1288972
runtime.goexit()
        /usr/local/Cellar/go/1.12.4/libexec/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc000137fc8 sp=0xc000137fc0 pc=0x105d6b1
created by net/http.(*persistConn).addTLS
        /usr/local/Cellar/go/1.12.4/libexec/src/net/http/transport.go:1186 +0x1ab

goroutine 1 [chan receive]:
testing.(*T).Run(0xc000214100, 0x14b8b88, 0x11, 0x14d2478, 0x107b616)
        /usr/local/Cellar/go/1.12.4/libexec/src/testing/testing.go:917 +0x381
testing.runTests.func1(0xc000214000)
        /usr/local/Cellar/go/1.12.4/libexec/src/testing/testing.go:1157 +0x78
testing.tRunner(0xc000214000, 0xc000213e30)
        /usr/local/Cellar/go/1.12.4/libexec/src/testing/testing.go:865 +0xc0
testing.runTests(0xc0000c96e0, 0x187e390, 0x1, 0x1, 0x0)
        /usr/local/Cellar/go/1.12.4/libexec/src/testing/testing.go:1155 +0x2a9
testing.(*M).Run(0xc00013c080, 0x0)
        /usr/local/Cellar/go/1.12.4/libexec/src/testing/testing.go:1072 +0x162
main.main()
        _testmain.go:42 +0x13e

goroutine 18 [select]:
net/http.(*Transport).getConn(0x1883de0, 0xc000230360, 0x0, 0xc000246030, 0x5, 0xc000246180, 0x24, 0x0, 0x0, 0x0, ...)
        /usr/local/Cellar/go/1.12.4/libexec/src/net/http/transport.go:1020 +0x63d
net/http.(*Transport).roundTrip(0x1883de0, 0xc0002e0000, 0x10, 0xc000220900, 0xc00013b6a0)
        /usr/local/Cellar/go/1.12.4/libexec/src/net/http/transport.go:467 +0x6ef
net/http.(*Transport).RoundTrip(0x1883de0, 0xc0002e0000, 0x1883de0, 0x0, 0x0)
        /usr/local/Cellar/go/1.12.4/libexec/src/net/http/roundtrip.go:17 +0x35
net/http.send(0xc0002e0000, 0x15c2520, 0x1883de0, 0x0, 0x0, 0x0, 0xc0000b4038, 0x14d2450, 0x1, 0x0)
        /usr/local/Cellar/go/1.12.4/libexec/src/net/http/client.go:250 +0x461
net/http.(*Client).send(0x1889be0, 0xc0002e0000, 0x0, 0x0, 0x0, 0xc0000b4038, 0x0, 0x1, 0x14b7119)
        /usr/local/Cellar/go/1.12.4/libexec/src/net/http/client.go:174 +0xfb
net/http.(*Client).do(0x1889be0, 0xc0002e0000, 0x0, 0x0, 0x0)
        /usr/local/Cellar/go/1.12.4/libexec/src/net/http/client.go:641 +0x279
net/http.(*Client).Do(...)
        /usr/local/Cellar/go/1.12.4/libexec/src/net/http/client.go:509
github.com/aws/aws-sdk-go/aws/corehandlers.sendFollowRedirects(0xc0002e2000, 0x105ef30, 0xc00013bac8, 0xbf269269fc3317a0)
        /Users/john/go/pkg/mod/github.com/aws/aws-sdk-go@v1.19.10/aws/corehandlers/handlers.go:120 +0x3c
github.com/aws/aws-sdk-go/aws/corehandlers.glob..func3(0xc0002e2000)
        /Users/john/go/pkg/mod/github.com/aws/aws-sdk-go@v1.19.10/aws/corehandlers/handlers.go:112 +0x7f
github.com/aws/aws-sdk-go/aws/request.(*HandlerList).Run(0xc0002e21b8, 0xc0002e2000)
        /Users/john/go/pkg/mod/github.com/aws/aws-sdk-go@v1.19.10/aws/request/handlers.go:216 +0x98
github.com/aws/aws-sdk-go/aws/request.(*Request).sendRequest(0xc0002e2000, 0x0, 0x0)
        /Users/john/go/pkg/mod/github.com/aws/aws-sdk-go@v1.19.10/aws/request/request.go:521 +0x93
github.com/aws/aws-sdk-go/aws/request.(*Request).Send(0xc0002e2000, 0x0, 0x0)
        /Users/john/go/pkg/mod/github.com/aws/aws-sdk-go@v1.19.10/aws/request/request.go:480 +0x117
github.com/aws/aws-sdk-go/service/ecs.(*ECS).DescribeServices(0xc000266040, 0xc00022e0c0, 0xe, 0xc000266040, 0x184e3e0)
        /Users/john/go/pkg/mod/github.com/aws/aws-sdk-go@v1.19.10/service/ecs/api.go:1412 +0x4d
tmp.TestRoutedService(0xc000214100)
        /Users/john/git/paidy/terraform-aws-ecs-deployment/test/main_test.go:39 +0x227
testing.tRunner(0xc000214100, 0x14d2478)
        /usr/local/Cellar/go/1.12.4/libexec/src/testing/testing.go:865 +0xc0
created by testing.(*T).Run
        /usr/local/Cellar/go/1.12.4/libexec/src/testing/testing.go:916 +0x35a

goroutine 50 [chan receive]:
net/http.(*persistConn).addTLS(0xc0002f0120, 0xc000246180, 0x20, 0x0, 0xc0002461a1, 0x3)
        /usr/local/Cellar/go/1.12.4/libexec/src/net/http/transport.go:1196 +0x1d4
net/http.(*Transport).dialConn(0x1883de0, 0x15c9fc0, 0xc0000ca000, 0x0, 0xc000246030, 0x5, 0xc000246180, 0x24, 0x0, 0x0, ...)
        /usr/local/Cellar/go/1.12.4/libexec/src/net/http/transport.go:1269 +0x1cde
net/http.(*Transport).getConn.func4(0x1883de0, 0x15c9fc0, 0xc0000ca000, 0xc000230390, 0xc000290120)
        /usr/local/Cellar/go/1.12.4/libexec/src/net/http/transport.go:1015 +0xa6
created by net/http.(*Transport).getConn
        /usr/local/Cellar/go/1.12.4/libexec/src/net/http/transport.go:1014 +0x455

I'd say this happens about one or two times in ten when running any Go binary that makes network calls.

@aclements
Copy link
Member

@aclements aclements commented Apr 19, 2019

@bitfield, is it possible to run this under gdb or lldb to see where the SIGSEGV is happening in the C code?

@bitfield
Copy link

@bitfield bitfield commented Apr 19, 2019

Sure, just let me know what command I need to run to do this.

@aclements
Copy link
Member

@aclements aclements commented Apr 19, 2019

In gdb, just start your program under gdb with "gdb --args <command...>" and then type "run". It will stop when there's a segfault. At that point the output of running "backtrace" and "info locals" is probably the most useful starting point.

I'm not an lldb user, so I'm not sure how to do this with lldb. @dr2chase?

@bitfield
Copy link

@bitfield bitfield commented Apr 19, 2019

lldb instructions would be useful, because I have that, but I don't have gdb. I could install gdb, but I'd have to install gcc...

Homebrew is like a really boring text adventure.

INSTALL MACVIM
You need to install Python first.

INSTALL PYTHON
You need to update your Xcode to do that!

PUT THE JUNK MAIL ON THE SATCHEL
You don't have that item.

BREW DOCTOR
You were eaten by a grue!
https://twitter.com/bitfield/status/986967919094648832

@aclements
Copy link
Member

@aclements aclements commented Apr 19, 2019

It appears it's basically the same in lldb:

lldb -- <command...>
run
# wait for it to segfault
bt
@bitfield
Copy link

@bitfield bitfield commented Apr 20, 2019

Thanks. Here's the simplest program that will demonstrate the crash:

package main

import (
	"fmt"
	"net/http"
)

func main() {
	res, _ := http.Get("https://postman-echo.com/get")
	fmt.Println(res.Status)
}

And here's the lldb session:

lldb -- ./cryptocrash 
(lldb) target create "./cryptocrash"
Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "/usr/local/Cellar/python@2/2.7.15_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 52, in <module>
    import weakref
  File "/usr/local/Cellar/python@2/2.7.15_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/weakref.py", line 14, in <module>
    from _weakref import (
ImportError: cannot import name _remove_dead_weakref
Current executable set to './cryptocrash' (x86_64).
(lldb) run
Process 59576 launched: './cryptocrash' (x86_64)
200 OK
Process 59576 exited with status = 0 (0x00000000) 
(lldb) run
Process 59586 launched: './cryptocrash' (x86_64)
200 OK
Process 59586 exited with status = 0 (0x00000000) 
(lldb) run
Process 59590 launched: './cryptocrash' (x86_64)
Process 59590 stopped
* thread #2: tid = 0x1286d12, 0x00007fff823244dd, stop reason = EXC_BAD_ACCESS (code=1, address=0x5a0128)
    frame #0: 0x00007fff823244dd
->  0x7fff823244dd: andl   0x18(%r11), %r10d
    0x7fff823244e1: shlq   $0x4, %r10
    0x7fff823244e5: addq   0x10(%r11), %r10
    0x7fff823244e9: cmpq   (%r10), %rsi
(lldb) bt
* thread #2: tid = 0x1286d12, 0x00007fff823244dd, stop reason = EXC_BAD_ACCESS (code=1, address=0x5a0128)
  * frame #0: 0x00007fff823244dd
    frame #1: 0x0000000001001472 cryptocrash`FetchPEMRoots + 770
    frame #2: 0x0000000001001865 cryptocrash`_cgo_595ff63b07d2_Cfunc_FetchPEMRoots + 37
    frame #3: 0x0000000001057ba0 cryptocrash`runtime.asmcgocall + 112
    frame #4: 0x00000000010311d0 cryptocrash`runtime.startTheWorldWithSema + 624
    frame #5: 0x00000000011636fd cryptocrash`crypto/x509._Cfunc_FetchPEMRoots + 77
    frame #6: 0x000000000116782d cryptocrash`crypto/x509.loadSystemRoots.func1 + 301
    frame #7: 0x000000000116386c cryptocrash`crypto/x509.loadSystemRoots + 236
    frame #8: 0x00000000011585b6 cryptocrash`crypto/x509.initSystemRoots + 38
    frame #9: 0x000000000106a203 cryptocrash`sync.(*Once).Do + 179
    frame #10: 0x000000000115c29a cryptocrash`crypto/x509.(*Certificate).Verify + 1738
    frame #11: 0x000000000118b32a cryptocrash`crypto/tls.(*Conn).verifyServerCertificate + 618
    frame #12: 0x0000000001189479 cryptocrash`crypto/tls.(*clientHandshakeState).doFullHandshake + 6249
    frame #13: 0x000000000118783b cryptocrash`crypto/tls.(*clientHandshakeState).handshake + 1019
    frame #14: 0x0000000001185e8d cryptocrash`crypto/tls.(*Conn).clientHandshake + 717
    frame #15: 0x000000000118455f cryptocrash`crypto/tls.(*Conn).Handshake + 239
    frame #16: 0x00000000012230c2 cryptocrash`net/http.(*persistConn).addTLS.func2 + 66
    frame #17: 0x0000000001058401 cryptocrash`runtime.goexit + 1
    frame #18: 0x000000000116782d cryptocrash`crypto/x509.loadSystemRoots.func1 + 301
    frame #19: 0x000000000116386c cryptocrash`crypto/x509.loadSystemRoots + 236
    frame #20: 0x00000000011585b6 cryptocrash`crypto/x509.initSystemRoots + 38
    frame #21: 0x000000000106a203 cryptocrash`sync.(*Once).Do + 179
    frame #22: 0x000000000115c29a cryptocrash`crypto/x509.(*Certificate).Verify + 1738
    frame #23: 0x000000000118b32a cryptocrash`crypto/tls.(*Conn).verifyServerCertificate + 618
    frame #24: 0x0000000001189479 cryptocrash`crypto/tls.(*clientHandshakeState).doFullHandshake + 6249
    frame #25: 0x000000000118783b cryptocrash`crypto/tls.(*clientHandshakeState).handshake + 1019
    frame #26: 0x0000000001185e8d cryptocrash`crypto/tls.(*Conn).clientHandshake + 717
    frame #27: 0x000000000118455f cryptocrash`crypto/tls.(*Conn).Handshake + 239
    frame #28: 0x00000000012230c2 cryptocrash`net/http.(*persistConn).addTLS.func2 + 66
    frame #29: 0x0000000001058401 cryptocrash`runtime.goexit + 1
    frame #30: 0x000000000116386c cryptocrash`crypto/x509.loadSystemRoots + 236
    frame #31: 0x00000000011585b6 cryptocrash`crypto/x509.initSystemRoots + 38
    frame #32: 0x000000000106a203 cryptocrash`sync.(*Once).Do + 179
    frame #33: 0x000000000115c29a cryptocrash`crypto/x509.(*Certificate).Verify + 1738
    frame #34: 0x000000000118b32a cryptocrash`crypto/tls.(*Conn).verifyServerCertificate + 618
    frame #35: 0x0000000001189479 cryptocrash`crypto/tls.(*clientHandshakeState).doFullHandshake + 6249
    frame #36: 0x000000000118783b cryptocrash`crypto/tls.(*clientHandshakeState).handshake + 1019
    frame #37: 0x0000000001185e8d cryptocrash`crypto/tls.(*Conn).clientHandshake + 717
    frame #38: 0x000000000118455f cryptocrash`crypto/tls.(*Conn).Handshake + 239
    frame #39: 0x00000000012230c2 cryptocrash`net/http.(*persistConn).addTLS.func2 + 66
    frame #40: 0x0000000001058401 cryptocrash`runtime.goexit + 1
    frame #41: 0x00000000011585b6 cryptocrash`crypto/x509.initSystemRoots + 38
    frame #42: 0x000000000106a203 cryptocrash`sync.(*Once).Do + 179
    frame #43: 0x000000000115c29a cryptocrash`crypto/x509.(*Certificate).Verify + 1738
    frame #44: 0x000000000118b32a cryptocrash`crypto/tls.(*Conn).verifyServerCertificate + 618
    frame #45: 0x0000000001189479 cryptocrash`crypto/tls.(*clientHandshakeState).doFullHandshake + 6249
    frame #46: 0x000000000118783b cryptocrash`crypto/tls.(*clientHandshakeState).handshake + 1019
    frame #47: 0x0000000001185e8d cryptocrash`crypto/tls.(*Conn).clientHandshake + 717
    frame #48: 0x000000000118455f cryptocrash`crypto/tls.(*Conn).Handshake + 239
    frame #49: 0x00000000012230c2 cryptocrash`net/http.(*persistConn).addTLS.func2 + 66
    frame #50: 0x0000000001058401 cryptocrash`runtime.goexit + 1
    frame #51: 0x000000000106a203 cryptocrash`sync.(*Once).Do + 179
    frame #52: 0x000000000115c29a cryptocrash`crypto/x509.(*Certificate).Verify + 1738
    frame #53: 0x000000000118b32a cryptocrash`crypto/tls.(*Conn).verifyServerCertificate + 618
    frame #54: 0x0000000001189479 cryptocrash`crypto/tls.(*clientHandshakeState).doFullHandshake + 6249
    frame #55: 0x000000000118783b cryptocrash`crypto/tls.(*clientHandshakeState).handshake + 1019
    frame #56: 0x0000000001185e8d cryptocrash`crypto/tls.(*Conn).clientHandshake + 717
    frame #57: 0x000000000118455f cryptocrash`crypto/tls.(*Conn).Handshake + 239
    frame #58: 0x00000000012230c2 cryptocrash`net/http.(*persistConn).addTLS.func2 + 66
    frame #59: 0x0000000001058401 cryptocrash`runtime.goexit + 1
    frame #60: 0x000000000115c29a cryptocrash`crypto/x509.(*Certificate).Verify + 1738
    frame #61: 0x000000000118b32a cryptocrash`crypto/tls.(*Conn).verifyServerCertificate + 618
    frame #62: 0x0000000001189479 cryptocrash`crypto/tls.(*clientHandshakeState).doFullHandshake + 6249
    frame #63: 0x000000000118783b cryptocrash`crypto/tls.(*clientHandshakeState).handshake + 1019
    frame #64: 0x0000000001185e8d cryptocrash`crypto/tls.(*Conn).clientHandshake + 717
    frame #65: 0x000000000118455f cryptocrash`crypto/tls.(*Conn).Handshake + 239
    frame #66: 0x00000000012230c2 cryptocrash`net/http.(*persistConn).addTLS.func2 + 66
    frame #67: 0x0000000001058401 cryptocrash`runtime.goexit + 1
    frame #68: 0x000000000118b32a cryptocrash`crypto/tls.(*Conn).verifyServerCertificate + 618
    frame #69: 0x0000000001189479 cryptocrash`crypto/tls.(*clientHandshakeState).doFullHandshake + 6249
    frame #70: 0x000000000118783b cryptocrash`crypto/tls.(*clientHandshakeState).handshake + 1019
    frame #71: 0x0000000001185e8d cryptocrash`crypto/tls.(*Conn).clientHandshake + 717
    frame #72: 0x000000000118455f cryptocrash`crypto/tls.(*Conn).Handshake + 239
    frame #73: 0x00000000012230c2 cryptocrash`net/http.(*persistConn).addTLS.func2 + 66
    frame #74: 0x0000000001058401 cryptocrash`runtime.goexit + 1
    frame #75: 0x0000000001189479 cryptocrash`crypto/tls.(*clientHandshakeState).doFullHandshake + 6249
    frame #76: 0x000000000118783b cryptocrash`crypto/tls.(*clientHandshakeState).handshake + 1019
    frame #77: 0x0000000001185e8d cryptocrash`crypto/tls.(*Conn).clientHandshake + 717
    frame #78: 0x000000000118455f cryptocrash`crypto/tls.(*Conn).Handshake + 239
    frame #79: 0x00000000012230c2 cryptocrash`net/http.(*persistConn).addTLS.func2 + 66
    frame #80: 0x0000000001058401 cryptocrash`runtime.goexit + 1
    frame #81: 0x000000000118783b cryptocrash`crypto/tls.(*clientHandshakeState).handshake + 1019
    frame #82: 0x0000000001185e8d cryptocrash`crypto/tls.(*Conn).clientHandshake + 717
    frame #83: 0x000000000118455f cryptocrash`crypto/tls.(*Conn).Handshake + 239
    frame #84: 0x00000000012230c2 cryptocrash`net/http.(*persistConn).addTLS.func2 + 66
    frame #85: 0x0000000001058401 cryptocrash`runtime.goexit + 1
    frame #86: 0x0000000001185e8d cryptocrash`crypto/tls.(*Conn).clientHandshake + 717
    frame #87: 0x000000000118455f cryptocrash`crypto/tls.(*Conn).Handshake + 239
    frame #88: 0x00000000012230c2 cryptocrash`net/http.(*persistConn).addTLS.func2 + 66
    frame #89: 0x0000000001058401 cryptocrash`runtime.goexit + 1
    frame #90: 0x000000000118455f cryptocrash`crypto/tls.(*Conn).Handshake + 239
    frame #91: 0x00000000012230c2 cryptocrash`net/http.(*persistConn).addTLS.func2 + 66
    frame #92: 0x0000000001058401 cryptocrash`runtime.goexit + 1
    frame #93: 0x00000000012230c2 cryptocrash`net/http.(*persistConn).addTLS.func2 + 66
    frame #94: 0x0000000001058401 cryptocrash`runtime.goexit + 1
    frame #95: 0x0000000001058401 cryptocrash`runtime.goexit + 1
(lldb) 
@bitfield
Copy link

@bitfield bitfield commented Apr 21, 2019

Also, just like OP, I've had this problem on every 1.11 and 1.12 version, and it was fine before that. My Go installation is Homebrew, and I'm working outside GOPATH.

@dr2chase
Copy link
Contributor

@dr2chase dr2chase commented Apr 22, 2019

All reports, it's good to reconfirm which version of Darwin it is, and maybe which version of XCode, too. I just did 200 repetitions of the tiny test program with no error.

Darwin version is 10.14.4 (Apple glyph -> "About this Mac")

cc --version
Apple LLVM version 10.0.1 (clang-1001.0.46.3)
Target: x86_64-apple-darwin18.5.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
@aclements
Copy link
Member

@aclements aclements commented Apr 24, 2019

@bitfield, could you dump the disassembly of FetchPEMRoots so we can see what call it's failing on? I think it should work to open your cryptocrash binary in lldb and run disassemble --name FetchPEMRoots. I don't think you even need to run the binary, but if you do it should probably work to type b main.main and run. You certainly shouldn't need to reproduce the crash to get this.

@aclements
Copy link
Member

@aclements aclements commented Apr 24, 2019

Actually, the output of go tool objdump -s FetchPEMRoots <binary> might be even better. If that works, paste that output. Otherwise paste the lldb output. Thanks!

@bitfield
Copy link

@bitfield bitfield commented Apr 24, 2019

No worries!

code-stdin-luj.txt

@aclements
Copy link
Member

@aclements aclements commented Apr 24, 2019

sigh Somehow objdump failed to symbolize that. The function that's crashing is at PC 0x1320bfa (called from PC 0x100146d), but I don't know what that is. Could you also paste go tool nm -n <binary>?

@bitfield
Copy link

@bitfield bitfield commented May 21, 2019

Any news on this? It's frustrating to get constant crashes in any Go program that accesses the network.

@FiloSottile
Copy link
Member

@FiloSottile FiloSottile commented May 21, 2019

I matched the unsybolized listing above to a binary I have symbols for, and that's a symbol stub for CFEqual.

  :-1			0x100143a		48c745d000000000	MOVQ $0x0, -0x30(BP)
  :-1			0x1001442		4c89f7			MOVQ R14, DI
  :-1			0x1001445		488b7580		MOVQ -0x80(BP), SI
  :-1			0x1001449		488d55d0		LEAQ -0x30(BP), DX
  :-1			0x100144d		e87ef73100		CALL 0x1320bd0
  :-1			0x1001452		4188c4			MOVL AL, R12
  :-1			0x1001455		4c89f7			MOVQ R14, DI
  :-1			0x1001458		e88bf73100		CALL 0x1320be8
  :-1			0x100145d		4584e4			TESTL R12, R12
  :-1			0x1001460		7433			JE 0x1001495
  :-1			0x1001462		488b7dd0		MOVQ -0x30(BP), DI
  :-1			0x1001466		488bb560ffffff		MOVQ 0xffffff60(BP), SI
  :-1			0x100146d		e888f73100		CALL 0x1320bfa
  :-1			0x1001472		84c0			TESTL AL, AL
mkcert[0x1001317] <+679>:  movq   $0x0, -0x30(%rbp)
mkcert[0x100131f] <+687>:  movq   %r14, %rdi
mkcert[0x1001322] <+690>:  movq   -0xb8(%rbp), %rsi
mkcert[0x1001329] <+697>:  leaq   -0x30(%rbp), %rdx
mkcert[0x100132d] <+701>:  callq  0x1276078                 ; symbol stub for: CFDictionaryGetValueIfPresent
mkcert[0x1001332] <+706>:  movl   %eax, %ebx
mkcert[0x1001334] <+708>:  movq   %r14, %rdi
mkcert[0x1001337] <+711>:  callq  0x127609c                 ; symbol stub for: CFRelease
mkcert[0x100133c] <+716>:  testb  %bl, %bl
mkcert[0x100133e] <+718>:  je     0x1001370                 ; <+768>
mkcert[0x1001340] <+720>:  movq   -0x30(%rbp), %rdi
mkcert[0x1001344] <+724>:  movq   -0xb0(%rbp), %rsi
mkcert[0x100134b] <+731>:  callq  0x12760a2                 ; symbol stub for: CFEqual
mkcert[0x1001350] <+736>:  testb  %al, %al

Judging from the calls around it, it's this function.

static bool isSSLPolicy(SecPolicyRef policyRef) {
	if (!policyRef) {
		return false;
	}
	CFDictionaryRef properties = SecPolicyCopyProperties(policyRef);
	if (properties == NULL) {
		return false;
	}
	CFTypeRef value = NULL;
	if (CFDictionaryGetValueIfPresent(properties, kSecPolicyOid, (const void **)&value)) {
		CFRelease(properties);
		return CFEqual(value, kSecPolicyAppleSSL);
	}
	CFRelease(properties);
	return false;
}

I'll go look at the docs of CFDictionaryGetValueIfPresent and I'm sure I'll learn how we are using it wrong.

@FiloSottile
Copy link
Member

@FiloSottile FiloSottile commented May 21, 2019

Yup, we are using it wrong! CFDictionaryGetValueIfPresent follows The Get Rule so it does not own value, and when properties gets released, if it's the only owner of value, the latter gets freed. Then CFEqual crashes.

https://developer.apple.com/library/archive/documentation/CoreFoundation/Conceptual/CFMemoryMgmt/Concepts/Ownership.html

I'll send a CL.

@gopherbot
Copy link

@gopherbot gopherbot commented May 22, 2019

Change https://golang.org/cl/178537 mentions this issue: crypto/x509: fix value ownership in isSSLPolicy on macOS

@gopherbot gopherbot closed this in a3d4655 May 22, 2019
@FiloSottile
Copy link
Member

@FiloSottile FiloSottile commented May 22, 2019

This should be now fixed at tip. Please test it with https://golang.org/dl/gotip and report back. If it works, I am going to file this for cherry-picking.

$ go get golang.org/dl/gotip
$ gotip download
$ GODEBUG=x509roots=1 gotip test -v -run TestSystemRoots crypto/x509
$ gotip run [YOUR_PROGRAM]
@bitfield
Copy link

@bitfield bitfield commented May 23, 2019

Woohoo! It works! Thanks @FiloSottile!

@FiloSottile
Copy link
Member

@FiloSottile FiloSottile commented May 28, 2019

@gopherbot please open backport issues for https://golang.org/cl/178537.

This fixes a crashing bug with no known workaround for certain macOS environments.

CL 178537 is very minimal and fit for backporting. (The rest of the chain, and CL 178539 in particular, are more speculative and only fix unrecognized roots for which there is a manual workaround, so let's not backport those.)

I feel like we should backport to both 1.11 and 1.12, since without this it's impossible to use 1.11 on certain macOS systems. (Although I guess using the next 1.12 point release could count as a "workaround"?)

@gopherbot
Copy link

@gopherbot gopherbot commented May 28, 2019

Backport issue(s) opened: #32281 (for 1.11), #32282 (for 1.12).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://golang.org/wiki/MinorReleases.

@gopherbot
Copy link

@gopherbot gopherbot commented May 29, 2019

Change https://golang.org/cl/179339 mentions this issue: [release-branch.go1.12] crypto/x509: fix value ownership in isSSLPolicy on macOS

@gopherbot
Copy link

@gopherbot gopherbot commented May 29, 2019

Change https://golang.org/cl/179340 mentions this issue: [release-branch.go1.11] crypto/x509: fix value ownership in isSSLPolicy on macOS

@bitfield
Copy link

@bitfield bitfield commented May 31, 2019

without this it's impossible to use 1.11 on certain macOS systems

Not impossible, just intensely annoying.

gopherbot pushed a commit that referenced this issue Jun 7, 2019
…cy on macOS

CFDictionaryGetValueIfPresent does not take ownership of the value, so
releasing the properties dictionary before passing the value to CFEqual
can crash. Not really clear why this works most of the time.

See https://developer.apple.com/library/archive/documentation/CoreFoundation/Conceptual/CFMemoryMgmt/Concepts/Ownership.html

Fixes #32281
Updates #28092
Updates #30763

Change-Id: I5ee7ca276b753a48abc3aedfb78b8af68b448dd4
Reviewed-on: https://go-review.googlesource.com/c/go/+/178537
Reviewed-by: Adam Langley <agl@golang.org>
(cherry picked from commit a3d4655)
Reviewed-on: https://go-review.googlesource.com/c/go/+/179340
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
gopherbot pushed a commit that referenced this issue Jun 7, 2019
…cy on macOS

CFDictionaryGetValueIfPresent does not take ownership of the value, so
releasing the properties dictionary before passing the value to CFEqual
can crash. Not really clear why this works most of the time.

See https://developer.apple.com/library/archive/documentation/CoreFoundation/Conceptual/CFMemoryMgmt/Concepts/Ownership.html

Fixes #32282
Updates #28092
Updates #30763

Change-Id: I5ee7ca276b753a48abc3aedfb78b8af68b448dd4
Reviewed-on: https://go-review.googlesource.com/c/go/+/178537
Reviewed-by: Adam Langley <agl@golang.org>
(cherry picked from commit a3d4655)
Reviewed-on: https://go-review.googlesource.com/c/go/+/179339
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
@gopherbot
Copy link

@gopherbot gopherbot commented Apr 14, 2020

Change https://golang.org/cl/227037 mentions this issue: crypto/x509: use Security.framework without cgo for roots on macOS

gopherbot pushed a commit that referenced this issue May 7, 2020
+----------------------------------------------------------------------+
| Hello, if you are reading this and run macOS, please test this code: |
|                                                                      |
| $ GO111MODULE=on go get golang.org/dl/gotip@latest                   |
| $ gotip download                                              |
| $ GODEBUG=x509roots=1 gotip test crypto/x509 -v -run TestSystemRoots |
+----------------------------------------------------------------------+

We currently have two code paths to extract system roots on macOS: one
uses cgo to invoke a maze of Security.framework APIs; the other is a
horrible fallback that runs "/usr/bin/security verify-cert" on every
root that has custom policies to check if it's trusted for SSL.

The fallback is not only terrifying because it shells out to a binary,
but also because it lets in certificates that are not trusted roots but
are signed by trusted roots, and because it applies some filters (EKUs
and expiration) only to roots with custom policies, as the others are
not passed to verify-cert. The other code path, of course, requires cgo,
so can't be used when cross-compiling and involves a large ball of C.

It's all a mess, and it broke oh-so-many times (#14514, #16532, #19436,
 #20990, #21416, #24437, #24652, #25649, #26073, #27958, #28025, #28092,
 #29497, #30471, #30672, #30763, #30889, #32891, #38215, #38365, ...).

Since macOS does not have a stable syscall ABI, we already dynamically
link and invoke libSystem.dylib regardless of cgo availability (#17490).

How that works is that functions in package syscall (like syscall.Open)
take the address of assembly trampolines (like libc_open_trampoline)
that jump to symbols imported with cgo_import_dynamic (like libc_open),
and pass them along with arguments to syscall.syscall (which is
implemented as runtime.syscall_syscall). syscall_syscall informs the
scheduler and profiler, and then uses asmcgocall to switch to a system
stack and invoke runtime.syscall. The latter is an assembly trampoline
that unpacks the Go ABI arguments passed to syscall.syscall, finally
calls the remote function, and puts the return value on the Go stack.
(This last bit is the part that cgo compiles from a C wrapper.)

We can do something similar to link and invoke Security.framework!

The one difference is that runtime.syscall and friends check errors
based on the errno convention, which Security doesn't follow, so I added
runtime.syscallNoErr which just skips interpreting the return value.
We only need a variant with six arguments because the calling convention
is register-based, and extra arguments simply zero out some registers.

That's plumbed through as crypto/x509/internal/macOS.syscall. The rest
of that package is a set of wrappers for Security.framework and Core
Foundation functions, like syscall is for libSystem. In theory, as long
as macOS respects ABI backwards compatibility (a.k.a. as long as
binaries built for a previous OS version keep running) this should be
stable, as the final result is not different from what a C compiler
would make. (One exception might be dictionary key strings, which we
make our own copy of instead of using the dynamic symbol. If they change
the value of those strings things might break. But why would they.)

Finally, I rewrote the crypto/x509 cgo logic in Go using those wrappers.
It works! I tried to make it match 1:1 the old logic, so that
root_darwin_amd64.go can be reviewed by comparing it to
root_cgo_darwin_amd64.go. The only difference is that we do proper error
handling now, and assume that if there is no error the return values are
there, while before we'd just check for nil pointers and move on.

I kept the cgo logic to help with review and testing, but we should
delete it once we are confident the new code works.

The nocgo logic is gone and we shall never speak of it again.

Fixes #32604
Fixes #19561
Fixes #38365
Awakens Cthulhu

Change-Id: Id850962bad667f71e3af594bdfebbbb1edfbcbb4
Reviewed-on: https://go-review.googlesource.com/c/go/+/227037
Reviewed-by: Katie Hockman <katie@golang.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
7 participants
You can’t perform that action at this time.