Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

crypto/x509: macos 10.14 SIGSEGV in crypto/x509._Cfunc_FetchPEMRoots #28092

Open
akamensky opened this Issue Oct 9, 2018 · 19 comments

Comments

Projects
None yet
6 participants
@akamensky
Copy link

akamensky commented Oct 9, 2018

What version of Go are you using (go version)?

go version go1.11.1 darwin/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/username/Library/Caches/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/username/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/j6/vhm757px2zz3jgvxy1w3gfzr0000gn/T/go-build447402874=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

Just running (and debugging) the code that was working perfectly fine on 1.10 (since 1.11 release a lot of weird issues like debugger cannot kill process sometimes, and runaway CPU usage randomly etc etc)

What did you expect to see?

No crashing

What did you see instead?

fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8c72e3e pc=0x8c57013]

runtime stack:
runtime.throw(0x1bea596, 0x2a)
	/usr/local/go/src/runtime/panic.go:608 +0x72
runtime.sigpanic()
	/usr/local/go/src/runtime/signal_unix.go:374 +0x288

goroutine 130 [syscall]:
runtime.cgocall(0x1001ba0, 0xc00071aa30, 0x1be9723)
	/usr/local/go/src/runtime/cgocall.go:128 +0x5e fp=0xc00071a9f0 sp=0xc00071a9b8 pc=0x1004afe
crypto/x509._Cfunc_FetchPEMRoots(0xc000193ef8, 0xc000193f20, 0xc000000000)
	_cgo_gotypes.go:108 +0x4d fp=0xc00071aa30 sp=0xc00071a9f0 pc=0x128c7cd
crypto/x509.loadSystemRoots.func1(0xc000193ef8, 0xc000193f20, 0x0)
	/usr/local/go/src/crypto/x509/root_cgo_darwin.go:172 +0x1b8 fp=0xc00071aac0 sp=0xc00071aa30 pc=0x1292148
crypto/x509.loadSystemRoots(0x0, 0x0, 0x0)
	/usr/local/go/src/crypto/x509/root_cgo_darwin.go:172 +0xbf fp=0xc00071abb0 sp=0xc00071aac0 pc=0x128c98f
crypto/x509.initSystemRoots()
	/usr/local/go/src/crypto/x509/root.go:21 +0x37 fp=0xc00071abf0 sp=0xc00071abb0 pc=0x1272227
sync.(*Once).Do(0x25992d8, 0x1bff2c8)
	/usr/local/go/src/sync/once.go:44 +0xd1 fp=0xc00071ac28 sp=0xc00071abf0 pc=0x107e511
crypto/x509.systemRootsPool(0x0)
	/usr/local/go/src/crypto/x509/root.go:16 +0x42 fp=0xc00071ac48 sp=0xc00071ac28 pc=0x12721d2
crypto/x509.(*Certificate).Verify(0xc00015e100, 0xc000632b60, 0x10, 0xc0007931a0, 0x0, 0xbee74173c72b9df8, 0xc24227121, 0x257b7a0, 0x0, 0x0, ...)
	/usr/local/go/src/crypto/x509/verify.go:746 +0xb61 fp=0xc00071b030 sp=0xc00071ac48 pc=0x1278901
crypto/tls.(*clientHandshakeState).doFullHandshake(0xc00071be58, 0x0, 0x0)
	/usr/local/go/src/crypto/tls/handshake_client.go:340 +0x28a1 fp=0xc00071b988 sp=0xc00071b030 pc=0x12bd2f1
crypto/tls.(*clientHandshakeState).handshake(0xc00071be58, 0x0, 0x0)
	/usr/local/go/src/crypto/tls/handshake_client.go:247 +0xa01 fp=0xc00071bcd0 sp=0xc00071b988 pc=0x12ba291
crypto/tls.(*Conn).clientHandshake(0xc0000a3180, 0x0, 0x0)
	/usr/local/go/src/crypto/tls/handshake_client.go:168 +0x7ab fp=0xc00071bf10 sp=0xc00071bcd0 pc=0x12b96cb
crypto/tls.(*Conn).Handshake(0xc0000a3180, 0x0, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:1272 +0x14d fp=0xc00071bf78 sp=0xc00071bf10 pc=0x12b7a7d
net/http.(*persistConn).addTLS.func2(0x0, 0xc0000a3180, 0xc00036b810, 0xc0004cdbc0)
	/usr/local/go/src/net/http/transport.go:1171 +0x61 fp=0xc00071bfc0 sp=0xc00071bf78 pc=0x13ffac1
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc00071bfc8 sp=0xc00071bfc0 pc=0x105f961
created by net/http.(*persistConn).addTLS
	/usr/local/go/src/net/http/transport.go:1167 +0x21d

goroutine 1 [IO wait]:
internal/poll.runtime_pollWait(0x27a9e58, 0x72, 0x8)
	/usr/local/go/src/runtime/netpoll.go:173 +0x5e
internal/poll.(*pollDesc).wait(0xc00053b698, 0x72, 0x2527800, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x96
internal/poll.(*pollDesc).waitRead(0xc00053b698, 0xffffffffffffff00, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x45
internal/poll.(*FD).Accept(0xc00053b680, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:384 +0x383
net.(*netFD).accept(0xc00053b680, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/fd_unix.go:238 +0x7d
net.(*TCPListener).accept(0xc0004c22f8, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/tcpsock_posix.go:139 +0x56
net.(*TCPListener).AcceptTCP(0xc0004c22f8, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/tcpsock.go:247 +0x79
net/http.tcpKeepAliveListener.Accept(0xc0004c22f8, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:3232 +0x53
net/http.(*Server).Serve(0xc000335c70, 0x1ecc120, 0xc0004c22f8, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:2826 +0x350
net/http.(*Server).ListenAndServe(0xc000335c70, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:2764 +0x1ed
net/http.ListenAndServe(0x1bd1719, 0xe, 0x1ec6b60, 0xc00033ed00, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:3004 +0xd6
main.main()
	/Users/username/go/src/bitbucket.org/{{redacted}}/main.go:39 +0x43f

goroutine 7 [chan receive]:
github.com/go-redis/redis/internal/pool.(*ConnPool).reaper(0xc00013a160, 0xdf8475800)
	/Users/username/go/src/github.com/go-redis/redis/internal/pool/pool.go:449 +0xb4
created by github.com/go-redis/redis/internal/pool.NewConnPool
	/Users/username/go/src/github.com/go-redis/redis/internal/pool/pool.go:99 +0x2a3

goroutine 80 [chan receive]:
github.com/go-redis/redis/internal/pool.(*ConnPool).reaper(0xc000377a20, 0xdf8475800)
	/Users/username/go/src/github.com/go-redis/redis/internal/pool/pool.go:449 +0xb4
created by github.com/go-redis/redis/internal/pool.NewConnPool
	/Users/username/go/src/github.com/go-redis/redis/internal/pool/pool.go:99 +0x2a3

goroutine 51 [IO wait]:
internal/poll.runtime_pollWait(0x27a9d88, 0x72, 0x23)
	/usr/local/go/src/runtime/netpoll.go:173 +0x5e
internal/poll.(*pollDesc).wait(0xc00053b718, 0x72, 0x1ec7000, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x96
internal/poll.(*pollDesc).waitRead(0xc00053b718, 0xc000177000, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x45
internal/poll.(*FD).Read(0xc00053b700, 0xc000177000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x36a
net.(*netFD).Read(0xc00053b700, 0xc000177000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/fd_unix.go:202 +0x7a
net.(*conn).Read(0xc0004c2308, 0xc000177000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:177 +0xb0
net/http.(*connReader).Read(0xc0001b5140, 0xc000177000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:786 +0x2c4
bufio.(*Reader).fill(0xc0000a8840)
	/usr/local/go/src/bufio/bufio.go:100 +0x2d2
bufio.(*Reader).ReadSlice(0xc0000a8840, 0xc0006a550a, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/bufio/bufio.go:341 +0x3d6
bufio.(*Reader).ReadLine(0xc0000a8840, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/bufio/bufio.go:370 +0x91
net/textproto.(*Reader).readLineSlice(0xc0006f2de0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/textproto/reader.go:55 +0xaf
net/textproto.(*Reader).ReadLine(0xc0006f2de0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/textproto/reader.go:36 +0x6f
net/http.readRequest(0xc0000a8840, 0x101000, 0xc0008cd100, 0x0, 0x0)
	/usr/local/go/src/net/http/request.go:958 +0xb0
net/http.(*conn).readRequest(0xc0001ec460, 0x1ecc5e0, 0xc000216300, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:966 +0x4b0
net/http.(*conn).serve(0xc0001ec460, 0x1ecc5e0, 0xc000216300)
	/usr/local/go/src/net/http/server.go:1788 +0xa31
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:2851 +0x7b0

goroutine 52 [IO wait]:
internal/poll.runtime_pollWait(0x27a9cb8, 0x72, 0x23)
	/usr/local/go/src/runtime/netpoll.go:173 +0x5e
internal/poll.(*pollDesc).wait(0xc00053b798, 0x72, 0x1ec7000, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x96
internal/poll.(*pollDesc).waitRead(0xc00053b798, 0xc000644000, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x45
internal/poll.(*FD).Read(0xc00053b780, 0xc000644000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x36a
net.(*netFD).Read(0xc00053b780, 0xc000644000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/fd_unix.go:202 +0x7a
net.(*conn).Read(0xc0004c2310, 0xc000644000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:177 +0xb0
net/http.(*connReader).Read(0xc0006369c0, 0xc000644000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:786 +0x2c4
bufio.(*Reader).fill(0xc000081200)
	/usr/local/go/src/bufio/bufio.go:100 +0x2d2
bufio.(*Reader).ReadSlice(0xc000081200, 0xc00068550a, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/bufio/bufio.go:341 +0x3d6
bufio.(*Reader).ReadLine(0xc000081200, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/bufio/bufio.go:370 +0x91
net/textproto.(*Reader).readLineSlice(0xc0006369f0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/textproto/reader.go:55 +0xaf
net/textproto.(*Reader).ReadLine(0xc0006369f0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/textproto/reader.go:36 +0x6f
net/http.readRequest(0xc000081200, 0x101000, 0xc00074a500, 0x0, 0x0)
	/usr/local/go/src/net/http/request.go:958 +0xb0
net/http.(*conn).readRequest(0xc0001ec5a0, 0x1ecc5e0, 0xc000339f80, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:966 +0x4b0
net/http.(*conn).serve(0xc0001ec5a0, 0x1ecc5e0, 0xc000339f80)
	/usr/local/go/src/net/http/server.go:1788 +0xa31
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:2851 +0x7b0

goroutine 37 [chan receive]:
github.com/go-redis/redis/internal/pool.(*ConnPool).reaper(0xc000912000, 0xdf8475800)
	/Users/username/go/src/github.com/go-redis/redis/internal/pool/pool.go:449 +0xb4
created by github.com/go-redis/redis/internal/pool.NewConnPool
	/Users/username/go/src/github.com/go-redis/redis/internal/pool/pool.go:99 +0x2a3

goroutine 57 [chan receive]:
github.com/go-redis/redis/internal/pool.(*ConnPool).reaper(0xc000376f20, 0xdf8475800)
	/Users/username/go/src/github.com/go-redis/redis/internal/pool/pool.go:449 +0xb4
created by github.com/go-redis/redis/internal/pool.NewConnPool
	/Users/username/go/src/github.com/go-redis/redis/internal/pool/pool.go:99 +0x2a3

goroutine 74 [IO wait]:
internal/poll.runtime_pollWait(0x27a9638, 0x72, 0x23)
	/usr/local/go/src/runtime/netpoll.go:173 +0x5e
internal/poll.(*pollDesc).wait(0xc00053b318, 0x72, 0x1ec7000, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x96
internal/poll.(*pollDesc).waitRead(0xc00053b318, 0xc0007ac000, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x45
internal/poll.(*FD).Read(0xc00053b300, 0xc0007ac000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x36a
net.(*netFD).Read(0xc00053b300, 0xc0007ac000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/fd_unix.go:202 +0x7a
net.(*conn).Read(0xc00033b8a0, 0xc0007ac000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:177 +0xb0
net/http.(*connReader).Read(0xc0007aa360, 0xc0007ac000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:786 +0x2c4
bufio.(*Reader).fill(0xc00076e7e0)
	/usr/local/go/src/bufio/bufio.go:100 +0x2d2
bufio.(*Reader).ReadSlice(0xc00076e7e0, 0xc00068950a, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/bufio/bufio.go:341 +0x3d6
bufio.(*Reader).ReadLine(0xc00076e7e0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/bufio/bufio.go:370 +0x91
net/textproto.(*Reader).readLineSlice(0xc000703fb0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/textproto/reader.go:55 +0xaf
net/textproto.(*Reader).ReadLine(0xc000703fb0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/textproto/reader.go:36 +0x6f
net/http.readRequest(0xc00076e7e0, 0x101000, 0xc0008dec00, 0x0, 0x0)
	/usr/local/go/src/net/http/request.go:958 +0xb0
net/http.(*conn).readRequest(0xc0001badc0, 0x1ecc5e0, 0xc000774580, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:966 +0x4b0
net/http.(*conn).serve(0xc0001badc0, 0x1ecc5e0, 0xc000774580)
	/usr/local/go/src/net/http/server.go:1788 +0xa31
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:2851 +0x7b0

goroutine 72 [IO wait]:
internal/poll.runtime_pollWait(0x27a97d8, 0x72, 0x23)
	/usr/local/go/src/runtime/netpoll.go:173 +0x5e
internal/poll.(*pollDesc).wait(0xc00053b218, 0x72, 0x1ec7000, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x96
internal/poll.(*pollDesc).waitRead(0xc00053b218, 0xc0007ba000, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x45
internal/poll.(*FD).Read(0xc00053b200, 0xc0007ba000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x36a
net.(*netFD).Read(0xc00053b200, 0xc0007ba000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/fd_unix.go:202 +0x7a
net.(*conn).Read(0xc00033b890, 0xc0007ba000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:177 +0xb0
net/http.(*connReader).Read(0xc0007aa840, 0xc0007ba000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:786 +0x2c4
bufio.(*Reader).fill(0xc00076e960)
	/usr/local/go/src/bufio/bufio.go:100 +0x2d2
bufio.(*Reader).ReadSlice(0xc00076e960, 0xc0008dd50a, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/bufio/bufio.go:341 +0x3d6
bufio.(*Reader).ReadLine(0xc00076e960, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/bufio/bufio.go:370 +0x91
net/textproto.(*Reader).readLineSlice(0xc00086dad0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/textproto/reader.go:55 +0xaf
net/textproto.(*Reader).ReadLine(0xc00086dad0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/textproto/reader.go:36 +0x6f
net/http.readRequest(0xc00076e960, 0x101000, 0xc0008ccc00, 0x0, 0x0)
	/usr/local/go/src/net/http/request.go:958 +0xb0
net/http.(*conn).readRequest(0xc0001ba8c0, 0x1ecc5e0, 0xc000774800, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:966 +0x4b0
net/http.(*conn).serve(0xc0001ba8c0, 0x1ecc5e0, 0xc000774800)
	/usr/local/go/src/net/http/server.go:1788 +0xa31
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:2851 +0x7b0

goroutine 73 [IO wait]:
internal/poll.runtime_pollWait(0x27a9708, 0x72, 0x23)
	/usr/local/go/src/runtime/netpoll.go:173 +0x5e
internal/poll.(*pollDesc).wait(0xc00053b298, 0x72, 0x1ec7000, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x96
internal/poll.(*pollDesc).waitRead(0xc00053b298, 0xc0007bc000, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x45
internal/poll.(*FD).Read(0xc00053b280, 0xc0007bc000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x36a
net.(*netFD).Read(0xc00053b280, 0xc0007bc000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/fd_unix.go:202 +0x7a
net.(*conn).Read(0xc00033b898, 0xc0007bc000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:177 +0xb0
net/http.(*connReader).Read(0xc0007aaab0, 0xc0007bc000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:786 +0x2c4
bufio.(*Reader).fill(0xc00076ea20)
	/usr/local/go/src/bufio/bufio.go:100 +0x2d2
bufio.(*Reader).ReadSlice(0xc00076ea20, 0xc0007c750a, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/bufio/bufio.go:341 +0x3d6
bufio.(*Reader).ReadLine(0xc00076ea20, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/bufio/bufio.go:370 +0x91
net/textproto.(*Reader).readLineSlice(0xc00086cd50, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/textproto/reader.go:55 +0xaf
net/textproto.(*Reader).ReadLine(0xc00086cd50, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/textproto/reader.go:36 +0x6f
net/http.readRequest(0xc00076ea20, 0x101000, 0xc00074b400, 0x0, 0x0)
	/usr/local/go/src/net/http/request.go:958 +0xb0
net/http.(*conn).readRequest(0xc0001bab40, 0x1ecc5e0, 0xc000774a00, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:966 +0x4b0
net/http.(*conn).serve(0xc0001bab40, 0x1ecc5e0, 0xc000774a00)
	/usr/local/go/src/net/http/server.go:1788 +0xa31
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:2851 +0x7b0

goroutine 71 [IO wait]:
internal/poll.runtime_pollWait(0x27a98a8, 0x72, 0x23)
	/usr/local/go/src/runtime/netpoll.go:173 +0x5e
internal/poll.(*pollDesc).wait(0xc00053b198, 0x72, 0x1ec7000, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x96
internal/poll.(*pollDesc).waitRead(0xc00053b198, 0xc0007ae000, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x45
internal/poll.(*FD).Read(0xc00053b180, 0xc0007ae000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x36a
net.(*netFD).Read(0xc00053b180, 0xc0007ae000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/fd_unix.go:202 +0x7a
net.(*conn).Read(0xc00033b888, 0xc0007ae000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:177 +0xb0
net/http.(*connReader).Read(0xc0007aa5d0, 0xc0007ae000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:786 +0x2c4
bufio.(*Reader).fill(0xc00076e8a0)
	/usr/local/go/src/bufio/bufio.go:100 +0x2d2
bufio.(*Reader).ReadSlice(0xc00076e8a0, 0xc0007b950a, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/bufio/bufio.go:341 +0x3d6
bufio.(*Reader).ReadLine(0xc00076e8a0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/bufio/bufio.go:370 +0x91
net/textproto.(*Reader).readLineSlice(0xc000890030, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/textproto/reader.go:55 +0xaf
net/textproto.(*Reader).ReadLine(0xc000890030, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/textproto/reader.go:36 +0x6f
net/http.readRequest(0xc00076e8a0, 0x101000, 0xc0008de400, 0x0, 0x0)
	/usr/local/go/src/net/http/request.go:958 +0xb0
net/http.(*conn).readRequest(0xc0001ba640, 0x1ecc5e0, 0xc0007746c0, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:966 +0x4b0
net/http.(*conn).serve(0xc0001ba640, 0x1ecc5e0, 0xc0007746c0)
	/usr/local/go/src/net/http/server.go:1788 +0xa31
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:2851 +0x7b0

goroutine 97 [select]:
net/http.(*Transport).getConn(0x256f3c0, 0xc0009524b0, 0xc000331d00, 0xc000680780, 0x5, 0xc000632b60, 0x14, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/transport.go:1004 +0xa32
net/http.(*Transport).roundTrip(0x256f3c0, 0xc0008de900, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/transport.go:451 +0xd0f
net/http.(*Transport).RoundTrip(0x256f3c0, 0xc0008de900, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/roundtrip.go:17 +0x5e
net/http.send(0xc0008de900, 0x1ec5a00, 0x256f3c0, 0x0, 0x0, 0x0, 0x0, 0x1c00850, 0x0, 0x0)
	/usr/local/go/src/net/http/client.go:250 +0x4ae
net/http.(*Client).send(0x257b460, 0xc0008de900, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/client.go:174 +0x1e2
net/http.(*Client).do(0x257b460, 0xc0008de900, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/client.go:641 +0x10af
net/http.(*Client).Do(0x257b460, 0xc0008de900, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/client.go:509 +0x5e
github.com/aws/aws-sdk-go/aws/corehandlers.sendFollowRedirects(0xc000045400, 0x0, 0x0, 0x0)
	/Users/username/go/src/github.com/aws/aws-sdk-go/aws/corehandlers/handlers.go:120 +0x72
github.com/aws/aws-sdk-go/aws/corehandlers.glob..func3(0xc000045400)
	/Users/username/go/src/github.com/aws/aws-sdk-go/aws/corehandlers/handlers.go:112 +0x246
github.com/aws/aws-sdk-go/aws/request.(*HandlerList).Run(0xc0000455a8, 0xc000045400)
	/Users/username/go/src/github.com/aws/aws-sdk-go/aws/request/handlers.go:213 +0xf2
github.com/aws/aws-sdk-go/aws/request.(*Request).Send(0xc000045400, 0x0, 0x0)
	/Users/username/go/src/github.com/aws/aws-sdk-go/aws/request/request.go:493 +0x553
github.com/aws/aws-sdk-go/service/s3.(*S3).GetBucketLocation(0xc0004c2530, 0xc0004c2538, 0x0, 0x0, 0x0)
	/Users/username/go/src/github.com/aws/aws-sdk-go/service/s3/api.go:2141 +0x92
bitbucket.org/{{redacted}}(0xc000199940, 0x0)
	/Users/username/go/src/bitbucket.org/{{redacted}}:199 +0x10b
bitbucket.org/{{redacted}}(0xc000199940, 0x0, 0x0)
	/Users/username/go/src/bitbucket.org/{{redacted}}:109 +0x4e
bitbucket.org/{{redacted}}(0xc000199940, 0x0, 0x0)
	/Users/username/go/src/bitbucket.org/{{redacted}}:94 +0x37
bitbucket.org/{{redacted}}(0xc000874900)
	/Users/username/go/src/bitbucket.org/{{redacted}}:60 +0xa1
bitbucket.org/{{redacted}}(0xc000874900)
	/Users/username/go/src/bitbucket.org/{{redacted}}:54 +0x49
created by bitbucket.org/{{redacted}}
	/Users/username/go/src/bitbucket.org/{{redacted}}:45 +0x3af

goroutine 108 [chan receive]:
net/http.(*persistConn).addTLS(0xc0008fb560, 0xc000632b60, 0x10, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/transport.go:1177 +0x246
net/http.(*Transport).dialConn(0x256f3c0, 0x1ecc620, 0xc00009c008, 0xc000331d00, 0xc000680780, 0x5, 0xc000632b60, 0x14, 0x0, 0x0, ...)
	/usr/local/go/src/net/http/transport.go:1322 +0xf31
net/http.(*Transport).getConn.func4(0x256f3c0, 0x1ecc620, 0xc00009c008, 0xc0009524e0, 0xc00034ac60)
	/usr/local/go/src/net/http/transport.go:999 +0x8d
created by net/http.(*Transport).getConn
	/usr/local/go/src/net/http/transport.go:998 +0x83d
fatal error: unexpected signal during runtime execution
panic during panic
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8c96b98 pc=0x8c167e7]

runtime stack:
runtime.throw(0x1bea596, 0x2a)
	/usr/local/go/src/runtime/panic.go:608 +0x72
runtime.sigpanic()
	/usr/local/go/src/runtime/signal_unix.go:374 +0x288
fatal error: unexpected signal during runtime execution
stack trace unavailable

@ianlancetaylor ianlancetaylor changed the title macos 10.14 frequent but seemingly random crash with SIGSEGV runtime: macos 10.14 frequent but seemingly random crash with SIGSEGV Oct 9, 2018

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

ianlancetaylor commented Oct 9, 2018

You say that the crash is random. Is it always in crypto/x509.loadSystemRoots, or do you have different stack traces?

CC @randall77

@akamensky

This comment has been minimized.

Copy link
Author

akamensky commented Oct 9, 2018

@ianlancetaylor when I said random means that it happens at random, not that it happens with random functions. The same code works fine until I just get this stack trace on the process that has been running for quite a bit, all with same compiled binary.

I did not compare stack traces so far. I will catch a couple more and report back on this if they all crash in same function or not.

@akamensky

This comment has been minimized.

Copy link
Author

akamensky commented Oct 23, 2018

Sorry for the delay, confirmed that it breaks in same place for me, always same stack trace. This happens when running anything that loads SSL/TLS related things, and as noted before it does not happen consistently, but rather at random.

For now I am falling back to 1.10.4 which works fine (I guess something that has not been backported to 1.10 branch is causing this?)

@aclements

This comment has been minimized.

Copy link
Member

aclements commented Dec 18, 2018

This traceback looks to me like the C code invoked by crypto/x509 had a bad memory reference. Re-titling to reflect that this is most likely an issue in crypto/x509, not in the runtime.

@aclements aclements changed the title runtime: macos 10.14 frequent but seemingly random crash with SIGSEGV crypto/x509: macos 10.14 SIGSEGV in crypto/x509._Cfunc_FetchPEMRoots Dec 18, 2018

@FiloSottile

This comment has been minimized.

Copy link
Member

FiloSottile commented Jan 2, 2019

@akamensky Can you check if this is still an issue at tip with the new macOS roots code?

@akamensky

This comment has been minimized.

Copy link
Author

akamensky commented Jan 28, 2019

@FiloSottile I currently have no way to verify whether this issue is resolved or not. Meanwhile after doing completely clean install of 1.11.4 (and now 1.11.5) with removing entire $GOPATH/* and then rebuilding everything I had not seen this issue for awhile. Which possibly suggests it may have been caused by using different go compiler version / different standard library version to compile one of dependencies and then linking it with binary(?)

Since I have not seen this issue for awhile, I think okay to close (I guess can reopen if starts appearing again?)

@FiloSottile

This comment has been minimized.

Copy link
Member

FiloSottile commented Jan 28, 2019

Thank you, indeed reopen if it happens again!

@bitfield

This comment has been minimized.

Copy link

bitfield commented Apr 16, 2019

I'm seeing something very similar:

fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0x1a3108 pc=0x7fff823244dd]

runtime stack:
runtime.throw(0x14ebcb4, 0x2a)
        /usr/local/Cellar/go/1.12.4/libexec/src/runtime/panic.go:617 +0x72
runtime.sigpanic()
        /usr/local/Cellar/go/1.12.4/libexec/src/runtime/signal_unix.go:374 +0x4a9

goroutine 39 [syscall]:
runtime.cgocall(0x1001e00, 0xc000153470, 0xc0000258e8)
        /usr/local/Cellar/go/1.12.4/libexec/src/runtime/cgocall.go:128 +0x5b fp=0xc000153440 sp=0xc000153408 pc=0x1004b8b
crypto/x509._Cfunc_FetchPEMRoots(0xc0000258e0, 0xc0000258e8, 0xc000065700, 0x0)
        _cgo_gotypes.go:110 +0x4d fp=0xc000153470 sp=0xc000153440 pc=0x124003d
crypto/x509.loadSystemRoots.func1(0xc0000258e0, 0xc0000258e8, 0x20300000000000)
        /usr/local/Cellar/go/1.12.4/libexec/src/crypto/x509/root_cgo_darwin.go:281 +0x12d fp=0xc0001534b0 sp=0xc000153470 pc=0x124416d
crypto/x509.loadSystemRoots(0x0, 0x0, 0x0)
        /usr/local/Cellar/go/1.12.4/libexec/src/crypto/x509/root_cgo_darwin.go:281 +0xec fp=0xc000153580 sp=0xc0001534b0 pc=0x12401ac
crypto/x509.initSystemRoots()
        /usr/local/Cellar/go/1.12.4/libexec/src/crypto/x509/root.go:21 +0x26 fp=0xc0001535a8 sp=0xc000153580 pc=0x1233466
sync.(*Once).Do(0x18bee98, 0x14f8c08)
        /usr/local/Cellar/go/1.12.4/libexec/src/sync/once.go:44 +0xb3 fp=0xc0001535d8 sp=0xc0001535a8 pc=0x106df93
crypto/x509.systemRootsPool(...)
        /usr/local/Cellar/go/1.12.4/libexec/src/crypto/x509/root.go:16
crypto/x509.(*Certificate).Verify(0xc0001a4000, 0xc000022620, 0x13, 0xc000162f90, 0x0, 0xbf2594e467f366d8, 0x640295f, 0x18a2340, 0x0, 0x0, ...)
        /usr/local/Cellar/go/1.12.4/libexec/src/crypto/x509/verify.go:744 +0x6ca fp=0xc0001536f0 sp=0xc0001535d8 pc=0x123714a
crypto/tls.(*Conn).verifyServerCertificate(0xc0000be380, 0xc0001780a0, 0x3, 0x3, 0xead, 0x0)
        /usr/local/Cellar/go/1.12.4/libexec/src/crypto/tls/handshake_client.go:838 +0x26a fp=0xc000153888 sp=0xc0001536f0 pc=0x12690fa
crypto/tls.(*clientHandshakeState).doFullHandshake(0xc000153dc8, 0xc00009e540, 0x60)
        /usr/local/Cellar/go/1.12.4/libexec/src/crypto/tls/handshake_client.go:454 +0x1869 fp=0xc000153bc0 sp=0xc000153888 pc=0x1267249
crypto/tls.(*clientHandshakeState).handshake(0xc000153dc8, 0xc000180120, 0x0)
        /usr/local/Cellar/go/1.12.4/libexec/src/crypto/tls/handshake_client.go:399 +0x3fb fp=0xc000153cc8 sp=0xc000153bc0 pc=0x126560b
crypto/tls.(*Conn).clientHandshake(0xc0000be380, 0x0, 0x0)
        /usr/local/Cellar/go/1.12.4/libexec/src/crypto/tls/handshake_client.go:208 +0x2cd fp=0xc000153f38 sp=0xc000153cc8 pc=0x1263c5d
crypto/tls.(*Conn).Handshake(0xc0000be380, 0x0, 0x0)
        /usr/local/Cellar/go/1.12.4/libexec/src/crypto/tls/conn.go:1343 +0xef fp=0xc000153f78 sp=0xc000153f38 pc=0x126226f
net/http.(*persistConn).addTLS.func2(0x0, 0xc0000be380, 0xc0001822d0, 0xc00017a660)
        /usr/local/Cellar/go/1.12.4/libexec/src/net/http/transport.go:1190 +0x42 fp=0xc000153fc0 sp=0xc000153f78 pc=0x1315752
runtime.goexit()
        /usr/local/Cellar/go/1.12.4/libexec/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc000153fc8 sp=0xc000153fc0 pc=0x1058ec1
created by net/http.(*persistConn).addTLS
        /usr/local/Cellar/go/1.12.4/libexec/src/net/http/transport.go:1186 +0x1ab

goroutine 1 [select]:
net/http.(*Transport).getConn(0x1897fe0, 0xc000162630, 0x0, 0xc000020700, 0x5, 0xc000022620, 0x17, 0x0, 0x0, 0x0, ...)
        /usr/local/Cellar/go/1.12.4/libexec/src/net/http/transport.go:1020 +0x63d
net/http.(*Transport).roundTrip(0x1897fe0, 0xc00016a700, 0xc000162570, 0xc000024160, 0xc000024154)
        /usr/local/Cellar/go/1.12.4/libexec/src/net/http/transport.go:467 +0x6ef
net/http.(*Transport).RoundTrip(0x1897fe0, 0xc00016a700, 0x1897fe0, 0xbf2594e6e1d59b18, 0x2542e412d)
        /usr/local/Cellar/go/1.12.4/libexec/src/net/http/roundtrip.go:17 +0x35
net/http.send(0xc00016a600, 0x157b800, 0x1897fe0, 0xbf2594e6e1d59b18, 0x2542e412d, 0x18a2340, 0xc000010070, 0xbf2594e6e1d59b18, 0x1, 0x0)
        /usr/local/Cellar/go/1.12.4/libexec/src/net/http/client.go:250 +0x461
net/http.(*Client).send(0xc000162420, 0xc00016a600, 0xbf2594e6e1d59b18, 0x2542e412d, 0x18a2340, 0xc000010070, 0x0, 0x1, 0x10188ff)
        /usr/local/Cellar/go/1.12.4/libexec/src/net/http/client.go:174 +0xfb
net/http.(*Client).do(0xc000162420, 0xc00016a600, 0x0, 0x0, 0x0)
        /usr/local/Cellar/go/1.12.4/libexec/src/net/http/client.go:641 +0x279
net/http.(*Client).Do(0xc000162420, 0xc00016a600, 0x0, 0x0, 0x1)
        /usr/local/Cellar/go/1.12.4/libexec/src/net/http/client.go:509 +0x35
github.com/bitfield/uptimerobot/pkg.(*Client).MakeAPICall(0x18a1fe0, 0x14dec4d, 0x11, 0xc00016e240, 0xc000157b30, 0x0, 0x0)
        /Users/john/git/bitfield/uptimerobot/pkg/uptimerobot.go:339 +0x6e9
github.com/bitfield/uptimerobot/pkg.(*Client).GetAccountDetails(0x18a1fe0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x2, 0xc00000e140)
        /Users/john/git/bitfield/uptimerobot/pkg/uptimerobot.go:193 +0xf1
github.com/bitfield/uptimerobot/cmd.glob..func1(0x1899820, 0x18bee08, 0x0, 0x0)
        /Users/john/git/bitfield/uptimerobot/cmd/account.go:15 +0x66
github.com/spf13/cobra.(*Command).execute(0x1899820, 0x18bee08, 0x0, 0x0, 0x1899820, 0x18bee08)
        /Users/john/go/pkg/mod/github.com/spf13/cobra@v0.0.3/command.go:766 +0x2ae
github.com/spf13/cobra.(*Command).ExecuteC(0x1898ea0, 0xc000157f68, 0x13fbe1e, 0x1898ea0)
        /Users/john/go/pkg/mod/github.com/spf13/cobra@v0.0.3/command.go:852 +0x2ec
github.com/spf13/cobra.(*Command).Execute(...)
        /Users/john/go/pkg/mod/github.com/spf13/cobra@v0.0.3/command.go:800
github.com/bitfield/uptimerobot/cmd.Execute()
        /Users/john/git/bitfield/uptimerobot/cmd/root.go:28 +0x32
main.main()
        /Users/john/git/bitfield/uptimerobot/main.go:8 +0x20

goroutine 5 [select]:
net/http.setRequestCancel.func3(0x0, 0xc000162570, 0xc000178000, 0xc000024160, 0xc000168120)
        /usr/local/Cellar/go/1.12.4/libexec/src/net/http/client.go:321 +0xcf
created by net/http.setRequestCancel
        /usr/local/Cellar/go/1.12.4/libexec/src/net/http/client.go:320 +0x251

goroutine 6 [chan receive]:
net/http.(*persistConn).addTLS(0xc000180000, 0xc000022620, 0x13, 0x0, 0xc000022634, 0x3)
        /usr/local/Cellar/go/1.12.4/libexec/src/net/http/transport.go:1196 +0x1d4
net/http.(*Transport).dialConn(0x1897fe0, 0x1585140, 0xc0000d2000, 0x0, 0xc000020700, 0x5, 0xc000022620, 0x17, 0x0, 0x0, ...)
        /usr/local/Cellar/go/1.12.4/libexec/src/net/http/transport.go:1269 +0x1cde
net/http.(*Transport).getConn.func4(0x1897fe0, 0x1585140, 0xc0000d2000, 0xc000162660, 0xc000168180)
        /usr/local/Cellar/go/1.12.4/libexec/src/net/http/transport.go:1015 +0xa6
created by net/http.(*Transport).getConn
        /usr/local/Cellar/go/1.12.4/libexec/src/net/http/transport.go:1014 +0x455

This happens intermittently when running any Go program that makes network requests (including go mod download!). I noticed it happening on various 1.11 versions and now on 1.12.1 and 1.12.4.

@aclements

This comment has been minimized.

Copy link
Member

aclements commented Apr 16, 2019

Re-opening.

@bitfield, could you provide a little more detail? For example, go version and go env. Thanks!

@aclements aclements reopened this Apr 16, 2019

@bitfield

This comment has been minimized.

Copy link

bitfield commented Apr 19, 2019

go version
go version go1.12.4 darwin/amd64

go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/john/Library/Caches/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/john/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/Cellar/go/1.12.4/libexec"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.12.4/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/john/git/paidy/terraform-aws-ecs-deployment/test/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/q3/ybqqxyh92yd0yc865zqk0vpm0000gn/T/go-build202550920=/tmp/go-build -gno-record-gcc-switches -fno-common"

Here's another one:

fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0x5418c8 pc=0x7fff823244dd]

runtime stack:
runtime.throw(0x14c66b4, 0x2a)
        /usr/local/Cellar/go/1.12.4/libexec/src/runtime/panic.go:617 +0x72
runtime.sigpanic()
        /usr/local/Cellar/go/1.12.4/libexec/src/runtime/signal_unix.go:374 +0x4a9

goroutine 55 [syscall]:
runtime.cgocall(0x1001840, 0xc000137470, 0xc00032c028)
        /usr/local/Cellar/go/1.12.4/libexec/src/runtime/cgocall.go:128 +0x5b fp=0xc000137440 sp=0xc000137408 pc=0x1004c4b
crypto/x509._Cfunc_FetchPEMRoots(0xc00032c020, 0xc00032c028, 0xc00032a100, 0x0)
        _cgo_gotypes.go:110 +0x4d fp=0xc000137470 sp=0xc000137440 pc=0x11cd37d
crypto/x509.loadSystemRoots.func1(0xc00032c020, 0xc00032c028, 0x1018c7f)
        /usr/local/Cellar/go/1.12.4/libexec/src/crypto/x509/root_cgo_darwin.go:281 +0x12d fp=0xc0001374b0 sp=0xc000137470 pc=0x11d14ad
crypto/x509.loadSystemRoots(0x0, 0x0, 0x0)
        /usr/local/Cellar/go/1.12.4/libexec/src/crypto/x509/root_cgo_darwin.go:281 +0xec fp=0xc000137580 sp=0xc0001374b0 pc=0x11cd4ec
crypto/x509.initSystemRoots()
        /usr/local/Cellar/go/1.12.4/libexec/src/crypto/x509/root.go:21 +0x26 fp=0xc0001375a8 sp=0xc000137580 pc=0x11c2006
sync.(*Once).Do(0x18a6880, 0x14d12a8)
        /usr/local/Cellar/go/1.12.4/libexec/src/sync/once.go:44 +0xb3 fp=0xc0001375d8 sp=0xc0001375a8 pc=0x1065e23
crypto/x509.systemRootsPool(...)
        /usr/local/Cellar/go/1.12.4/libexec/src/crypto/x509/root.go:16
crypto/x509.(*Certificate).Verify(0xc000318000, 0xc000246180, 0x20, 0xc000230f00, 0x0, 0xbf26926a6b079060, 0x2c8c60a752, 0x1889e80, 0x0, 0x0, ...)
        /usr/local/Cellar/go/1.12.4/libexec/src/crypto/x509/verify.go:744 +0x6ca fp=0xc0001376f0 sp=0xc0001375d8 pc=0x11c5cea
crypto/tls.(*Conn).verifyServerCertificate(0xc000236380, 0xc0000b72c0, 0x4, 0x4, 0x1324, 0x0)
        /usr/local/Cellar/go/1.12.4/libexec/src/crypto/tls/handshake_client.go:838 +0x26a fp=0xc000137888 sp=0xc0001376f0 pc=0x11f4faa
crypto/tls.(*clientHandshakeState).doFullHandshake(0xc000137dc8, 0xc000290540, 0x57)
        /usr/local/Cellar/go/1.12.4/libexec/src/crypto/tls/handshake_client.go:454 +0x1869 fp=0xc000137bc0 sp=0xc000137888 pc=0x11f30f9
crypto/tls.(*clientHandshakeState).handshake(0xc000137dc8, 0xc0002f0240, 0x0)
        /usr/local/Cellar/go/1.12.4/libexec/src/crypto/tls/handshake_client.go:399 +0x3fb fp=0xc000137cc8 sp=0xc000137bc0 pc=0x11f14bb
crypto/tls.(*Conn).clientHandshake(0xc000236380, 0x0, 0x0)
        /usr/local/Cellar/go/1.12.4/libexec/src/crypto/tls/handshake_client.go:208 +0x2cd fp=0xc000137f38 sp=0xc000137cc8 pc=0x11efb0d
crypto/tls.(*Conn).Handshake(0xc000236380, 0x0, 0x0)
        /usr/local/Cellar/go/1.12.4/libexec/src/crypto/tls/conn.go:1343 +0xef fp=0xc000137f78 sp=0xc000137f38 pc=0x11ee1df
net/http.(*persistConn).addTLS.func2(0x0, 0xc000236380, 0xc00023e230, 0xc0000b69c0)
        /usr/local/Cellar/go/1.12.4/libexec/src/net/http/transport.go:1190 +0x42 fp=0xc000137fc0 sp=0xc000137f78 pc=0x1288972
runtime.goexit()
        /usr/local/Cellar/go/1.12.4/libexec/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc000137fc8 sp=0xc000137fc0 pc=0x105d6b1
created by net/http.(*persistConn).addTLS
        /usr/local/Cellar/go/1.12.4/libexec/src/net/http/transport.go:1186 +0x1ab

goroutine 1 [chan receive]:
testing.(*T).Run(0xc000214100, 0x14b8b88, 0x11, 0x14d2478, 0x107b616)
        /usr/local/Cellar/go/1.12.4/libexec/src/testing/testing.go:917 +0x381
testing.runTests.func1(0xc000214000)
        /usr/local/Cellar/go/1.12.4/libexec/src/testing/testing.go:1157 +0x78
testing.tRunner(0xc000214000, 0xc000213e30)
        /usr/local/Cellar/go/1.12.4/libexec/src/testing/testing.go:865 +0xc0
testing.runTests(0xc0000c96e0, 0x187e390, 0x1, 0x1, 0x0)
        /usr/local/Cellar/go/1.12.4/libexec/src/testing/testing.go:1155 +0x2a9
testing.(*M).Run(0xc00013c080, 0x0)
        /usr/local/Cellar/go/1.12.4/libexec/src/testing/testing.go:1072 +0x162
main.main()
        _testmain.go:42 +0x13e

goroutine 18 [select]:
net/http.(*Transport).getConn(0x1883de0, 0xc000230360, 0x0, 0xc000246030, 0x5, 0xc000246180, 0x24, 0x0, 0x0, 0x0, ...)
        /usr/local/Cellar/go/1.12.4/libexec/src/net/http/transport.go:1020 +0x63d
net/http.(*Transport).roundTrip(0x1883de0, 0xc0002e0000, 0x10, 0xc000220900, 0xc00013b6a0)
        /usr/local/Cellar/go/1.12.4/libexec/src/net/http/transport.go:467 +0x6ef
net/http.(*Transport).RoundTrip(0x1883de0, 0xc0002e0000, 0x1883de0, 0x0, 0x0)
        /usr/local/Cellar/go/1.12.4/libexec/src/net/http/roundtrip.go:17 +0x35
net/http.send(0xc0002e0000, 0x15c2520, 0x1883de0, 0x0, 0x0, 0x0, 0xc0000b4038, 0x14d2450, 0x1, 0x0)
        /usr/local/Cellar/go/1.12.4/libexec/src/net/http/client.go:250 +0x461
net/http.(*Client).send(0x1889be0, 0xc0002e0000, 0x0, 0x0, 0x0, 0xc0000b4038, 0x0, 0x1, 0x14b7119)
        /usr/local/Cellar/go/1.12.4/libexec/src/net/http/client.go:174 +0xfb
net/http.(*Client).do(0x1889be0, 0xc0002e0000, 0x0, 0x0, 0x0)
        /usr/local/Cellar/go/1.12.4/libexec/src/net/http/client.go:641 +0x279
net/http.(*Client).Do(...)
        /usr/local/Cellar/go/1.12.4/libexec/src/net/http/client.go:509
github.com/aws/aws-sdk-go/aws/corehandlers.sendFollowRedirects(0xc0002e2000, 0x105ef30, 0xc00013bac8, 0xbf269269fc3317a0)
        /Users/john/go/pkg/mod/github.com/aws/aws-sdk-go@v1.19.10/aws/corehandlers/handlers.go:120 +0x3c
github.com/aws/aws-sdk-go/aws/corehandlers.glob..func3(0xc0002e2000)
        /Users/john/go/pkg/mod/github.com/aws/aws-sdk-go@v1.19.10/aws/corehandlers/handlers.go:112 +0x7f
github.com/aws/aws-sdk-go/aws/request.(*HandlerList).Run(0xc0002e21b8, 0xc0002e2000)
        /Users/john/go/pkg/mod/github.com/aws/aws-sdk-go@v1.19.10/aws/request/handlers.go:216 +0x98
github.com/aws/aws-sdk-go/aws/request.(*Request).sendRequest(0xc0002e2000, 0x0, 0x0)
        /Users/john/go/pkg/mod/github.com/aws/aws-sdk-go@v1.19.10/aws/request/request.go:521 +0x93
github.com/aws/aws-sdk-go/aws/request.(*Request).Send(0xc0002e2000, 0x0, 0x0)
        /Users/john/go/pkg/mod/github.com/aws/aws-sdk-go@v1.19.10/aws/request/request.go:480 +0x117
github.com/aws/aws-sdk-go/service/ecs.(*ECS).DescribeServices(0xc000266040, 0xc00022e0c0, 0xe, 0xc000266040, 0x184e3e0)
        /Users/john/go/pkg/mod/github.com/aws/aws-sdk-go@v1.19.10/service/ecs/api.go:1412 +0x4d
tmp.TestRoutedService(0xc000214100)
        /Users/john/git/paidy/terraform-aws-ecs-deployment/test/main_test.go:39 +0x227
testing.tRunner(0xc000214100, 0x14d2478)
        /usr/local/Cellar/go/1.12.4/libexec/src/testing/testing.go:865 +0xc0
created by testing.(*T).Run
        /usr/local/Cellar/go/1.12.4/libexec/src/testing/testing.go:916 +0x35a

goroutine 50 [chan receive]:
net/http.(*persistConn).addTLS(0xc0002f0120, 0xc000246180, 0x20, 0x0, 0xc0002461a1, 0x3)
        /usr/local/Cellar/go/1.12.4/libexec/src/net/http/transport.go:1196 +0x1d4
net/http.(*Transport).dialConn(0x1883de0, 0x15c9fc0, 0xc0000ca000, 0x0, 0xc000246030, 0x5, 0xc000246180, 0x24, 0x0, 0x0, ...)
        /usr/local/Cellar/go/1.12.4/libexec/src/net/http/transport.go:1269 +0x1cde
net/http.(*Transport).getConn.func4(0x1883de0, 0x15c9fc0, 0xc0000ca000, 0xc000230390, 0xc000290120)
        /usr/local/Cellar/go/1.12.4/libexec/src/net/http/transport.go:1015 +0xa6
created by net/http.(*Transport).getConn
        /usr/local/Cellar/go/1.12.4/libexec/src/net/http/transport.go:1014 +0x455

I'd say this happens about one or two times in ten when running any Go binary that makes network calls.

@aclements

This comment has been minimized.

Copy link
Member

aclements commented Apr 19, 2019

@bitfield, is it possible to run this under gdb or lldb to see where the SIGSEGV is happening in the C code?

@bitfield

This comment has been minimized.

Copy link

bitfield commented Apr 19, 2019

Sure, just let me know what command I need to run to do this.

@aclements

This comment has been minimized.

Copy link
Member

aclements commented Apr 19, 2019

In gdb, just start your program under gdb with "gdb --args <command...>" and then type "run". It will stop when there's a segfault. At that point the output of running "backtrace" and "info locals" is probably the most useful starting point.

I'm not an lldb user, so I'm not sure how to do this with lldb. @dr2chase?

@bitfield

This comment has been minimized.

Copy link

bitfield commented Apr 19, 2019

lldb instructions would be useful, because I have that, but I don't have gdb. I could install gdb, but I'd have to install gcc...

Homebrew is like a really boring text adventure.

INSTALL MACVIM
You need to install Python first.

INSTALL PYTHON
You need to update your Xcode to do that!

PUT THE JUNK MAIL ON THE SATCHEL
You don't have that item.

BREW DOCTOR
You were eaten by a grue!
https://twitter.com/bitfield/status/986967919094648832

@aclements

This comment has been minimized.

Copy link
Member

aclements commented Apr 19, 2019

It appears it's basically the same in lldb:

lldb -- <command...>
run
# wait for it to segfault
bt
@bitfield

This comment has been minimized.

Copy link

bitfield commented Apr 20, 2019

Thanks. Here's the simplest program that will demonstrate the crash:

package main

import (
	"fmt"
	"net/http"
)

func main() {
	res, _ := http.Get("https://postman-echo.com/get")
	fmt.Println(res.Status)
}

And here's the lldb session:

lldb -- ./cryptocrash 
(lldb) target create "./cryptocrash"
Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "/usr/local/Cellar/python@2/2.7.15_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 52, in <module>
    import weakref
  File "/usr/local/Cellar/python@2/2.7.15_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/weakref.py", line 14, in <module>
    from _weakref import (
ImportError: cannot import name _remove_dead_weakref
Current executable set to './cryptocrash' (x86_64).
(lldb) run
Process 59576 launched: './cryptocrash' (x86_64)
200 OK
Process 59576 exited with status = 0 (0x00000000) 
(lldb) run
Process 59586 launched: './cryptocrash' (x86_64)
200 OK
Process 59586 exited with status = 0 (0x00000000) 
(lldb) run
Process 59590 launched: './cryptocrash' (x86_64)
Process 59590 stopped
* thread #2: tid = 0x1286d12, 0x00007fff823244dd, stop reason = EXC_BAD_ACCESS (code=1, address=0x5a0128)
    frame #0: 0x00007fff823244dd
->  0x7fff823244dd: andl   0x18(%r11), %r10d
    0x7fff823244e1: shlq   $0x4, %r10
    0x7fff823244e5: addq   0x10(%r11), %r10
    0x7fff823244e9: cmpq   (%r10), %rsi
(lldb) bt
* thread #2: tid = 0x1286d12, 0x00007fff823244dd, stop reason = EXC_BAD_ACCESS (code=1, address=0x5a0128)
  * frame #0: 0x00007fff823244dd
    frame #1: 0x0000000001001472 cryptocrash`FetchPEMRoots + 770
    frame #2: 0x0000000001001865 cryptocrash`_cgo_595ff63b07d2_Cfunc_FetchPEMRoots + 37
    frame #3: 0x0000000001057ba0 cryptocrash`runtime.asmcgocall + 112
    frame #4: 0x00000000010311d0 cryptocrash`runtime.startTheWorldWithSema + 624
    frame #5: 0x00000000011636fd cryptocrash`crypto/x509._Cfunc_FetchPEMRoots + 77
    frame #6: 0x000000000116782d cryptocrash`crypto/x509.loadSystemRoots.func1 + 301
    frame #7: 0x000000000116386c cryptocrash`crypto/x509.loadSystemRoots + 236
    frame #8: 0x00000000011585b6 cryptocrash`crypto/x509.initSystemRoots + 38
    frame #9: 0x000000000106a203 cryptocrash`sync.(*Once).Do + 179
    frame #10: 0x000000000115c29a cryptocrash`crypto/x509.(*Certificate).Verify + 1738
    frame #11: 0x000000000118b32a cryptocrash`crypto/tls.(*Conn).verifyServerCertificate + 618
    frame #12: 0x0000000001189479 cryptocrash`crypto/tls.(*clientHandshakeState).doFullHandshake + 6249
    frame #13: 0x000000000118783b cryptocrash`crypto/tls.(*clientHandshakeState).handshake + 1019
    frame #14: 0x0000000001185e8d cryptocrash`crypto/tls.(*Conn).clientHandshake + 717
    frame #15: 0x000000000118455f cryptocrash`crypto/tls.(*Conn).Handshake + 239
    frame #16: 0x00000000012230c2 cryptocrash`net/http.(*persistConn).addTLS.func2 + 66
    frame #17: 0x0000000001058401 cryptocrash`runtime.goexit + 1
    frame #18: 0x000000000116782d cryptocrash`crypto/x509.loadSystemRoots.func1 + 301
    frame #19: 0x000000000116386c cryptocrash`crypto/x509.loadSystemRoots + 236
    frame #20: 0x00000000011585b6 cryptocrash`crypto/x509.initSystemRoots + 38
    frame #21: 0x000000000106a203 cryptocrash`sync.(*Once).Do + 179
    frame #22: 0x000000000115c29a cryptocrash`crypto/x509.(*Certificate).Verify + 1738
    frame #23: 0x000000000118b32a cryptocrash`crypto/tls.(*Conn).verifyServerCertificate + 618
    frame #24: 0x0000000001189479 cryptocrash`crypto/tls.(*clientHandshakeState).doFullHandshake + 6249
    frame #25: 0x000000000118783b cryptocrash`crypto/tls.(*clientHandshakeState).handshake + 1019
    frame #26: 0x0000000001185e8d cryptocrash`crypto/tls.(*Conn).clientHandshake + 717
    frame #27: 0x000000000118455f cryptocrash`crypto/tls.(*Conn).Handshake + 239
    frame #28: 0x00000000012230c2 cryptocrash`net/http.(*persistConn).addTLS.func2 + 66
    frame #29: 0x0000000001058401 cryptocrash`runtime.goexit + 1
    frame #30: 0x000000000116386c cryptocrash`crypto/x509.loadSystemRoots + 236
    frame #31: 0x00000000011585b6 cryptocrash`crypto/x509.initSystemRoots + 38
    frame #32: 0x000000000106a203 cryptocrash`sync.(*Once).Do + 179
    frame #33: 0x000000000115c29a cryptocrash`crypto/x509.(*Certificate).Verify + 1738
    frame #34: 0x000000000118b32a cryptocrash`crypto/tls.(*Conn).verifyServerCertificate + 618
    frame #35: 0x0000000001189479 cryptocrash`crypto/tls.(*clientHandshakeState).doFullHandshake + 6249
    frame #36: 0x000000000118783b cryptocrash`crypto/tls.(*clientHandshakeState).handshake + 1019
    frame #37: 0x0000000001185e8d cryptocrash`crypto/tls.(*Conn).clientHandshake + 717
    frame #38: 0x000000000118455f cryptocrash`crypto/tls.(*Conn).Handshake + 239
    frame #39: 0x00000000012230c2 cryptocrash`net/http.(*persistConn).addTLS.func2 + 66
    frame #40: 0x0000000001058401 cryptocrash`runtime.goexit + 1
    frame #41: 0x00000000011585b6 cryptocrash`crypto/x509.initSystemRoots + 38
    frame #42: 0x000000000106a203 cryptocrash`sync.(*Once).Do + 179
    frame #43: 0x000000000115c29a cryptocrash`crypto/x509.(*Certificate).Verify + 1738
    frame #44: 0x000000000118b32a cryptocrash`crypto/tls.(*Conn).verifyServerCertificate + 618
    frame #45: 0x0000000001189479 cryptocrash`crypto/tls.(*clientHandshakeState).doFullHandshake + 6249
    frame #46: 0x000000000118783b cryptocrash`crypto/tls.(*clientHandshakeState).handshake + 1019
    frame #47: 0x0000000001185e8d cryptocrash`crypto/tls.(*Conn).clientHandshake + 717
    frame #48: 0x000000000118455f cryptocrash`crypto/tls.(*Conn).Handshake + 239
    frame #49: 0x00000000012230c2 cryptocrash`net/http.(*persistConn).addTLS.func2 + 66
    frame #50: 0x0000000001058401 cryptocrash`runtime.goexit + 1
    frame #51: 0x000000000106a203 cryptocrash`sync.(*Once).Do + 179
    frame #52: 0x000000000115c29a cryptocrash`crypto/x509.(*Certificate).Verify + 1738
    frame #53: 0x000000000118b32a cryptocrash`crypto/tls.(*Conn).verifyServerCertificate + 618
    frame #54: 0x0000000001189479 cryptocrash`crypto/tls.(*clientHandshakeState).doFullHandshake + 6249
    frame #55: 0x000000000118783b cryptocrash`crypto/tls.(*clientHandshakeState).handshake + 1019
    frame #56: 0x0000000001185e8d cryptocrash`crypto/tls.(*Conn).clientHandshake + 717
    frame #57: 0x000000000118455f cryptocrash`crypto/tls.(*Conn).Handshake + 239
    frame #58: 0x00000000012230c2 cryptocrash`net/http.(*persistConn).addTLS.func2 + 66
    frame #59: 0x0000000001058401 cryptocrash`runtime.goexit + 1
    frame #60: 0x000000000115c29a cryptocrash`crypto/x509.(*Certificate).Verify + 1738
    frame #61: 0x000000000118b32a cryptocrash`crypto/tls.(*Conn).verifyServerCertificate + 618
    frame #62: 0x0000000001189479 cryptocrash`crypto/tls.(*clientHandshakeState).doFullHandshake + 6249
    frame #63: 0x000000000118783b cryptocrash`crypto/tls.(*clientHandshakeState).handshake + 1019
    frame #64: 0x0000000001185e8d cryptocrash`crypto/tls.(*Conn).clientHandshake + 717
    frame #65: 0x000000000118455f cryptocrash`crypto/tls.(*Conn).Handshake + 239
    frame #66: 0x00000000012230c2 cryptocrash`net/http.(*persistConn).addTLS.func2 + 66
    frame #67: 0x0000000001058401 cryptocrash`runtime.goexit + 1
    frame #68: 0x000000000118b32a cryptocrash`crypto/tls.(*Conn).verifyServerCertificate + 618
    frame #69: 0x0000000001189479 cryptocrash`crypto/tls.(*clientHandshakeState).doFullHandshake + 6249
    frame #70: 0x000000000118783b cryptocrash`crypto/tls.(*clientHandshakeState).handshake + 1019
    frame #71: 0x0000000001185e8d cryptocrash`crypto/tls.(*Conn).clientHandshake + 717
    frame #72: 0x000000000118455f cryptocrash`crypto/tls.(*Conn).Handshake + 239
    frame #73: 0x00000000012230c2 cryptocrash`net/http.(*persistConn).addTLS.func2 + 66
    frame #74: 0x0000000001058401 cryptocrash`runtime.goexit + 1
    frame #75: 0x0000000001189479 cryptocrash`crypto/tls.(*clientHandshakeState).doFullHandshake + 6249
    frame #76: 0x000000000118783b cryptocrash`crypto/tls.(*clientHandshakeState).handshake + 1019
    frame #77: 0x0000000001185e8d cryptocrash`crypto/tls.(*Conn).clientHandshake + 717
    frame #78: 0x000000000118455f cryptocrash`crypto/tls.(*Conn).Handshake + 239
    frame #79: 0x00000000012230c2 cryptocrash`net/http.(*persistConn).addTLS.func2 + 66
    frame #80: 0x0000000001058401 cryptocrash`runtime.goexit + 1
    frame #81: 0x000000000118783b cryptocrash`crypto/tls.(*clientHandshakeState).handshake + 1019
    frame #82: 0x0000000001185e8d cryptocrash`crypto/tls.(*Conn).clientHandshake + 717
    frame #83: 0x000000000118455f cryptocrash`crypto/tls.(*Conn).Handshake + 239
    frame #84: 0x00000000012230c2 cryptocrash`net/http.(*persistConn).addTLS.func2 + 66
    frame #85: 0x0000000001058401 cryptocrash`runtime.goexit + 1
    frame #86: 0x0000000001185e8d cryptocrash`crypto/tls.(*Conn).clientHandshake + 717
    frame #87: 0x000000000118455f cryptocrash`crypto/tls.(*Conn).Handshake + 239
    frame #88: 0x00000000012230c2 cryptocrash`net/http.(*persistConn).addTLS.func2 + 66
    frame #89: 0x0000000001058401 cryptocrash`runtime.goexit + 1
    frame #90: 0x000000000118455f cryptocrash`crypto/tls.(*Conn).Handshake + 239
    frame #91: 0x00000000012230c2 cryptocrash`net/http.(*persistConn).addTLS.func2 + 66
    frame #92: 0x0000000001058401 cryptocrash`runtime.goexit + 1
    frame #93: 0x00000000012230c2 cryptocrash`net/http.(*persistConn).addTLS.func2 + 66
    frame #94: 0x0000000001058401 cryptocrash`runtime.goexit + 1
    frame #95: 0x0000000001058401 cryptocrash`runtime.goexit + 1
(lldb) 
@bitfield

This comment has been minimized.

Copy link

bitfield commented Apr 21, 2019

Also, just like OP, I've had this problem on every 1.11 and 1.12 version, and it was fine before that. My Go installation is Homebrew, and I'm working outside GOPATH.

@dr2chase

This comment has been minimized.

Copy link
Contributor

dr2chase commented Apr 22, 2019

All reports, it's good to reconfirm which version of Darwin it is, and maybe which version of XCode, too. I just did 200 repetitions of the tiny test program with no error.

Darwin version is 10.14.4 (Apple glyph -> "About this Mac")

cc --version
Apple LLVM version 10.0.1 (clang-1001.0.46.3)
Target: x86_64-apple-darwin18.5.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
@bitfield

This comment has been minimized.

Copy link

bitfield commented Apr 22, 2019

OS X 10.11.6 (15G22010
Apple LLVM version 8.0.0 (clang-800.0.42.1)
Target: x86_64-apple-darwin15.6.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.