crypto/x509: FetchPEMRoots in CGO crashed with signal SIGSEGV on go get gopkg.in/sourcemap.v1 on macOS Sierra (10.12.6)

[ericreis]

What version of Go are you using (go version)?

go version go1.10 darwin/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

macOS Sierra (10.12.6)

GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/ericreis/Library/Caches/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/ericreis/dev/golang"
GORACE=""
GOROOT="/usr/local/Cellar/go/1.10/libexec"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.10/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/bx/xqyzvybj06scstvsdxpcd7980000gn/T/go-build639091728=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

1. Just tried to run go get github.com/robertkrimen/otto
2. After many attempts to fix (reinstalling go, updating from OS X El Capitain to macOS Sierra)
3. Tried to install the same otto package in my windows desktop and it was successful
4. Opened an issue in the otto github (Is there a way to install otto via go get? Error when running go get. robertkrimen/otto#291)
5. Discovered that the problem was actually in the gopkg.in/sourcemap.v1 package (the only dependency of the otto package, according to the collaborator that helped me with this issue)
6. Tried to run go get gopkg.in/sourcemap.v1 and the same problem occurred
7. The last thing I tried was running the following code that the otto collaborator asked me to run:

package main

import (
	"fmt"
	"net/http"
)

func main() {
	res, err := http.Get("https://gopkg.in/sourcemap.v1?go-get=1")
	if err != nil {
		panic(err)
	}

	fmt.Printf("%#v\n", res)
}

What did you expect to see?

Expected to not see the same SIGSEGV error.

What did you see instead?

The same SIGSEGV error:

go run main.go 
fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0xb01dfacedebac1e pc=0x143e7f9]

runtime stack:
runtime.throw(0x127a124, 0x2a)
	/usr/local/Cellar/go/1.10/libexec/src/runtime/panic.go:619 +0x81
runtime.sigpanic()
	/usr/local/Cellar/go/1.10/libexec/src/runtime/signal_unix.go:372 +0x28e

goroutine 21 [syscall]:
runtime.cgocall(0x1001840, 0xc4201814a8, 0x12077c0)
	/usr/local/Cellar/go/1.10/libexec/src/runtime/cgocall.go:128 +0x64 fp=0xc420181478 sp=0xc420181440 pc=0x1003194
crypto/x509._Cfunc_FetchPEMRoots(0xc4200a10c0, 0xc4200a10c8, 0xc400000000)
	_cgo_gotypes.go:104 +0x4d fp=0xc4201814a8 sp=0xc420181478 pc=0x1150a1d
crypto/x509.loadSystemRoots.func1(0xc4200a10c0, 0xc4200a10c8, 0xc420113540)
	/usr/local/Cellar/go/1.10/libexec/src/crypto/x509/root_cgo_darwin.go:226 +0x123 fp=0xc4201814e0 sp=0xc4201814a8 pc=0x1153f93
crypto/x509.loadSystemRoots(0x0, 0x0, 0x0)
	/usr/local/Cellar/go/1.10/libexec/src/crypto/x509/root_cgo_darwin.go:226 +0xf6 fp=0xc4201815b8 sp=0xc4201814e0 pc=0x1150be6
crypto/x509.initSystemRoots()
	/usr/local/Cellar/go/1.10/libexec/src/crypto/x509/root.go:21 +0x22 fp=0xc4201815e0 sp=0xc4201815b8 pc=0x1145532
sync.(*Once).Do(0x13e4948, 0x1281b88)
	/usr/local/Cellar/go/1.10/libexec/src/sync/once.go:44 +0xbe fp=0xc420181618 sp=0xc4201815e0 pc=0x106734e
crypto/x509.systemRootsPool(0x6)
	/usr/local/Cellar/go/1.10/libexec/src/crypto/x509/root.go:16 +0x39 fp=0xc420181638 sp=0xc420181618 pc=0x11454e9
crypto/x509.(*Certificate).Verify(0xc4200e8580, 0xc4200a0400, 0x8, 0xc42009b440, 0x0, 0xbe9e0209cd38ac76, 0x18c986bf, 0x13c7100, 0x0, 0x0, ...)
	/usr/local/Cellar/go/1.10/libexec/src/crypto/x509/verify.go:815 +0x734 fp=0xc420181890 sp=0xc420181638 pc=0x1149374
crypto/tls.(*clientHandshakeState).doFullHandshake(0xc420181e70, 0xc42009e0c0, 0x3a)
	/usr/local/Cellar/go/1.10/libexec/src/crypto/tls/handshake_client.go:339 +0x526 fp=0xc420181cd8 sp=0xc420181890 pc=0x116f556
crypto/tls.(*clientHandshakeState).handshake(0xc420181e70, 0xc4200e4280, 0x0)
	/usr/local/Cellar/go/1.10/libexec/src/crypto/tls/handshake_client.go:247 +0x481 fp=0xc420181df0 sp=0xc420181cd8 pc=0x116e9f1
crypto/tls.(*Conn).clientHandshake(0xc4200a6a80, 0x1282380, 0xc4200a6ba0)
	/usr/local/Cellar/go/1.10/libexec/src/crypto/tls/handshake_client.go:168 +0x38c fp=0xc420181f28 sp=0xc420181df0 pc=0x116e31c
crypto/tls.(*Conn).Handshake(0xc4200a6a80, 0x0, 0x0)
	/usr/local/Cellar/go/1.10/libexec/src/crypto/tls/conn.go:1329 +0x185 fp=0xc420181f78 sp=0xc420181f28 pc=0x116d095
net/http.(*persistConn).addTLS.func2(0x0, 0xc4200a6a80, 0xc4200ae280, 0xc4201262a0)
	/usr/local/Cellar/go/1.10/libexec/src/net/http/transport.go:1068 +0x42 fp=0xc420181fc0 sp=0xc420181f78 pc=0x11e1862
runtime.goexit()
	/usr/local/Cellar/go/1.10/libexec/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc420181fc8 sp=0xc420181fc0 pc=0x10557e1
created by net/http.(*persistConn).addTLS
	/usr/local/Cellar/go/1.10/libexec/src/net/http/transport.go:1064 +0x19f

goroutine 1 [select]:
net/http.(*Transport).getConn(0x13c26c0, 0xc42009ade0, 0x0, 0x1278f09, 0x5, 0xc4200a0400, 0xc, 0x0, 0x0, 0x0)
	/usr/local/Cellar/go/1.10/libexec/src/net/http/transport.go:962 +0x558
net/http.(*Transport).RoundTrip(0x13c26c0, 0xc42012c000, 0x13c26c0, 0x0, 0x0)
	/usr/local/Cellar/go/1.10/libexec/src/net/http/transport.go:409 +0x632
net/http.send(0xc42012c000, 0x129e940, 0x13c26c0, 0x0, 0x0, 0x0, 0xc4200ac038, 0xf8, 0xc420067cf8, 0x1)
	/usr/local/Cellar/go/1.10/libexec/src/net/http/client.go:252 +0x185
net/http.(*Client).send(0x13c6e40, 0xc42012c000, 0x0, 0x0, 0x0, 0xc4200ac038, 0x0, 0x1, 0x1011ef8)
	/usr/local/Cellar/go/1.10/libexec/src/net/http/client.go:176 +0xfa
net/http.(*Client).Do(0x13c6e40, 0xc42012c000, 0x1278f09, 0x26, 0x0)
	/usr/local/Cellar/go/1.10/libexec/src/net/http/client.go:615 +0x28d
net/http.(*Client).Get(0x13c6e40, 0x1278f09, 0x26, 0xc42009abd0, 0xc4200b4780, 0xc42009aba0)
	/usr/local/Cellar/go/1.10/libexec/src/net/http/client.go:396 +0x9d
net/http.Get(0x1278f09, 0x26, 0x0, 0xc42009a990, 0xc42009a870)
	/usr/local/Cellar/go/1.10/libexec/src/net/http/client.go:370 +0x41
main.main()
	/Users/ericreis/dev/golang/src/github.com/ericreis/test/main.go:9 +0x3a

goroutine 19 [chan receive]:
net/http.(*persistConn).addTLS(0xc42013c000, 0xc4200a0400, 0x8, 0x0, 0xc4200a0409, 0x3)
	/usr/local/Cellar/go/1.10/libexec/src/net/http/transport.go:1074 +0x1c8
net/http.(*Transport).dialConn(0x13c26c0, 0x12a0fe0, 0xc4200a0020, 0x0, 0x1278f09, 0x5, 0xc4200a0400, 0xc, 0x0, 0x0, ...)
	/usr/local/Cellar/go/1.10/libexec/src/net/http/transport.go:1153 +0x14c3
net/http.(*Transport).getConn.func4(0x13c26c0, 0x12a0fe0, 0xc4200a0020, 0xc42009ae10, 0xc42008c120)
	/usr/local/Cellar/go/1.10/libexec/src/net/http/transport.go:957 +0x78
created by net/http.(*Transport).getConn
	/usr/local/Cellar/go/1.10/libexec/src/net/http/transport.go:956 +0x363
exit status 2

[odeke-em]

Hello there @ericreis, thank you for reporting this issue. So I have a somewhat similar environment on OS X

$ go env
GOARCH="amd64"
GOBIN="/Users/emmanuelodeke/go/bin"
GOCACHE="/Users/emmanuelodeke/Library/Caches/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/emmanuelodeke/go"
GORACE=""
GOROOT="/Users/emmanuelodeke/go/src/go.googlesource.com/go"
GOTMPDIR=""
GOTOOLDIR="/Users/emmanuelodeke/go/src/go.googlesource.com/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/v3/7z434qpx5v3bw0wh8h2myfpw0000gn/T/go-build194242034=/tmp/go-build -gno-record-gcc-switches -fno-common"

But I can't seem to reproduce this issue

$ for ((i=0;i<10;i++)) do go run main.go;done
&http.Response{Status:"200 OK", StatusCode:200, Proto:"HTTP/2.0", ProtoMajor:2, ProtoMinor:0, Header:http.Header{"Content-Type":[]string{"text/html"}, "Content-Length":[]string{"358"}, "Date":[]string{"Fri, 02 Mar 2018 01:31:12 GMT"}}, Body:http.http2transportResponseBody{cs:(*http.http2clientStream)(0xc000238a00)}, ContentLength:358, TransferEncoding:[]string(nil), Close:false, Uncompressed:false, Trailer:http.Header(nil), Request:(*http.Request)(0xc00010c000), TLS:(*tls.ConnectionState)(0xc000492370)}
&http.Response{Status:"200 OK", StatusCode:200, Proto:"HTTP/2.0", ProtoMajor:2, ProtoMinor:0, Header:http.Header{"Content-Type":[]string{"text/html"}, "Content-Length":[]string{"358"}, "Date":[]string{"Fri, 02 Mar 2018 01:31:13 GMT"}}, Body:http.http2transportResponseBody{cs:(*http.http2clientStream)(0xc000408f00)}, ContentLength:358, TransferEncoding:[]string(nil), Close:false, Uncompressed:false, Trailer:http.Header(nil), Request:(*http.Request)(0xc00010c000), TLS:(*tls.ConnectionState)(0xc0000bcb00)}
&http.Response{Status:"200 OK", StatusCode:200, Proto:"HTTP/2.0", ProtoMajor:2, ProtoMinor:0, Header:http.Header{"Content-Type":[]string{"text/html"}, "Content-Length":[]string{"358"}, "Date":[]string{"Fri, 02 Mar 2018 01:31:14 GMT"}}, Body:http.http2transportResponseBody{cs:(*http.http2clientStream)(0xc0003ddb80)}, ContentLength:358, TransferEncoding:[]string(nil), Close:false, Uncompressed:false, Trailer:http.Header(nil), Request:(*http.Request)(0xc000126000), TLS:(*tls.ConnectionState)(0xc0004a6790)}
&http.Response{Status:"200 OK", StatusCode:200, Proto:"HTTP/2.0", ProtoMajor:2, ProtoMinor:0, Header:http.Header{"Content-Type":[]string{"text/html"}, "Content-Length":[]string{"358"}, "Date":[]string{"Fri, 02 Mar 2018 01:31:15 GMT"}}, Body:http.http2transportResponseBody{cs:(*http.http2clientStream)(0xc000312a00)}, ContentLength:358, TransferEncoding:[]string(nil), Close:false, Uncompressed:false, Trailer:http.Header(nil), Request:(*http.Request)(0xc00010c000), TLS:(*tls.ConnectionState)(0xc000488790)}
&http.Response{Status:"200 OK", StatusCode:200, Proto:"HTTP/2.0", ProtoMajor:2, ProtoMinor:0, Header:http.Header{"Content-Type":[]string{"text/html"}, "Content-Length":[]string{"358"}, "Date":[]string{"Fri, 02 Mar 2018 01:31:16 GMT"}}, Body:http.http2transportResponseBody{cs:(*http.http2clientStream)(0xc00048a3c0)}, ContentLength:358, TransferEncoding:[]string(nil), Close:false, Uncompressed:false, Trailer:http.Header(nil), Request:(*http.Request)(0xc00010c000), TLS:(*tls.ConnectionState)(0xc0000bcb00)}
&http.Response{Status:"200 OK", StatusCode:200, Proto:"HTTP/2.0", ProtoMajor:2, ProtoMinor:0, Header:http.Header{"Content-Type":[]string{"text/html"}, "Content-Length":[]string{"358"}, "Date":[]string{"Fri, 02 Mar 2018 01:31:17 GMT"}}, Body:http.http2transportResponseBody{cs:(*http.http2clientStream)(0xc0002588c0)}, ContentLength:358, TransferEncoding:[]string(nil), Close:false, Uncompressed:false, Trailer:http.Header(nil), Request:(*http.Request)(0xc00010c000), TLS:(*tls.ConnectionState)(0xc0002f84d0)}
&http.Response{Status:"200 OK", StatusCode:200, Proto:"HTTP/2.0", ProtoMajor:2, ProtoMinor:0, Header:http.Header{"Date":[]string{"Fri, 02 Mar 2018 01:31:18 GMT"}, "Content-Type":[]string{"text/html"}, "Content-Length":[]string{"358"}}, Body:http.http2transportResponseBody{cs:(*http.http2clientStream)(0xc00025e8c0)}, ContentLength:358, TransferEncoding:[]string(nil), Close:false, Uncompressed:false, Trailer:http.Header(nil), Request:(*http.Request)(0xc00010c000), TLS:(*tls.ConnectionState)(0xc0000bc840)}
&http.Response{Status:"200 OK", StatusCode:200, Proto:"HTTP/2.0", ProtoMajor:2, ProtoMinor:0, Header:http.Header{"Content-Type":[]string{"text/html"}, "Content-Length":[]string{"358"}, "Date":[]string{"Fri, 02 Mar 2018 01:31:18 GMT"}}, Body:http.http2transportResponseBody{cs:(*http.http2clientStream)(0xc00040fa40)}, ContentLength:358, TransferEncoding:[]string(nil), Close:false, Uncompressed:false, Trailer:http.Header(nil), Request:(*http.Request)(0xc00010c000), TLS:(*tls.ConnectionState)(0xc0000bcb00)}
&http.Response{Status:"200 OK", StatusCode:200, Proto:"HTTP/2.0", ProtoMajor:2, ProtoMinor:0, Header:http.Header{"Content-Type":[]string{"text/html"}, "Content-Length":[]string{"358"}, "Date":[]string{"Fri, 02 Mar 2018 01:31:19 GMT"}}, Body:http.http2transportResponseBody{cs:(*http.http2clientStream)(0xc00040edc0)}, ContentLength:358, TransferEncoding:[]string(nil), Close:false, Uncompressed:false, Trailer:http.Header(nil), Request:(*http.Request)(0xc00010c000), TLS:(*tls.ConnectionState)(0xc0000bcb00)}
&http.Response{Status:"200 OK", StatusCode:200, Proto:"HTTP/2.0", ProtoMajor:2, ProtoMinor:0, Header:http.Header{"Content-Type":[]string{"text/html"}, "Content-Length":[]string{"358"}, "Date":[]string{"Fri, 02 Mar 2018 01:31:20 GMT"}}, Body:http.http2transportResponseBody{cs:(*http.http2clientStream)(0xc00025e8c0)}, ContentLength:358, TransferEncoding:[]string(nil), Close:false, Uncompressed:false, Trailer:http.Header(nil), Request:(*http.Request)(0xc00010c000), TLS:(*tls.ConnectionState)(0xc000492790)}

How often does this occur? Can you try running this in a loop like I did?

for ((i=0;i<10;i++)) do go run main.go;done

Is libexec a result of installing via Homebrew or some other ontap solution?

Let me page some folks, @ianlancetaylor for cgo and signals and @bradfitz for HTTP.

We both have CGO enabled but the biggest difference here is that I am on OS X El Capitan 10.11.6 (15G1611) while you are on Sierra 10.12.6. In the past we've encountered problems with Sierra, @bradfitz @ianlancetaylor perhaps this could be a useful hint?

[ericreis]

Hello @odeke-em. I'll answer each one of your questions:

How often does this occur?

I've tried like 50 times, across all the attempts to make it work (reinstalling go, upgrading macOS, etc.). All of them resulted in the same error.

Can you try running this in a loop like I did?

Just did and the outcome was the same in all the cases. The only difference between them was the goroutine number. Do you want me to post the full outcome of the 10 tries?

Is libexec a result of installing via Homebrew or some other ontap solution?

Yes, I installed go via Homebrew (brew install go). Also, I already tried with go@1.9 and go@1.8, both resulting in the same outcome.

We both have CGO enabled but the biggest difference here is that I am on OS X El Capitan 10.11.6 (15G1611) while you are on Sierra 10.12.6. In the past we've encountered problems with Sierra, @bradfitz @ianlancetaylor perhaps this could be a useful hint?

When I first tried I was on OS X El Capitan 10.11.6 too, so I think that it's not OS version related.

[odeke-em]

Great, this is awesome, thank you for answering all my questions.

Just one more, if you've got a C compiler on that Mac, could you please help me with the output of this code?

#include <stdio.h>
#include <string.h>
#include <sys/sysctl.h>

int main() {
  char str[256];
  size_t size = sizeof(str);
  memset(str, 0, size);
  sysctlbyname("kern.osrelease", str, &size, NULL, 0);
  printf("OS X release: %s\n", str);
}

[odeke-em]

$ gcc main.c -o main && ./main

Just did and the outcome was the same in all the cases. The only difference between them was the goroutine number. Do you want me to post the full outcome of the 10 tries?

I guess the entire program crashed so no need to :)

[ericreis]

@odeke-em output:OS X release: 16.7.0

[odeke-em]

Thank you @ericreis for providing as much information, it is going to be useful later on.

I have also installed Go via Homebrew like you did

and then I have run it 10 times, all passing each time

At least we've collected as much information, @ianlancetaylor and @bradfitz the floor is all yours.

[ericreis]

yeah, I really cannot understand what's going on... it seems to me that it's a problem with my machine. I was going to factory reset it and upgrade everything to see it solves the problem, but I'll wait to see if this is a real bug affecting other people. If it is, I want to help as much as I can in order to help improve the project.

[ericreis]

updating: It also happens with the following gopkg.in packages:

• gopkg.in/sourcemap.v2
• gopkg.in/yaml.v2
• gopkg.in/kataras/iris.v10

In all of them, I managed to get the packages from the latest releases (github.com/...). So I am guessing the problem is with all packages in gopkg.in.

[odeke-em]

Oh interesting. @agl @FiloSottile any ideas/insights perhaps into why crypto.FetchPEMRoots might be failing for @ericreis or perhaps something weird about those certificates from gopkg.in/*?

[ericreis]

Any news about my problem? I still haven't formatted my MacBook in order to solve this problem.

[josharian]

Any chance you could sprinkle some printfs into the C code for FetchPEMRoots to figure out which line is segfaulting, and what the arguments are? This is not my area, but having details about exactly where the failure is occurring and what data is involved may help diagnose. Do take care not to expose any private certs though. :)

[ericreis]

@josharian I am sorry for the delay in response, I was busy with work. I searched a bit for this FetchPEMRoots C code but I didn't find it. Can you give me a hint on where it should be?