New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
random panics when performing provider verification #269
Comments
OK, so I’ve been seeing them indeterminately also but have never been able to put my finger on why. I’ve shared with my colleague who has more experience with low level stuff like this, and might have some thoughts on it also. For context, i’ve not seen this problem in other languages (e.g. JS, which I also maintain). This seems to suggest why: https://docs.studygolang.com/pkg/os/signal/#hdr-Go_programs_that_use_cgo_or_SWIG I can’t see how we can configure the SA_ONSTACK flag in the rust core (the core engine we ship to all languages), which would be great to be able to test to see if it does resolve. |
👋 Hi! The 'smartbear-supported' label has just been added to this issue, which will create an internal tracking ticket in PactFlow's Jira (PACT-813). We will use this to prioritise and assign a team member to this task. All activity will be public on this ticket. For now, sit tight and we'll update this ticket once we have more information on the next steps. |
Upgrading glibc to pull in a fix for go interoperability makes it go away in my debian container: The native rust code seems to set the flag. Could use something like eBPF to grab a stack trace when pthread_create is called to further narrow down the culprit. |
Some further debugging notes. It only seems to affect plugins, and only during verification. If I run the I was hoping a fix (e.g. pact-foundation/pact-reference@0af0035 and those just after) might have helped here, but it seems like the issue is separate.
or in the case of the grpc tests:
NOTE: I've tried not calling |
Further update - I attempted to trace this using dtrace/dtruss, however you can't instrument rosetta runs. I then installed go in a native Apple environment and ran into permission issues, but I thought before I boot into recovery and see if I can work around that, I would see if the problem is replicated - alas, running the examples on a loop I couldn't get it to repro in > 100 iterations. So I think plausible some rosetta translation magic is part of this issue, or there is a separate issue with the It's interesting that both variants link the same libraries (presumably because Rosetta is just an emulation layer): arm
x86_64
|
Just reporting that we are piloting Pact for our Go GRPC services and are running into the same error. Currently CI is executing on x86_64;
It happens relatively often, so I am worried that if devs have to continually re-run jobs, it will impact our CI flows' reliability/DX. |
Thanks Matt, sadly this has been a nasty one to try and chase down. |
Software versions
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/dev/.cache/go-build"
GOENV="/home/dev/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/dev/go/pkg/mod"
GONOPROXY="github.hpe.com"
GONOSUMDB="github.hpe.com"
GOOS="linux"
GOPATH="/home/dev/go"
GOPRIVATE="github.hpe.com"
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.19.2"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/dev/ws/sc-authz/go.mod"
GOWORK=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build4109696144=/tmp/go-build -gno-record-gcc-switches"
Expected behaviour
Pact provider verification for gRPC contract runs successfully.
Actual behaviour
Randomly fails due to uncaught signal 17.
Steps to reproduce
Run
go test -count=1 .
in 2.x.x branch under folder examples/grpc repeatedly until you see the panic.Relevent log files
The text was updated successfully, but these errors were encountered: