crypto/tls: Seg Fault In TLS Handshake in RKE

[blewis12]

Go version

go1.25.7

Output of go env in your module/workspace:

GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/root/.cache/go-build'
GOENV='/root/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/root/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/root/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/lib/go-1.22'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/usr/lib/go-1.22/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.22.2'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/dev/null'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build3015496837=/tmp/go-build -gno-record-gcc-switches'

What did you do?

Unfortunately this is hard to reproduce since it's happening on an external system - I am mostly submitting this just in case others have run into this issue and it's reproducible for them.

We have a case where we see a seg fault coming from the crypto/tls module in the go library, coming from Grafana Alloy deployed on a v1.32.5+rke2r1 rancher kubernetes cluster. We are aware that the go version in that cluster server is go1.23.x - and when we downgrade Alloy to a version that contains that same version of go, we see these seg faults disappear

If anyone has any guidance on what could cause behavior like this that would be much appreciated

What did you see happen?

The main error message is

unexpected fault address 0x8541
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8541 pc=0x482047]

goroutine 44 gp=0xc000e6ca80 m=3 mp=0xc0000e9008 [running]:
runtime.throw({0xd548f55?, 0xbfc6680?})
        /usr/local/go/src/runtime/panic.go:1094 +0x48 fp=0xc001a0b920 sp=0xc001a0b8f0 pc=0x485668
runtime.sigpanic()
        /usr/local/go/src/runtime/signal_unix.go:939 +0x26c fp=0xc001a0b980 sp=0xc001a0b920 pc=0x487ccc
runtime.getitab(0xbfc6680, 0x852d, 0x10?)
        /usr/local/go/src/runtime/iface.go:50 +0x27 fp=0xc001a0b9c8 sp=0xc001a0b980 pc=0x482047
runtime.typeAssert(0x15fda680, 0x852d)
        /usr/local/go/src/runtime/iface.go:474 +0x3f fp=0xc001a0ba08 sp=0xc001a0b9c8 pc=0x41cc1f
crypto/tls.(*halfConn).setErrorLocked(...)
        /usr/local/go/src/crypto/tls/conn.go:197
crypto/tls.(*Conn).unmarshalHandshakeMessage(0xc0011fc008, {0xc0010ba080, 0xc, 0xc000c2ec00?}, {0x0, 0x0})
        /usr/local/go/src/crypto/tls/conn.go:1183 +0x607 fp=0xc001a0ba88 sp=0xc001a0ba08 pc=0x7bd3e7
crypto/tls.(*Conn).readHandshake(0xc0011fc008, {0x0, 0x0})
        /usr/local/go/src/crypto/tls/conn.go:1121 +0x238 fp=0xc001a0bb18 sp=0xc001a0ba88 pc=0x7bcc18
crypto/tls.(*Conn).clientHandshake(0xc0011fc008, {0xec2a3d8, 0xc0008d2c80})
        /usr/local/go/src/crypto/tls/handshake_client.go:328 +0x505 fp=0xc001a0bd40 sp=0xc001a0bb18 pc=0x7c85c5
crypto/tls.(*Conn).clientHandshake-fm({0xec2a3d8?, 0xc0008d2c80?})
        <autogenerated>:1 +0x33 fp=0xc001a0bd68 sp=0xc001a0bd40 pc=0x8063d3
crypto/tls.(*Conn).handshakeContext(0xc0011fc008, {0xec2a3d8, 0xc0019b1ea0})
        /usr/local/go/src/crypto/tls/conn.go:1575 +0x372 fp=0xc001a0bf70 sp=0xc001a0bd68 pc=0x7bf572
crypto/tls.(*Conn).HandshakeContext(...)
        /usr/local/go/src/crypto/tls/conn.go:1515
net/http.(*persistConn).addTLS.func2()
        /usr/local/go/src/net/http/transport.go:1706 +0x6e fp=0xc001a0bfe0 sp=0xc001a0bf70 pc=0x8988ee
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc001a0bfe8 sp=0xc001a0bfe0 pc=0x48e441
created by net/http.(*persistConn).addTLS in goroutine 250
        /usr/local/go/src/net/http/transport.go:1702 +0x309

But i've also attached a full stack trace here

debug-logs.log

What did you expect to see?

No seg fault, but an error message displaying the real cause (perhaps some incompatibility issue?)

[gabyhelp]

Related Issues

• crypto/x509: panic during tls handshake from http client #70374 (closed)
• crypto/tls: segfault when calling `tlsrsakex.IncNonDefault()` #65991 (closed)
• runtime: Panics and segmentation violation (Go 1.19.8) #61552 (closed)
• runtime: segmentation violation with linkshared #44117 (closed)
• net/http: crash in Transport.queueForIdleConn(...) #42810 (closed)

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[cagedmantis]

cc @FiloSottile @rolandshoemaker @cpu @golang/security

[rolandshoemaker]

This seems like a runtime issue, based on the stack trace? cc @golang/runtime

[FiloSottile]

Yeah the panic seems to come from the second line in

func (hc *halfConn) setErrorLocked(err error) error {
	if e, ok := err.(net.Error); ok {

and it looks like memory corruption to me.

Does it always present with the same stack trace?

[dmitshur]

Are you able to reproduce this when using either Go 1.26 or 1.25? In the go env output, it looked like the Go toolchain version was 1.22.

[randall77]

runtime.typeAssert(0x15fda680, 0x852d)
/usr/local/go/src/runtime/iface.go:474 +0x3f fp=0xc001a0ba08 sp=0xc001a0b9c8 pc=0x41cc1f

The second argument here looks bad. It's supposed to be the address of the net.Conn type descriptor.
I don't know how that could be bad, it gets loaded into a register almost immediately before the call. Maybe the binary is corrupted?

[blewis12]

Thanks for the input everyone!

Does it always present with the same stack trace?

I've attached another stack trace, it does always seem to originate from the TLS handshake code 🤔
stack-trace.txt

Are you able to reproduce this when using either Go 1.26 or 1.25? In the go env output, it looked like the Go toolchain version was 1.22

Hm i'm a bit confused, the stack trace comes from running Grafana Alloy v1.13.1 which ships with go 1.25. Let me verify how we resolved the envs in the environment this is occurring in. Whenever we downgrade to Alloy which contains go 1.23 which is the same version as the k8s server implementation, we stop getting seg faults

edit: ah, in order to run go env we had to install go in a running container, which must have been the older version - but the version of go running in the alloy binary is 1.25

Maybe the binary is corrupted?

How might something like this happen?