runtime: spurious crashes related to timers in go 1.23.0 #69625

fiber · 2024-09-25T14:17:10Z

Go version

go version go1.23.0 linux/amd64

Output of `go env` in your module/workspace:

GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/root/.cache/go-build'
GOENV='/root/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/root/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/root/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/go'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.23.0'
GODEBUG=''
GOTELEMETRY='local'
GOTELEMETRYDIR='/root/.config/go/telemetry'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/gocode/src/go.gtt.net/go.mod'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build3222024183=/tmp/go-build -gno-record-gcc-switches'

What did you do?

recompiled go code that has been running fine for a long time.

What did you see happen?

we are experiencing frequent crashes that follow this pattern:

unexpected fault address 0x30000000009
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x1 addr=0x30000000009 pc=0x466ed2]

goroutine 1014077 gp=0xc0073fd340 m=251 mp=0xc01370b508 [running]:
runtime.throw({0x1d00572?, 0xc01b3a0d20?})
        /go/src/runtime/panic.go:1067 +0x48 fp=0xc01b3a0af0 sp=0xc01b3a0ac0 pc=0x47e548
runtime.sigpanic()
        /go/src/runtime/signal_unix.go:914 +0x26c fp=0xc01b3a0b50 sp=0xc01b3a0af0 pc=0x4808ac
runtime.(*timer).maybeRunChan(0x0?)
        /go/src/runtime/time.go:1244 +0x12 fp=0xc01b3a0b80 sp=0xc01b3a0b50 pc=0x466ed2
runtime.selectgo(0xc01b3a0d20, 0xc01b3a0ce0, 0x22?, 0x1, 0x81ec0a?, 0x1)
        /go/src/runtime/select.go:177 +0x13f fp=0xc01b3a0ca8 sp=0xc01b3a0b80 pc=0x459b1f
io.(*pipe).write(0xc0f0037560, {0xc078532000, 0x1000, 0x1000})
        /go/src/io/pipe.go:86 +0x1e5 fp=0xc01b3a0d80 sp=0xc01b3a0ca8 pc=0x4c12c5
io.(*PipeWriter).Write(0x41f2cb?, {0xc078532000?, 0xc04473bd40?, 0xc0072bc430?})
        /go/src/io/pipe.go:161 +0x1a fp=0xc01b3a0db0 sp=0xc01b3a0d80 pc=0x4c173a
bufio.(*Writer).Flush(0xc065c60040)
        /go/src/bufio/bufio.go:639 +0x55 fp=0xc01b3a0df8 sp=0xc01b3a0db0 pc=0x4d6ef5
bufio.(*Writer).Write(0xc065c60040, {0xc0410de480?, 0x1cb89e0?, 0xc01b3a0fa8?})
        /go/src/bufio/bufio.go:683 +0xcd fp=0xc01b3a0e58 sp=0xc01b3a0df8 pc=0x4d71ad

What did you expect to see?

no crash.

The text was updated successfully, but these errors were encountered:

gabyhelp · 2024-09-25T14:17:22Z

Related Issues and Documentation

runtime #60196 (closed)
runtime: runtime crashes, reproducer same as #20427 #36371 (closed)
fatal error: unexpected signal during runtime execution [signal SIGSEGV: segmentation violation code=0x2 addr=0xc002000000 pc=0x46e698] Linux version 5.10.135-yocto-standard #58377 (closed)
runtime/traceback: segmentation violation failures from unwinding crash #64030
runtime: SIGSEGV in the new timer code #35367 (closed)
runtime: crash without a message, potentially in the signal handler #56701 (closed)
runtime: panic only with GOGC=on and highly concurrent programs also using cgo #30276 (closed)
runtime: crash when trying go run or go build #37638 (closed)
cmd/go: get in docker buildx gets SIGSEGV error #40969 (closed)
runtime: crash in race detector when execution tracer reads from CPU profile buffer #65607 (closed)

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

seankhliao · 2024-09-25T14:32:35Z

are you copying timer values and not pointers?

we need a reproducer for this to be actionable

fiber · 2024-09-25T14:57:55Z

This is a sizeable piece of code. I would not exclude bugs, however it's unlikely that we I am copying timer values, since we're using *time.Timer/*time.Ticker everywhere.

the below crashes may be related, although I don't understand how.
I am rolling back to 1.22.7 to check

unexpected fault address 0x30000000009
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x1 addr=0x30000000009 pc=0x466ed2]

goroutine 1014077 gp=0xc0073fd340 m=251 mp=0xc01370b508 [running]:
runtime.throw({0x1d00572?, 0xc01b3a0d20?})
        /go/src/runtime/panic.go:1067 +0x48 fp=0xc01b3a0af0 sp=0xc01b3a0ac0 pc=0x47e548
runtime.sigpanic()
        /go/src/runtime/signal_unix.go:914 +0x26c fp=0xc01b3a0b50 sp=0xc01b3a0af0 pc=0x4808ac
runtime.(*timer).maybeRunChan(0x0?)
        /go/src/runtime/time.go:1244 +0x12 fp=0xc01b3a0b80 sp=0xc01b3a0b50 pc=0x466ed2
runtime.selectgo(0xc01b3a0d20, 0xc01b3a0ce0, 0x22?, 0x1, 0x81ec0a?, 0x1)
        /go/src/runtime/select.go:177 +0x13f fp=0xc01b3a0ca8 sp=0xc01b3a0b80 pc=0x459b1f
io.(*pipe).write(0xc0f0037560, {0xc078532000, 0x1000, 0x1000})
        /go/src/io/pipe.go:86 +0x1e5 fp=0xc01b3a0d80 sp=0xc01b3a0ca8 pc=0x4c12c5
io.(*PipeWriter).Write(0x41f2cb?, {0xc078532000?, 0xc04473bd40?, 0xc0072bc430?})
        /go/src/io/pipe.go:161 +0x1a fp=0xc01b3a0db0 sp=0xc01b3a0d80 pc=0x4c173a
bufio.(*Writer).Flush(0xc065c60040)
        /go/src/bufio/bufio.go:639 +0x55 fp=0xc01b3a0df8 sp=0xc01b3a0db0 pc=0x4d6ef5
bufio.(*Writer).Write(0xc065c60040, {0xc0410de480?, 0x1cb89e0?, 0xc01b3a0fa8?})
        /go/src/bufio/bufio.go:683 +0xcd fp=0xc01b3a0e58 sp=0xc01b3a0df8 pc=0x4d71ad
...


unexpected fault address 0x0
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x80 addr=0x0 pc=0x8e0055]

goroutine 72153 gp=0xc004e9bc00 m=518 mp=0xc013509508 [running]:
runtime.throw({0x1d105d2?, 0xc0514c19c9?})
        /go/src/runtime/panic.go:1067 +0x48 fp=0xc0076ca970 sp=0xc0076ca940 pc=0x47e548
runtime.sigpanic()
        /go/src/runtime/signal_unix.go:914 +0x26c fp=0xc0076ca9d0 sp=0xc0076ca970 pc=0x4808ac
google.golang.org/protobuf/internal/impl.sizeInt64PackedSlice({0xc0514c19c8?}, 0x9b?, {0x80?})
        /root/go/pkg/mod/google.golang.org/protobuf@v1.34.2/internal/impl/codec_gen.go:2061 +0x15 fp=0xc0076ca9d8 sp=0xc0076ca9d0 pc=0x8e0055
google.golang.org/protobuf/internal/impl.(*MessageInfo).sizePointerSlow(0xc000262150, {0x1d659d2?}, {0x88?})
        /root/go/pkg/mod/google.golang.org/protobuf@v1.34.2/internal/impl/encode.go:82 +0xd3 fp=0xc0076caa30 sp=0xc0076ca9d8 pc=0x9013d3
google.golang.org/protobuf/internal/impl.(*MessageInfo).sizePointer(0x0?, {0x0?}, {0x0?})
        /root/go/pkg/mod/google.golang.org/protobuf@v1.34.2/internal/impl/encode.go:59 +0x6f fp=0xc0076caa68 sp=0xc0076caa30 pc=0x9012af
google.golang.org/protobuf/internal/impl.sizeMessageSliceInfo({0xc004e9bc00?}, 0xc00c9195d0, {0x49?})
        /root/go/pkg/mod/google.golang.org/protobuf@v1.34.2/internal/impl/codec_field.go:486 +0x50 fp=0xc0076caab8 sp=0xc0076caa68 pc=0x8d5370
google.golang.org/protobuf/internal/impl.(*MessageInfo).sizePointerSlow(0xc000262298, {0xc0514c1950?}, {0x89?})
        /root/go/pkg/mod/google.golang.org/protobuf@v1.34.2/internal/impl/encode.go:82 +0xd3 fp=0xc0076cab10 sp=0xc0076caab8 pc=0x9013d3
google.golang.org/protobuf/internal/impl.(*MessageInfo).sizePointer(0x92?, {0x92?}, {0xb0?})
        /root/go/pkg/mod/google.golang.org/protobuf@v1.34.2/internal/impl/encode.go:59 +0x6f fp=0xc0076cab48 sp=0xc0076cab10 pc=0x9012af
google.golang.org/protobuf/internal/impl.(*MessageInfo).size(0x5d7?, {{}, {0x22d52e0?, 0xc049ec4000?}, 0x84?})
        /root/go/pkg/mod/google.golang.org/protobuf@v1.34.2/internal/impl/encode.go:40 +0x52 fp=0xc0076cab70 sp=0xc0076cab48 pc=0x9011d2
google.golang.org/protobuf/internal/impl.(*MessageInfo).size-fm({{}, {0x22d52e0?, 0xc049ec4000?}, 0x48?})
        <autogenerated>:1 +0x35 fp=0xc0076caba0 sp=0xc0076cab70 pc=0x91fdb5
google.golang.org/protobuf/proto.MarshalOptions.marshal({{}, 0x0?, 0xbc?, 0xe9?}, {0x0, 0x0, 0x0}, {0x22d52e0, 0xc049ec4000})
        /root/go/pkg/mod/google.golang.org/protobuf@v1.34.2/proto/encode.go:184 +0x11b fp=0xc0076cac70 sp=0xc0076caba0 pc=0x8a815b
google.golang.org/protobuf/proto.Marshal({0x22b1580, 0xc049ec4000})
        /root/go/pkg/mod/google.golang.org/protobuf@v1.34.2/proto/encode.go:110 +0x50 fp=0xc0076cacc0 sp=0xc0076cac70 pc=0x8a7c90



unexpected fault address 0x0
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x80 addr=0x0 pc=0x8e0055]

goroutine 63842 gp=0xc0025eae00 m=312 mp=0xc012afdc08 [running]:
runtime.throw({0x1d00572?, 0xc01c07046f?})
        /go/src/runtime/panic.go:1067 +0x48 fp=0xc000e1b970 sp=0xc000e1b940 pc=0x47e548
runtime.sigpanic()
        /go/src/runtime/signal_unix.go:914 +0x26c fp=0xc000e1b9d0 sp=0xc000e1b970 pc=0x4808ac
google.golang.org/protobuf/internal/impl.sizeInt64PackedSlice({0xc01c07046e?}, 0x14b?, {0x0?})
        /root/go/pkg/mod/google.golang.org/protobuf@v1.34.2/internal/impl/codec_gen.go:2061 +0x15 fp=0xc000e1b9d8 sp=0xc000e1b9d0 pc=0x8e0055
google.golang.org/protobuf/internal/impl.(*MessageInfo).sizePointerSlow(0xc0002f0150, {0x1d55878?}, {0x88?})
        /root/go/pkg/mod/google.golang.org/protobuf@v1.34.2/internal/impl/encode.go:82 +0xd3 fp=0xc000e1ba30 sp=0xc000e1b9d8 pc=0x9013d3
google.golang.org/protobuf/internal/impl.(*MessageInfo).sizePointer(0x0?, {0x0?}, {0x0?})
        /root/go/pkg/mod/google.golang.org/protobuf@v1.34.2/internal/impl/encode.go:59 +0x6f fp=0xc000e1ba68 sp=0xc000e1ba30 pc=0x9012af
google.golang.org/protobuf/internal/impl.sizeMessageSliceInfo({0xc0025eae00?}, 0xc0058300d0, {0x49?})
        /root/go/pkg/mod/google.golang.org/protobuf@v1.34.2/internal/impl/codec_field.go:486 +0x50 fp=0xc000e1bab8 sp=0xc000e1ba68 pc=0x8d5370
google.golang.org/protobuf/internal/impl.(*MessageInfo).sizePointerSlow(0xc0002f0298, {0xc01c070300?}, {0x7f?})
        /root/go/pkg/mod/google.golang.org/protobuf@v1.34.2/internal/impl/encode.go:82 +0xd3 fp=0xc000e1bb10 sp=0xc000e1bab8 pc=0x9013d3
google.golang.org/protobuf/internal/impl.(*MessageInfo).sizePointer(0x188?, {0x188?}, {0xb0?})
        /root/go/pkg/mod/google.golang.org/protobuf@v1.34.2/internal/impl/encode.go:59 +0x6f fp=0xc000e1bb48 sp=0xc000e1bb10 pc=0x9012af
google.golang.org/protobuf/internal/impl.(*MessageInfo).size(0x400?, {{}, {0x22c3760?, 0xc018dc5930?}, 0xc?})
        /root/go/pkg/mod/google.golang.org/protobuf@v1.34.2/internal/impl/encode.go:40 +0x52 fp=0xc000e1bb70 sp=0xc000e1bb48 pc=0x9011d2
google.golang.org/protobuf/internal/impl.(*MessageInfo).size-fm({{}, {0x22c3760?, 0xc018dc5930?}, 0x48?})
        <autogenerated>:1 +0x35 fp=0xc000e1bba0 sp=0xc000e1bb70 pc=0x91fdb5
google.golang.org/protobuf/proto.MarshalOptions.marshal({{}, 0x0?, 0xae?, 0x5e?}, {0x0, 0x0, 0x0}, {0x22c3760, 0xc018dc5930})
        /root/go/pkg/mod/google.golang.org/protobuf@v1.34.2/proto/encode.go:184 +0x11b fp=0xc000e1bc70 sp=0xc000e1bba0 pc=0x8a815b
google.golang.org/protobuf/proto.Marshal({0x229fb40, 0xc018dc5930})
        /root/go/pkg/mod/google.golang.org/protobuf@v1.34.2/proto/encode.go:110 +0x50 fp=0xc000e1bcc0 sp=0xc000e1bc70 pc=0x8a7c90

ianlancetaylor · 2024-09-25T20:23:31Z

CC @golang/runtime

A fault address like 0x30000000009 looks like memory corruption. It's not necessarily in code related to timers; that may just be where it is showing up. Look closely at any uses of unsafe or cgo.

gopherbot · 2024-10-25T15:10:06Z

Timed out in state WaitingForInfo. Closing.

(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)

seankhliao added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Sep 25, 2024

seankhliao changed the title ~~runtime/time.go: spurious crashes related to timers in go 1.23.0~~ time: spurious crashes related to timers in go 1.23.0 Sep 25, 2024

seankhliao added WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. and removed WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. labels Sep 25, 2024

ianlancetaylor changed the title ~~time: spurious crashes related to timers in go 1.23.0~~ runtime: spurious crashes related to timers in go 1.23.0 Sep 25, 2024

gopherbot closed this as not planned Won't fix, can't repro, duplicate, stale Oct 25, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

runtime: spurious crashes related to timers in go 1.23.0 #69625

runtime: spurious crashes related to timers in go 1.23.0 #69625

fiber commented Sep 25, 2024 •

edited by prattmic

Loading

gabyhelp commented Sep 25, 2024

seankhliao commented Sep 25, 2024

fiber commented Sep 25, 2024 •

edited by prattmic

Loading

ianlancetaylor commented Sep 25, 2024

gopherbot commented Oct 25, 2024

runtime: spurious crashes related to timers in go 1.23.0 #69625

runtime: spurious crashes related to timers in go 1.23.0 #69625

Comments

fiber commented Sep 25, 2024 • edited by prattmic Loading

Go version

Output of go env in your module/workspace:

What did you do?

What did you see happen?

What did you expect to see?

gabyhelp commented Sep 25, 2024

seankhliao commented Sep 25, 2024

fiber commented Sep 25, 2024 • edited by prattmic Loading

ianlancetaylor commented Sep 25, 2024

gopherbot commented Oct 25, 2024

fiber commented Sep 25, 2024 •

edited by prattmic

Loading

Output of `go env` in your module/workspace:

fiber commented Sep 25, 2024 •

edited by prattmic

Loading