Skip to content

runtime: intermittent startup panics across multiple Go applications on Windows 11 #79249

Description

@Whissi

What did you do?

I regularly run various Go-based applications on Windows 11, most notably kopia and rclone. Both are invoked normally from the command line as part of my daily workflow (kopia is used 1-2 times per day for backups). Most invocations use the official Windows release binaries; one of the traces below comes from a self-compiled kopia build (with only minimal, unrelated changes -- see commit history).

What did you expect to see?

The applications should start up reliably and execute their requested commands without crashing.

What did you see instead?

On rare occasions, the application panics immediately at startup. Across the four traces I have collected, I have seen four distinct symptoms -- but always with the same overall pattern: a fatal crash very early during program startup, at a different code location each time, that does not reproduce on retry. Re-running the exact same command immediately afterwards always succeeds.

The four observed panic types are:

  1. runtime error: invalid memory address or nil pointer dereference deep inside application code (kopia index parsing).
  2. runtime error: invalid memory address or nil pointer dereference inside an init() function parsing a static constant byte slice (rclone, golang.org/x/crypto/ssh).
  3. runtime error: slice bounds out of range [-4:] inside an init() function parsing an embedded protobuf descriptor (kopia → google.golang.org/protobufs2a-go).
  4. fatal: morestack on g0 -- a runtime invariant violation inside runtime.morestack itself, while running an init() function that parses an embedded protobuf descriptor (kopia → google.golang.org/protobufenvoyproxy/go-control-plane).

The fourth one is what prompted me to file this issue in the runtime's tracker rather than against any individual application: the crash is in the runtime's own stack-growth path, not in user code.

Frequency

The issue is rare and intermittent. With kopia running 1-2 times a day, I observe such a panic roughly once every ~15 days. I have been observing this pattern across different versions of these tools for at least the past 2 years.

System information

  • OS: Microsoft Windows 11 Pro for Workstations, Build 26200.8328 (issue observed across many builds over the past 2+ years)
  • CPU: AMD Ryzen Threadripper 3970X (AMD64 Family 23 Model 49 Stepping 0)
  • AGESA Version: CastlePeakPI-SP3r3-1.0.0.F (runtime invariant violation inside runtime.morestack happened with 1.0.0.7)
  • RAM: 128 GB ECC
  • Go versions: the official kopia/rclone binaries are built with whatever Go toolchain those projects ship with at the respective release; the morestack on g0 trace below is from a self-compiled binary using go1.24.5 (visible in the trace).

Could this be a hardware issue?

I have considered this and I'm aware ECC alone does not rule it out:

  • The system uses ECC memory and no correctable/uncorrectable errors have been logged.
  • I run PassMark MemTest86 Pro (latest version) regularly with no findings.
  • I'm aware ECC does not protect CPU caches, registers, the memory controller, or DMA paths, and that all four traces could in principle be explained by a single-bit flip somewhere along those paths.

That said, I'd appreciate the maintainers' read on the traces. The morestack on g0 one in particular looks like something other than ordinary data corruption to me, since it indicates the runtime's own scheduler bookkeeping was in an impossible state, but I'd rather defer to the experts.

Example panics

kopia 0.22.3 -- nil pointer dereference in index loading (official Windows release binary)
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x3 pc=0x7ff7d4d93add]

goroutine 1 [running]:
github.com/kopia/kopia/repo/content/index.v1ReadHeader({0x1b7588e0000?, 0xa0?, 0x3f?})
        github.com/kopia/kopia/repo/content/index/index_v1.go:408 +0x5d
github.com/kopia/kopia/repo/content/index.Open({0x1b7588e0000, 0x6746fe, 0x6746fe}, 0xc006c84b70, 0xc0001892f0)
        github.com/kopia/kopia/repo/content/index/index.go:29 +0x3c
github.com/kopia/kopia/repo/content.(*diskCommittedContentIndexCache).openIndex(0xc006b6cc60, {0x7ff7d6539418, 0xc006b6ccf0}, {0xc006aeee00, 0x3a})
        github.com/kopia/kopia/repo/content/committed_content_index_disk_cache.go:45 +0xe7
github.com/kopia/kopia/repo/content.(*committedContentIndex).merge(0xc000219cc0, {0x7ff7d6539418, 0xc006b6ccf0}, {0xc006cda488, 0x28, 0x28?})
        github.com/kopia/kopia/repo/content/committed_content_index.go:175 +0x18c
github.com/kopia/kopia/repo/content.(*committedContentIndex).use(0xc000219cc0, {0x7ff7d6539418, 0xc006b6ccf0}, {0xc006cda488, 0x28, 0x47}, {0x47?, 0x7ff7d63024f8?, 0x7ff7d767aa00?})
        github.com/kopia/kopia/repo/content/committed_content_index.go:220 +0x2c7
github.com/kopia/kopia/repo/content.(*SharedManager).loadPackIndexesLocked(0xc0004062c0, {0x7ff7d6539418, 0xc000534e10})
        github.com/kopia/kopia/repo/content/committed_read_manager.go:256 +0x6ef
github.com/kopia/kopia/repo/content.NewSharedManager({0x7ff7d6539418, 0xc000534e10}, {0x7ff7d654f110, 0xc00060fc40}, {0x7ff7d654efa8, 0xc00028ce00}, 0xc000545398, 0x2?, 0xc00034ccf0, 0xc00060f1c0)
        github.com/kopia/kopia/repo/content/committed_read_manager.go:640 +0x679
github.com/kopia/kopia/repo.openWithConfig({0x7ff7d6539418, 0xc000534e10}, {0x7ff7d654f638, 0xc000201860}, {{0xc00020b4c8, 0x4}, {0xc00020b4d0, 0xc}, 0x0, 0x0, ...}, ...)
        github.com/kopia/kopia/repo/open.go:334 +0xfe6
github.com/kopia/kopia/repo.openDirect({0x7ff7d6539418, 0xc000534e10}, {0xc00004b2a0, 0x20}, 0xc000294c40, {0xc00004d3e0, 0x2a}, 0xc0000619a0)
        github.com/kopia/kopia/repo/open.go:230 +0x274
github.com/kopia/kopia/repo.Open({0x7ff7d6539418?, 0xc000534d50?}, {0xc00004afa0, 0x20}, {0xc00004d3e0, 0x2a}, 0xc0000619a0)
        github.com/kopia/kopia/repo/open.go:136 +0x41a
github.com/kopia/kopia/cli.(*App).openRepository(0xc0002f8808, {0x7ff7d6539418, 0xc000534d50}, 0x1)
        github.com/kopia/kopia/cli/config.go:58 +0x285
github.com/kopia/kopia/cli.(*App).repositoryWriterAction.(*App).repositoryAction.func2({0x7ff7d6539418, 0xc000534d50})
        github.com/kopia/kopia/cli/app.go:529 +0x5d
github.com/kopia/kopia/cli.(*App).runAppWithContext.func1({0x7ff7d6539418?, 0xc000534d50?})
        github.com/kopia/kopia/cli/app.go:498 +0x5e
github.com/kopia/kopia/cli.(*observabilityFlags).run(0xc0002f88a8, {0x7ff7d6539418, 0xc000534d20}, {0xc00020ac30, 0xf}, 0xc000061d08)
        github.com/kopia/kopia/cli/observability_flags.go:145 +0x2ae
github.com/kopia/kopia/cli.(*App).runAppWithContext(0xc0002f8808, 0xc0003d2750, 0xc00018d7c0)
        github.com/kopia/kopia/cli/app.go:495 +0x1b2
github.com/kopia/kopia/cli.(*App).repositoryWriterAction.(*App).repositoryAction.(*App).baseActionWithContext.func3(0xc00020ac01?)
        github.com/kopia/kopia/cli/app.go:521 +0x25
github.com/alecthomas/kingpin/v2.(*actionMixin).applyActions(...)
        github.com/alecthomas/kingpin/v2@v2.4.0/actions.go:28
github.com/alecthomas/kingpin/v2.(*Application).applyActions(0xc00017e700?, 0xc00034cab0)
        github.com/alecthomas/kingpin/v2@v2.4.0/app.go:568 +0xd0
github.com/alecthomas/kingpin/v2.(*Application).execute(0xc00017e700, 0xc00034cab0, {0xc0006c4240, 0x2, 0x2})
        github.com/alecthomas/kingpin/v2@v2.4.0/app.go:398 +0x65
github.com/alecthomas/kingpin/v2.(*Application).Parse(0xc00017e700, {0xc0001b2010?, 0xc00017e700?, 0x7ff7d6508d04?})
        github.com/alecthomas/kingpin/v2@v2.4.0/app.go:230 +0x13f
main.main()
        github.com/kopia/kopia/main.go:77 +0x1cc
kopia 0.22.3 -- slice bounds out of range during protobuf init (official Windows release binary)
panic: runtime error: slice bounds out of range [-4:]

goroutine 1 [running]:
google.golang.org/protobuf/internal/filedesc.(*File).unmarshalSeed(0xc00025e000, {0x7ff7ea7f9400, 0x21d, 0x21d})
        google.golang.org/protobuf@v1.36.10/internal/filedesc/desc_init.go:174 +0x16d9
google.golang.org/protobuf/internal/filedesc.newRawFile({{0x7ff7e8f13ae0, 0x37}, {0x7ff7ea7f9400, 0x21d, 0x21d}, 0x2, 0x2, 0x0, 0x0, {0x7ff7e97e3618, ...}, ...})
        google.golang.org/protobuf@v1.36.10/internal/filedesc/desc_init.go:30 +0x1f3
google.golang.org/protobuf/internal/filedesc.Builder.Build({{0x7ff7e8f13ae0, 0x37}, {0x7ff7ea7f9400, 0x21d, 0x21d}, 0x2, 0x2, 0x0, 0x0, {0x7ff7e97e3618, ...}, ...})
        google.golang.org/protobuf@v1.36.10/internal/filedesc/build.go:105 +0xf8
google.golang.org/protobuf/internal/filetype.Builder.Build({{{0x7ff7e8f13ae0, 0x37}, {0x7ff7ea7f9400, 0x21d, 0x21d}, 0x2, 0x2, 0x0, 0x0, {0x0, ...}, ...}, ...})
        google.golang.org/protobuf@v1.36.10/internal/filetype/build.go:138 +0x185
github.com/google/s2a-go/internal/proto/common_go_proto.file_internal_proto_common_common_proto_init()
        github.com/google/s2a-go@v0.1.9/internal/proto/common_go_proto/common.pb.go:383 +0x238
github.com/google/s2a-go/internal/proto/common_go_proto.init.0()
        github.com/google/s2a-go@v0.1.9/internal/proto/common_go_proto/common.pb.go:343 +0xf
rclone 1.74.0 -- nil pointer dereference during ssh package init (official Windows release binary)
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x46 pc=0x1400e1a9e]

goroutine 1 [running]:
strings.(*Reader).ReadByte(0x3c3386a820c0?)
        strings/reader.go:71 +0x1e
math/big.scanSign({0x142c29e78, 0x3c3386a820c0})
        math/big/intconv.go:208 +0x54
math/big.(*Int).scan(0x3c3386a820a0, {0x142c29e78, 0x3c3386a820c0}, 0x10)
        math/big/intconv.go:182 +0x31
math/big.(*Int).setFromScanner(0x3c3386a820a0, {0x142c29e78, 0x3c3386a820c0}, 0x3c3386a82001?)
        math/big/int.go:506 +0x26
math/big.(*Int).SetString(0x3c3386a820a0, {0x142b43079, 0x200}, 0x10)
        math/big/int.go:500 +0x85
golang.org/x/crypto/ssh.init.2()
        golang.org/x/crypto@v0.49.0/ssh/kex.go:429 +0x451
kopia 0.21.1 (self-compiled, go1.24.5) -- `fatal: morestack on g0` during protobuf init
fatal: morestack on g0
Exception 0x80000003 0x0 0x0 0x313161
PC=0x313161

runtime.abort()
        runtime/asm_amd64.s:1162 +0x1 fp=0xc00045b1b8 sp=0xc00045b1b0 pc=0x313161
runtime.morestack()
        runtime/asm_amd64.s:598 +0x3d fp=0xc00045b1c0 sp=0xc00045b1b8 pc=0x3113dd

goroutine 1 gp=0xc0000021c0 m=0 mp=0x3398000 [running, locked to thread]:
google.golang.org/protobuf/encoding/protowire.ConsumeBytes({0x3284ae1, 0x743, 0x743})
        google.golang.org/protobuf@v1.36.6/encoding/protowire/wire.go:436 +0xae
google.golang.org/protobuf/internal/filedesc.(*File).unmarshalSeed(0xc000604000, {0x3284ae0, 0x744, 0x744})
        google.golang.org/protobuf@v1.36.6/internal/filedesc/desc_init.go:109 +0x212
google.golang.org/protobuf/internal/filedesc.newRawFile({{0x19aa560, 0x3f}, {0x3284ae0, 0x744, 0x744}, 0x0, 0x1, 0x0, 0x0, {0x223d250, ...}, ...})
        google.golang.org/protobuf@v1.36.6/internal/filedesc/desc_init.go:30 +0x1f7
google.golang.org/protobuf/internal/filedesc.Builder.Build({...})
        google.golang.org/protobuf@v1.36.6/internal/filedesc/build.go:105 +0xf8
google.golang.org/protobuf/internal/filetype.Builder.Build({{{0x19aa560, 0x3f}, {0x3284ae0, 0x744, 0x744}, ...}, ...})
        google.golang.org/protobuf@v1.36.6/internal/filetype/build.go:138 +0x17d
github.com/envoyproxy/go-control-plane/envoy/config/listener/v3.file_envoy_config_listener_v3_quic_config_proto_init()
        github.com/envoyproxy/go-control-plane/envoy@v1.32.4/config/listener/v3/quic_config.pb.go:412 +0x198
github.com/envoyproxy/go-control-plane/envoy/config/listener/v3.file_envoy_config_listener_v3_udp_listener_config_proto_init()
        github.com/envoyproxy/go-control-plane/envoy@v1.32.4/config/listener/v3/udp_listener_config.pb.go:238 +0x30
github.com/envoyproxy/go-control-plane/envoy/config/listener/v3.file_envoy_config_listener_v3_listener_proto_init()
        github.com/envoyproxy/go-control-plane/envoy@v1.32.4/config/listener/v3/listener.pb.go:1444 +0x3a
github.com/envoyproxy/go-control-plane/envoy/config/listener/v3.init.1()
        github.com/envoyproxy/go-control-plane/envoy@v1.32.4/config/listener/v3/listener.pb.go:1437 +0xf
runtime.doInit1(0x32556e0)
        runtime/proc.go:7371 +0xdd
runtime.doInit(...)
        runtime/proc.go:7338
runtime.main()
        runtime/proc.go:254 +0x325

[~80 idle GC worker / runtime housekeeping goroutines omitted]

rax     0x17
rbx     0x33982f8
rcx     0x3398000
rdx     0x0
rdi     0xc360653000
rsi     0x3398000
rbp     0xc00045b1d0
rsp     0xc00045b1b0
r8      0xc00045ae68
r9      0x0
r10     0x0
r11     0x246
r12     0x0
r13     0x0
r14     0x3396700
r15     0x0
rip     0x313161
rflags  0x202
cs      0x33
fs      0x53
gs      0x2b

(I've trimmed ~80 idle GC worker (idle) / runtime housekeeping goroutines and the user-name path components from the original trace; the original is available on request if needed.)

Metadata

Metadata

Assignees

No one assigned

    Labels

    BugReportIssues describing a possible bug in the Go implementation.WaitingForInfoIssue is not actionable because of missing required information, which needs to be provided.compiler/runtimeIssues related to the Go compiler and/or runtime.

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    Status
    No status

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions