panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x3 pc=0x7ff7d4d93add]
goroutine 1 [running]:
github.com/kopia/kopia/repo/content/index.v1ReadHeader({0x1b7588e0000?, 0xa0?, 0x3f?})
github.com/kopia/kopia/repo/content/index/index_v1.go:408 +0x5d
github.com/kopia/kopia/repo/content/index.Open({0x1b7588e0000, 0x6746fe, 0x6746fe}, 0xc006c84b70, 0xc0001892f0)
github.com/kopia/kopia/repo/content/index/index.go:29 +0x3c
github.com/kopia/kopia/repo/content.(*diskCommittedContentIndexCache).openIndex(0xc006b6cc60, {0x7ff7d6539418, 0xc006b6ccf0}, {0xc006aeee00, 0x3a})
github.com/kopia/kopia/repo/content/committed_content_index_disk_cache.go:45 +0xe7
github.com/kopia/kopia/repo/content.(*committedContentIndex).merge(0xc000219cc0, {0x7ff7d6539418, 0xc006b6ccf0}, {0xc006cda488, 0x28, 0x28?})
github.com/kopia/kopia/repo/content/committed_content_index.go:175 +0x18c
github.com/kopia/kopia/repo/content.(*committedContentIndex).use(0xc000219cc0, {0x7ff7d6539418, 0xc006b6ccf0}, {0xc006cda488, 0x28, 0x47}, {0x47?, 0x7ff7d63024f8?, 0x7ff7d767aa00?})
github.com/kopia/kopia/repo/content/committed_content_index.go:220 +0x2c7
github.com/kopia/kopia/repo/content.(*SharedManager).loadPackIndexesLocked(0xc0004062c0, {0x7ff7d6539418, 0xc000534e10})
github.com/kopia/kopia/repo/content/committed_read_manager.go:256 +0x6ef
github.com/kopia/kopia/repo/content.NewSharedManager({0x7ff7d6539418, 0xc000534e10}, {0x7ff7d654f110, 0xc00060fc40}, {0x7ff7d654efa8, 0xc00028ce00}, 0xc000545398, 0x2?, 0xc00034ccf0, 0xc00060f1c0)
github.com/kopia/kopia/repo/content/committed_read_manager.go:640 +0x679
github.com/kopia/kopia/repo.openWithConfig({0x7ff7d6539418, 0xc000534e10}, {0x7ff7d654f638, 0xc000201860}, {{0xc00020b4c8, 0x4}, {0xc00020b4d0, 0xc}, 0x0, 0x0, ...}, ...)
github.com/kopia/kopia/repo/open.go:334 +0xfe6
github.com/kopia/kopia/repo.openDirect({0x7ff7d6539418, 0xc000534e10}, {0xc00004b2a0, 0x20}, 0xc000294c40, {0xc00004d3e0, 0x2a}, 0xc0000619a0)
github.com/kopia/kopia/repo/open.go:230 +0x274
github.com/kopia/kopia/repo.Open({0x7ff7d6539418?, 0xc000534d50?}, {0xc00004afa0, 0x20}, {0xc00004d3e0, 0x2a}, 0xc0000619a0)
github.com/kopia/kopia/repo/open.go:136 +0x41a
github.com/kopia/kopia/cli.(*App).openRepository(0xc0002f8808, {0x7ff7d6539418, 0xc000534d50}, 0x1)
github.com/kopia/kopia/cli/config.go:58 +0x285
github.com/kopia/kopia/cli.(*App).repositoryWriterAction.(*App).repositoryAction.func2({0x7ff7d6539418, 0xc000534d50})
github.com/kopia/kopia/cli/app.go:529 +0x5d
github.com/kopia/kopia/cli.(*App).runAppWithContext.func1({0x7ff7d6539418?, 0xc000534d50?})
github.com/kopia/kopia/cli/app.go:498 +0x5e
github.com/kopia/kopia/cli.(*observabilityFlags).run(0xc0002f88a8, {0x7ff7d6539418, 0xc000534d20}, {0xc00020ac30, 0xf}, 0xc000061d08)
github.com/kopia/kopia/cli/observability_flags.go:145 +0x2ae
github.com/kopia/kopia/cli.(*App).runAppWithContext(0xc0002f8808, 0xc0003d2750, 0xc00018d7c0)
github.com/kopia/kopia/cli/app.go:495 +0x1b2
github.com/kopia/kopia/cli.(*App).repositoryWriterAction.(*App).repositoryAction.(*App).baseActionWithContext.func3(0xc00020ac01?)
github.com/kopia/kopia/cli/app.go:521 +0x25
github.com/alecthomas/kingpin/v2.(*actionMixin).applyActions(...)
github.com/alecthomas/kingpin/v2@v2.4.0/actions.go:28
github.com/alecthomas/kingpin/v2.(*Application).applyActions(0xc00017e700?, 0xc00034cab0)
github.com/alecthomas/kingpin/v2@v2.4.0/app.go:568 +0xd0
github.com/alecthomas/kingpin/v2.(*Application).execute(0xc00017e700, 0xc00034cab0, {0xc0006c4240, 0x2, 0x2})
github.com/alecthomas/kingpin/v2@v2.4.0/app.go:398 +0x65
github.com/alecthomas/kingpin/v2.(*Application).Parse(0xc00017e700, {0xc0001b2010?, 0xc00017e700?, 0x7ff7d6508d04?})
github.com/alecthomas/kingpin/v2@v2.4.0/app.go:230 +0x13f
main.main()
github.com/kopia/kopia/main.go:77 +0x1cc
What did you do?
I regularly run various Go-based applications on Windows 11, most notably kopia and rclone. Both are invoked normally from the command line as part of my daily workflow (kopia is used 1-2 times per day for backups). Most invocations use the official Windows release binaries; one of the traces below comes from a self-compiled kopia build (with only minimal, unrelated changes -- see commit history).
What did you expect to see?
The applications should start up reliably and execute their requested commands without crashing.
What did you see instead?
On rare occasions, the application panics immediately at startup. Across the four traces I have collected, I have seen four distinct symptoms -- but always with the same overall pattern: a fatal crash very early during program startup, at a different code location each time, that does not reproduce on retry. Re-running the exact same command immediately afterwards always succeeds.
The four observed panic types are:
runtime error: invalid memory address or nil pointer dereferencedeep inside application code (kopia index parsing).runtime error: invalid memory address or nil pointer dereferenceinside aninit()function parsing a static constant byte slice (rclone,golang.org/x/crypto/ssh).runtime error: slice bounds out of range [-4:]inside aninit()function parsing an embedded protobuf descriptor (kopia →google.golang.org/protobuf→s2a-go).fatal: morestack on g0-- a runtime invariant violation insideruntime.morestackitself, while running aninit()function that parses an embedded protobuf descriptor (kopia →google.golang.org/protobuf→envoyproxy/go-control-plane).The fourth one is what prompted me to file this issue in the runtime's tracker rather than against any individual application: the crash is in the runtime's own stack-growth path, not in user code.
Frequency
The issue is rare and intermittent. With kopia running 1-2 times a day, I observe such a panic roughly once every ~15 days. I have been observing this pattern across different versions of these tools for at least the past 2 years.
System information
runtime.morestackhappened with 1.0.0.7)morestack on g0trace below is from a self-compiled binary usinggo1.24.5(visible in the trace).Could this be a hardware issue?
I have considered this and I'm aware ECC alone does not rule it out:
That said, I'd appreciate the maintainers' read on the traces. The
morestack on g0one in particular looks like something other than ordinary data corruption to me, since it indicates the runtime's own scheduler bookkeeping was in an impossible state, but I'd rather defer to the experts.Example panics
kopia 0.22.3 -- nil pointer dereference in index loading (official Windows release binary)
kopia 0.22.3 -- slice bounds out of range during protobuf init (official Windows release binary)
rclone 1.74.0 -- nil pointer dereference during ssh package init (official Windows release binary)
kopia 0.21.1 (self-compiled, go1.24.5) -- `fatal: morestack on g0` during protobuf init
(I've trimmed ~80 idle
GC worker (idle)/ runtime housekeeping goroutines and the user-name path components from the original trace; the original is available on request if needed.)