runtime: possible GC bug with msan

[RaduBerinde]

Go version

go version go1.25.3 linux/amd64

Output of go env in your module/workspace:

AR='ar'
CC='gcc'
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_ENABLED='1'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
CXX='g++'
GCCGO='gccgo'
GO111MODULE=''
GOAMD64='v1'
GOARCH='amd64'
GOAUTH='netrc'
GOBIN=''
GOCACHE='/home/runner/.cache/go-build'
GOCACHEPROG=''
GODEBUG=''
GOENV='/home/runner/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFIPS140='off'
GOFLAGS=''
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build1722691548=/tmp/go-build -gno-record-gcc-switches'
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMOD='/work/go.mod'
GOMODCACHE='/home/runner/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/home/runner/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/local/go'
GOSUMDB='sum.golang.org'
GOTELEMETRY='local'
GOTELEMETRYDIR='/home/runner/.config/go/telemetry'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/usr/local/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.25.3'
GOWORK=''
PKG_CONFIG='pkg-config'

What did you do?

I have an uninitialized memory crash with -msan that I can only reproduce in Github CI and using a docker container that mirrors the CI runner.

This issue is a request to guide me towards obtaining more information.

I'll start with the relevant parts of my program. The relevant source code is here: https://github.com/RaduBerinde/pebble/blob/e016d44c914386a5b45e7956d5d5dce1aa5090d8/internal/deletepacer/delete_pacer.go#L236

Basically I have a queue storing entries of this type:

type queueEntry struct {
	ObsoleteFile
	JobID int
}
type ObsoleteFile struct {
	FileType base.FileType
	FS       vfs.FS
	Path     string
	FileNum  base.DiskFileNum
	FileSize uint64 // approx for log files
	IsLocal  bool
}

The queue is implemented using a structure (https://github.com/RaduBerinde/pebble/blob/e016d44c914386a5b45e7956d5d5dce1aa5090d8/internal/deletepacer/queue.go) which will essentially allocate this object (with T = queueEntry):

type queueNode[T any] struct {
	buf       [queueNodeSize]T
	head, len int32
	next      *queueNode[T]
}

There is something important about this specific structure. If I replace this queue with a simple slice I can no longer reproduce the failure.

In my reproduction case, we push exactly one entry into this queue. A single background goroutine (mainLoop) pops it from the queue - first it makes a copy here: https://github.com/RaduBerinde/pebble/blob/e016d44c914386a5b45e7956d5d5dce1aa5090d8/internal/deletepacer/delete_pacer.go#L236

The problem is with the file.Path field. At some later point, when we actually try to delete the file, msan complains that this string is uninitialized. The string is generated via fmt.Sprintf() and passed through a path.Join; it's a simple heap string, no unsafe shenanigans. I made sure the string pointer is correct (same one that was enqueued).

I cannot reproduce if I turn the GC off. I used runtime.trace and GODEBUG=traceallocfree=1 and confirmed that at some point after we pop the element from the queue, the string object gets freed.

I figured out that if I add a runtime.GC call right after the pop (this is the code version I linked), the GC finds this problem right there:

file to delete _meta/foo/standard-012/data/000003.log 0xc00048a030 38

PushBack 0xc00048a030 _meta/foo/standard-012/data/000003.log
notification wait end


popped
 - [JOB 6] compacting(move) L0 [000006] (3.2KB) Score=0.00 + L6 [] (0B) Score=0.00; OverlappingRatio: Single 0.00, Multi 0.00
runtime: marked free object in span 0x7a50dc78d660, elemsize=48 freeindex=1 (bad use of unsafe.Pointer or having race conditions? try -d=checkptr or -race)
0xc00048a000 alloc marked  
0xc00048a030 free  marked   zombie              <-----------

I looked at the assembly and found where the structure is stored on the stack, and later we retrieve the string from the stack. So the pointer is on the stack for this time period and does not change. I looked at the stkobj map and from my understanding it does mark that stack slot as containing a pointer.

This is part of a larger codebase that does employ various unsafe tricks. However, I am able to reproduce with a fairly small amount of code actually running in-between the time we push the entry onto the queue and the failure, and I couldn't find any unsafe use there.

I was never able to reproduce any failure without -msan (I tried -race, -asan). I am suspecting some obscure bug that is specific to msan integration.

I can provide instructions on how I reproduced, but it would be pretty tedious so I believe at this point it would be easiest if I did the work to get more information. I would appreciate some guidance on how to narrow down the problem.

What did you see happen?

Uninitialized memory crashes, and later (after adding runtime.GC() at the right place) fatal error: found pointer to free object

What did you expect to see?

No failure.

[RaduBerinde]

Here is the relevant part of the -gcflags=all=-S output: https://gist.github.com/RaduBerinde/6926260c67c06fdfc61171fdc1cb392b

The problematic string pointer is stored in ~r0+416(SP):
https://gist.github.com/RaduBerinde/6926260c67c06fdfc61171fdc1cb392b#file-gistfile1-txt-L566

[gabyhelp]

Related Issues

• cmd/cgo: memory sanitizer error due to uninitialized memory in C.CString when using cgo with sanitizers enabled #60912 (closed)
• runtime: marked free object in span #44614 (closed)
• cmd/go: -msan failure when rebuilding runtime/cgo with -fsanitize=memory #13815 (closed)

Related Discussions

• What is the correct usage for the -msan flag?
• Crash using gccheckmark=1, possible GC bug?

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[RaduBerinde]

I looked at HeapObjectFree traces again: the *queueNode pointer is not getting freed by GC.

[thepudds]

Hi @RaduBerinde, do you know if this reproduces on arm64, and/or with Go 1.24?

Separately, here are some triage-level things to possibly try.

I don't know that any individual test will have a high probability of pointing directly at the problem, but the absence of the problem triggering might be a hint, and these might hopefully be simple experiments you can try in the background without needing to do much analysis aside from seeing whether or not the problem triggers:

• try with Go tip (including without any extra diagnostic flags or sanitizers).

  • There are enough changes in tip (e.g., green tea, size-specialized malloc, etc.) that whether or not it reproduces on tip might be a hint. I think for size specialized mallocs, it might currently be disabled if you enable -msan, so I think probably worth trying on tip at least once without -msan and at least once with -msan.

• try with -gcflags=all=-d=checkptr=2 on tip.

  • The 2 makes it more precise than 1 (especially for stack variables), and it is useful to try this on tip with runtime: checkptr doesn't validate unsafe.Add function calls #74431 fixed on tip so that unsafe.Add is checked as well.

• try with GODEBUG=clobberfree=1.

  • This overwrites memory with 0xdeadbeef as it is freed. Some chance it might trigger some other test to fail, which might help localize where the core problem might be occurring, and also the GC specifically looks for that pattern in some cases, which might let the GC trigger a fatal throw earlier.

And then maybe a next tranche of:

• GODEBUG=gcshrinkstackoff=1

• GODEBUG=asyncpreemptoff=1

• try running runtime.GC() continuously in a background goroutine in the hopes of the GC flagging a problem closer to what might be a true root cause.

And here's maybe a third tranche, though might be getting progressively less useful:

• maybe try with GOEXPERIMENT=cgocheck2. I've never personally seen this catch anything, but could be worth a try if you have cgo in use.

• can you try with cgo disabled? (It sounds like this is for Pebble -- I don't know if Pebble supports that).

• enable the doubleCheckMalloc const. The malloc code is an important part of the integration with msan, so extra invariant checking in malloc could trigger something more specific. You could also grep the runtime source for other doubleCheck consts and enable them.

• gccheckmark=1

[RaduBerinde]

do you know if this reproduces on arm64, and/or with Go 1.24?

I hit this with go 1.23 and 1.24 and 1.25. I haven't tried to reproduce on arm64. But I can't reproduce even on amd64 on a vanilla ubuntu system, only from within this container: ghcr.io/christopherhx/runner-images:ubuntu24-runner-large-latest

Thank you so much for the tips; I will report back.

[RaduBerinde]

Some data with 1.25.3 before setting up the latest go tip:

• GODEBUG=gccheckmark=1: fails during the same runtime.GC() with runtime: checkmarks found unexpected unmarked object obj=0xc000430de0
• GODEBUG=cloberfree=1: fails during the same runtime.GC(). If I remove that call, it fails later at LOCKED POISON, where I print out the pointer and it is still correct. So the *queueNode does not get clobbered, which confirms my comment above (that I didn't see that object being freed). Edit: I checked this again, and if I print strings.Clone(file.Path) at the failure point, it still shows the correct string; I expected the string data itself to get clobbered..
• GODEBUG=gcshrinkstackoff=1: still crashes the same
• GODEBUG=asyncpreemptoff=1: still crashes
• disabling cgo: I can't do that because go: -msan requires cgo; enable cgo by setting CGO_ENABLED=1. I was never able to repro the problem without msan.
• -gcflags=all=-d=checkptr=2: it crashes the same in runtime.GC(). But if I remove that call to let the program run further, I get this:

releasep: m=0xc000099808 m->p=0xc000056c08 p->m=0xc000099808 p->status=3
fatal error: releasep: invalid p state

[RaduBerinde]

Not much difference with the tip (385dc33). Reproduces only with -msan; -gcflags=all=-d=checkptr=2 doesn't change anything (same crash in runtime.GC()). No crash with GODEBUG=clobberfree=1 and without -msan.

try running runtime.GC() continuously in a background goroutine in the hopes of the GC flagging a problem closer to what might be a true root cause.

I will try this next.

[RaduBerinde]

try running runtime.GC() continuously in a background goroutine in the hopes of the GC flagging a problem closer to what might be a true root cause.

I haven't yet seen a case where this crashes any earlier than before.

I noticed that if I add a runtime.GC() before file := dp.mu.queue.Front(), the problem no longer reproduces (even with the runtime.GC() call after the pop still there). Same thing if I instead add it before the PopFront().

[RaduBerinde]

It really feels like the compiler doesn't treat that pointer on the stack as a live pointer at that point in the code. How can I figure out if that's the case? I spent quite a bit of time with ChatGPT trying to figure out where in the gclocals data I would need to look for that "liveness map" but didn't quite get there.

[thepudds]

Hi @RaduBerinde, off the cuff, I'm not sure of best way to see that, but you could try go build -gcflags=-live=2 or similar.

And more generally, GOSSAFUNC=YourFunc go build is very powerful if you haven't tried it here. (See for example https://github.com/golang/go/blob/master/src/cmd/compile/internal/ssa/README.md).

You could also try to get some write barrier info via wb compiler debug flag or writebarrier SSA phase.

You can also poke around these flags (copy/pasting from a cmd/compile "Tips" section I added a couple of years ago here):

$ go tool compile -h              # compiler flags, e.g., go build -gcflags='-m=1 -l'
$ go tool compile -d help         # debug flags, e.g., go build -gcflags=-d=checkptr=2
$ go tool compile -d ssa/help     # ssa flags, e.g., go build -gcflags=-d=ssa/prove/debug=2

So that's more of a quick "some places to look" type of answer (rather than an actual answer 😅).

[RaduBerinde]

Thanks, this is useful, will try more falgs.

I did get this ssa.html the other day but it's pretty opaque to me.
ssa.html

[RaduBerinde]

The problem still reproes when using //go:noinline for the Front() and/or PopFront() calls. The liveness info around these calls looks right, file shows up as live after Front().

internal/deletepacer/delete_pacer.go:237:29: live at call to com/cockroachdb/pebble/internal/deletepacer.(*Queue[go.shape.struct { github.com/cockroachdb/pebble/internal/deletepacer.ObsoleteFile; JobID int }]).Front: .autotmp_267 .autotmp_268 .autotmp_269 .autotmp_330 .autotmp_334 dp m rateCalc timer
internal/deletepacer/delete_pacer.go:255:24: live at call to com/cockroachdb/pebble/internal/deletepacer.(*Queue[go.shape.struct { github.com/cockroachdb/pebble/internal/deletepacer.ObsoleteFile; JobID int }]).PopFront: .autotmp_267 .autotmp_268 .autotmp_269 .autotmp_330 dp file m rateCalc timer