Repository navigation
runtime: reportZombies prints "unmarked" for every slot and dumps nothing under Green Tea GC #80799
Description
Go version
go version go1.26.5 darwin/arm64
Also reproduced on go1.27rc2 darwin/arm64. The relevant code is unchanged at master.
Output of go env in your module/workspace:
GOARCH='arm64'
GOOS='darwin'
GOHOSTARCH='arm64'
GOHOSTOS='darwin'
GOVERSION='go1.26.5'
GOEXPERIMENT=''
CGO_ENABLED='1'
GOFLAGS=''
GODEBUG=''
GOTOOLCHAIN='auto'(GOEXPERIMENT is empty, i.e. defaults — which on 1.26 means greenteagc is on.)
What did you do?
Ran the runtime's own zombie-object reproducer:
cd $GOROOT/src/runtime/testdata/testprog
go build -o /tmp/testprog .
GODEBUG=invalidptr=0 /tmp/testprog GCZombie
What did you see happen?
runtime: marked free object in span 0x127480968, elemsize=192 freeindex=0 (bad use of unsafe.Pointer or having race conditions? try -d=checkptr or -race)
0x3a1d01360000 alloc unmarked
0x3a1d013600c0 free unmarked
0x3a1d01360180 alloc unmarked
0x3a1d01360240 free unmarked
0x3a1d01360300 alloc unmarked
0x3a1d013603c0 free unmarked
[... 42 slots in total: 21 "alloc unmarked", 21 "free unmarked", nothing else ...]
fatal error: found pointer to free object
Every slot is reported unmarked, no slot is tagged zombie, and the hexdumpWords call is therefore never reached — even though the sweeper threw precisely because it found a marked-but-free object in this span. The report contradicts the error it accompanies, and the object contents that make the failure diagnosable are not printed at all.
Same program, same toolchain, Green Tea disabled:
GOEXPERIMENT=nogreenteagc go build -o /tmp/testprog .
GODEBUG=invalidptr=0 /tmp/testprog GCZombie
runtime: marked free object in span 0x10263a5e0, elemsize=192 freeindex=0 (bad use of unsafe.Pointer or having race conditions? try -d=checkptr or -race)
0x23a2e5f90000 alloc marked
0x23a2e5f900c0 free marked zombie
7 6 5 4 3 2 1 0 f e d c b a 9 8 0123456789abcdef
000023a2e5f900c0: 00000000 00000000 00000000 00000000 ................
000023a2e5f900d0: 00000000 00000000 00000000 00000000 ................
[...]
21 slots tagged zombie, each with its hex dump.
What did you expect to see?
The second output.
Analysis
(*sweepLocked).sweep merges the inline mark bits into gcmarkBits, clearing them, and then runs the zombie check against gcmarkBits:
// Copy over and clear the inline mark bits if necessary.
if gcUsesSpanInlineMarkBits(s.elemsize) {
s.moveInlineMarks(s.gcmarkBits)
}
// Check for zombie objects.
if s.freeindex < s.nelems {
...
if (*s.gcmarkBits.bytep(obj / 8)&^*s.allocBits.bytep(obj / 8))>>(obj%8) != 0 {
s.reportZombies()
}moveInlineMarks ends with imb.init(s.spanclass, true), which resets the inline bits.
reportZombies then reads them back:
mbits := s.markBitsForBase()and under goexperiment.greenteagc that returns the inline bits whenever gcUsesSpanInlineMarkBits(s.elemsize):
func (s *mspan) markBitsForBase() markBits {
if gcUsesSpanInlineMarkBits(s.elemsize) {
return markBits{&s.inlineMarkBits().marks[0], uint8(1), 0}
}
return markBits{&s.gcmarkBits.x, uint8(1), 0}
}So mbits reads back all zero, zombie := mbits.isMarked() && !alloc is never true, and nothing is dumped.
Scope. gcUsesSpanInlineMarkBits(size) is heapBitsInSpan(size) && size >= 16, so this affects spans whose elemsize is in [16, 512] on 64-bit — size classes 2..26 of 67, which covers most Go allocations. Larger objects and the 8-byte class still report correctly; running the same reproducer with const size = 600 (elemsize 640) on stock go1.26.5 prints the zombie tags and hex dumps as expected.
Affected releases. Go 1.25 with GOEXPERIMENT=greenteagc, Go 1.26 and later where it is on by default, and master.
TestGcZombieReporting does not catch this because it only asserts that the output contains found pointer to free object, which is printed by the throw rather than by the loop.
Suggested fix
reportZombies should consult the same bits the zombie check that called it consulted:
- mbits := s.markBitsForBase()
+ // Not markBitsForBase: under Green Tea that returns the span's inline
+ // mark bits, which sweep has already drained into s.gcmarkBits and
+ // cleared by the time we get here.
+ mbits := markBits{&s.gcmarkBits.x, 1, 0}The only other markBitsForBase in mgcsweep.go (in the trace / clobberfree / race / msan / asan loop) runs before moveInlineMarks and is correct as it stands.
Activity
gabyhelp commentedon Aug 9, 2026
Related Issues
Related Code Changes
(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)
qiulaidongfeng commentedon Aug 9, 2026
Do you want to send a CL?
https://go.dev/doc/contribute
RaduBerinde commentedon Aug 9, 2026
I don't have very high confidence in the findings, I'd rather let someone who knows the code do it. BTW ideally we should backport to 1.26 too - I am trying to track a hard-to-reproduce failure and this logging would have helped.
gopherbot commentedon Aug 10, 2026
Change https://go.dev/cl/812800 mentions this issue:
runtime: report zombies using swept mark bitsignoramous commentedon Aug 27, 2026
Unsure if related, but we're seeing gc/memory sweep related crashes reported on Android (cgo via gomobile) on go1.26.7:
traceback
(all threads).
Let a run choose its GC experiment, and prove it got one
Repair the report instead of removing the collector
denisvmedia commentedon Aug 30, 2026
Confirming this with a real failure and a before/after comparison. Note, I am not claiming a collector correctness bug here. This comment is only about the broken zombie report.
Details
I have been investigating a recurring
on windows/amd64.
With the normal Go runtime, every collected report looked like this:
There were 496 object rows: 335
alloc unmarked, 161free unmarked, zero marked objects, zero zombies, and no hexdumps.This is especially confusing because the sweeper has already detected a marked free object, but
reportZombiesthen reports every object as unmarked.I applied the suggested change to
reportZombiesonly:Nothing else changed: same collector, same commit, same test matrix.
The next reproduction reported:
Again there were 496 rows, but now the report found 95 zombies and printed their contents: 81 string headers, 8 funcvals, and 6 zeroed objects. Seven funcvals were symbolized as
database/sqlmethod values.That changed the failure from an almost empty runtime abort into something I could actually investigate.
Standalone reproducer
I also made a small standalone reproducer. It does not need a GOROOT checkout:
I tested it with go1.27.0 on linux/amd64, windows/amd64, and darwin/arm64:
GOEXPERIMENT=nogreenteagcDisabling Green Tea also makes the report work, but it changes the collector itself, as shown by the different object count. The patch fixes the diagnostic without changing the collector being investigated.
Reproducer and tests for Linux, macOS, and Windows:
https://github.com/denisvmedia/ptah-2365-repro/tree/d590cc753764517b4815429f91b5542b2a932889/zombiereport
One related diagnostic gap
There may also be a separate issue with
GODEBUG=gccheckmark=1.For spans using inline mark bits, normal marking can go through
tryDeferToSpanScan, sogreyobjectdoes not perform thespan.isFreecheck. During the checkmark pass,greyobjectis reached, but theuseCheckmarkbranch callssetCheckmarkinstead of checkingspan.isFree.So for these size classes,
gccheckmark=1also does not catch this bad pointer earlier.RaduBerinde commentedon Aug 30, 2026
Nice find - I tried a lot to reproduce my original problem with
gccheckmark=1with no luck. We should definitely fix this too.RaduBerinde commentedon Sep 6, 2026
Not sure if related, but I hit this panic:
RaduBerinde commentedon Sep 17, 2026
Would be great to get a fix here. Some tests keep hitting a failure like this once in a blue moon and the missing debug data would help (no luck reproducing it or I would try with a modified runtime).
ci(agent): Windows agent tests on the classic GC to A/B the Green Tea…