runtime: garbage collector panic

[pgimalac]

Go version

1.24.9 to 1.25.5

Output of go env in your module/workspace:

- OS: Linux
- Architecture: amd64 and arm64
- Environment: containerized (k8s)
- All kinds of large instance types on various cloud providers: c2d-standard-56, n2d-standard-32, standard_d32as_v5, c7g.12xlarge, m7g.16xlarge

What did you do?

A Go service is hitting a garbage collector panic in internal deployments.

runtime: full=0x600467c8000067 next=1153 jobs=1144 nDataRoots=2 nBSSRoots=22 nSpanRoots=64 nStackRoots=1053 spanq.n=0
panic: non-empty mark queue after concurrent mark
fatal error: panic on system stack

I know that it was already happening when we used Go 1.24.9, and it is still happening with 1.25.5. I don't have older logs available so I don't know when it actually started.

I don't have a reproducible example but it's happening 10~20 times a day across our prod environment, and I can deploy a custom version / build with a custom Go if needed to collect information.

I checked all Go memstats metrics and some host metrics of an occurence and didn't see anything remarkable, it was not that close to the memory limits, not allocating that much, not using a lot of CPU, etc.

I'm pretty sure we don't set GOMEMLIMIT, we set GOGC to 50, and one occurence had GOMAXPROCS set to 21 (but that can vary a lot depending on the host).

If it changes anything the service is privileged and uses eBPF for system monitoring. The entrypoint is defined here, but it's not a trivial binary.

What did you see happen?

Here is the start of a stack trace of an occurrence, I can share the full stack trace if it's useful.

Stacktrace

runtime: full=0x600467c8000067 next=1153 jobs=1144 nDataRoots=2 nBSSRoots=22 nSpanRoots=64 nStackRoots=1053 spanq.n=0
panic: non-empty mark queue after concurrent mark
fatal error: panic on system stack
runtime stack:
runtime.throw({0x2890e61?, 0x2c8be90?})
	runtime/panic.go:1094 +0x48 fp=0x74aa367fbba0 sp=0x74aa367fbb70 pc=0x48b008
panic({0x22eac60?, 0x2c8be90?})
	runtime/panic.go:747 +0x33b fp=0x74aa367fbc50 sp=0x74aa367fbba0 pc=0x48addb
runtime.gcMark(0x4956cd?)
	runtime/mgc.go:1633 +0x409 fp=0x74aa367fbcc8 sp=0x74aa367fbc50 pc=0x4327a9
runtime.gcMarkTermination.func1()
	runtime/mgc.go:1059 +0x17 fp=0x74aa367fbce0 sp=0x74aa367fbcc8 pc=0x431b57
runtime.systemstack(0x800000)
	runtime/asm_amd64.s:513 +0x4a fp=0x74aa367fbcf0 sp=0x74aa367fbce0 pc=0x491d0a
goroutine 29 gp=0xc0003936c0 m=71 mp=0xc004181808 [garbage collection]:
runtime.systemstack_switch()
	runtime/asm_amd64.s:478 +0x8 fp=0xc003f1fa28 sp=0xc003f1fa18 pc=0x491ca8
runtime.gcMarkTermination({0x1?, 0x12401ba40c627?, 0x12401ba411b63?, 0x2f6b3?})
	runtime/mgc.go:1058 +0x135 fp=0xc003f1fe98 sp=0xc003f1fa28 pc=0x430a35
runtime.gcMarkDone()
	runtime/mgc.go:1032 +0x369 fp=0xc003f1ff38 sp=0xc003f1fe98 pc=0x4306a9
runtime.gcBgMarkWorker(0xc000066930)
	runtime/mgc.go:1600 +0x345 fp=0xc003f1ffc8 sp=0xc003f1ff38 pc=0x432085
runtime.gcBgMarkStartWorkers.gowrap1()
	runtime/mgc.go:1373 +0x25 fp=0xc003f1ffe0 sp=0xc003f1ffc8 pc=0x431d05
runtime.goexit({})
	runtime/asm_amd64.s:1693 +0x1 fp=0xc003f1ffe8 sp=0xc003f1ffe0 pc=0x493cc1
created by runtime.gcBgMarkStartWorkers in goroutine 1
	runtime/mgc.go:1373 +0x105

What did you expect to see?

The garbage collector should complete without panicking.

[pgimalac]

Let me know if there is any configuration I can try or any knob to tweak to provide more information.

I can try deploying a build using go1.26rc2 and see if it still reproduces.

[gabyhelp]

Related Issues

• runtime: panic: non-empty mark queue after concurrent mark (Go1.14, Go1.15) #41303 (closed)
• runtime: non-empty mark queue after concurrent mark #69803 (closed)
• runtime: go runtime GC panic #45401 (closed)
• runtime: panics during gc #11640 (closed)
• runtime: GC crash "checkmark found unmarked object" #17694 (closed)
• runtime: go microservice is crashing in runtime during garbage collection running #64682 (closed)
• Runtime error is seen with GC in go 1.18.1 #61332 (closed)
• runtime: post-mortem analysis of Go 1.13 runtime/mgcsweep.go fatal error ("error: non in-use span in unswept list") #58681 (closed)
• runtime: panic marked free object in span[1.17] #54247 (closed)
• runtime.gc bug #23715 (closed)

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[DanielMorsing]

If you could verify that this happens on go1.26rc2, that'd both help identify whether we incidentally fixed it and see if it happens on the new greentea garbage collector.

[dmitshur]

CC @golang/runtime.

[mknyszek]

Green Tea shouldn't have anything to do with this. Trying with Go 1.26 RC2 is not a bad idea, but I suspect nothing will be different. (Well... the mark termination conditions changed slightly, but everything still follows the same fallback paths. To not take those fallback paths and end up here anyway requires a very specific set of circumstances...)

This is about mark termination (which is hard, subtle, and we know has problems, see #27993, which has had a workaround for years and no known crashes since), and the fundamentals haven't been touched in a long time, except for a relatively recent tweak (I think around Go 1.24) to support the weak package.

Do you know if weak.Pointer is used frequently in your codebase? That's my only guess.

[mknyszek]

Also, your original text omitted the line printed before the panic. That would help in narrowing down the issue. Could you please provide that too?

(There is no identifiable information in this, and is safe to share. It's all just runtime-internal metadata.)

[mknyszek]

Green Tea shouldn't have anything to do with this.

Ugh, actually, if the check that's failing is not a root job (unlikely) but the work.full != 0 check, then yeah, Green Tea might appear to fix the issue since global work queue for small objects with Green Tea anymore. However, Green Tea likely would not have truly fixed any such issues, only hidden them, because work.full still exists and is used, but is exclusively for larger objects. So even if it works with Go 1.26 RC2, I don't think we should consider it fixed without understanding the issue for that reason but there's an even worse potential issue...

Well... the mark termination conditions changed slightly, but everything still follows the same fallback paths. To not take those fallback paths and end up here anyway requires a very specific set of circumstances...

For some additional context, "specific set of circumstances" means "no work in local queues, yet still somehow a piece of global work was missed." Green Tea's small object span queues are only local, which means if the problem still exists with Green Tea but appears to go away, it might just be translating the crash into an inefficiency (a mark restart).

This would be pretty bad because we'd have very little insight into the issue, and now we'll be carrying around a performance bug. (We should really fix #27993, but it'll probably take a good bit of effort to prove (in the academic sense) that the fix really works. 😞 We accept the workaround because we have many reasons to believe it's extremely rare for the workaround to fire, but there may be a new bug here.)

[prattmic]

If it changes anything the service is privileged and uses eBPF for system monitoring.

I don't think it could cause failures like this, but since you mention eBPF, note that uretprobes (#64030 (comment)) can at least cause crashes in traceback, so it's worth a check that you aren't using them.

[pgimalac]

Also, your original text omitted the line printed before the panic. That would help in narrowing down the issue. Could you please provide that too?

I put this line in the (collapsed) stack trace section, I didn't want to clutter the description.
I updated the description to add it along with the error message.

[mknyszek]

Also, your original text omitted the line printed before the panic. That would help in narrowing down the issue. Could you please provide that too?

I put this line in the (collapsed) stack trace section, I didn't want to clutter the description. I updated the description to add it along with the error message.

Ahhh sorry! I missed that. Thanks!

OK, as suspected, work.full != 0, the root jobs are fine.

[pgimalac]

Do you know if weak.Pointer is used frequently in your codebase?

Not used explicitly, but we do have a few uses of unique, and they're only in this binary (we build other binaries from the same codebase but they don't use that code), so that's suspicious.

However the oldest occurrences of the panic I can find predate this code being added, and before that the only uses were from crypto and net/netip (though unique).

[pgimalac]

I don't think it could cause failures like this, but since you mention eBPF, note that uretprobes (#64030 (comment)) can at least cause crashes in traceback, so it's worth a check that you aren't using them.

The binary uses uretprobes to probe some other processes, but not on itself (and nothing else should be probing it either)