You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently, when crashing due to receiving a fatal signal (e.g.: SIGABRT), Go outputs backtraces for all Gs, and then the registers of the thread on which the signal hit:
// tracebackothers on original m skipped this one; trace it now.
goroutineheader(mp.curg)
traceback(^uintptr(0), ^uintptr(0), 0, mp.curg)
} elseifcrashing.Load() ==0 {
tracebackothers(gp)
print("\n")
}
dumpregs(c)
}
.
We're observing an issue where a Go program appears to stall (making no significant progress, if any) that we have only seen in production. We have not been able to reproduce at will. The G backtraces indicate that there are many (1000s) runnable goroutines when this happens, yet only 3 out of ~10 GOMAXPROCS Ms appear assigned to a goroutine. It is unclear what the other Ms are doing. Seeing where they are may provide a hint as to what's going wrong.
I believe this to be useful in general, although in an otherwise healthy process I'd expect at least the GOMAXPROCS Ms to be mentioned in goroutine backtraces like this:
But that would still leave the Ms that are in cgo-calls or syscalls.
Regrettably, we have not been able to extract a coredump from such a crash from our environment, which would be another way of getting at this useful information.
It's fine if this functionality is hidden behind a GODEBUG flag, but I'd say there's a case for enabling it by default. The format (header) would have to be somewhat different from regular goroutine stacks, as otherwise tools parsing this may get confused.
I agree that #13161 seems quite similar. It sounds like what I want is subsumed by @aclements proposed GOTRACEBACK=all, given:
Currently, GOTRACEBACK=all is a misnomer. It prints stacks for all goroutines that happen to be non-running or running on the current OS thread, but it does not print stacks for goroutines that are running on other OS threads.
Proposal Details
Currently, when crashing due to receiving a fatal signal (e.g.:
SIGABRT
), Go outputs backtraces for all Gs, and then the registers of the thread on which the signal hit:go/src/runtime/signal_unix.go
Lines 739 to 752 in beaf7f3
We're observing an issue where a Go program appears to stall (making no significant progress, if any) that we have only seen in production. We have not been able to reproduce at will. The G backtraces indicate that there are many (1000s) runnable goroutines when this happens, yet only 3 out of ~10 GOMAXPROCS Ms appear assigned to a goroutine. It is unclear what the other Ms are doing. Seeing where they are may provide a hint as to what's going wrong.
I believe this to be useful in general, although in an otherwise healthy process I'd expect at least the GOMAXPROCS Ms to be mentioned in goroutine backtraces like this:
But that would still leave the Ms that are in cgo-calls or syscalls.
Regrettably, we have not been able to extract a coredump from such a crash from our environment, which would be another way of getting at this useful information.
It's fine if this functionality is hidden behind a
GODEBUG
flag, but I'd say there's a case for enabling it by default. The format (header) would have to be somewhat different from regular goroutine stacks, as otherwise tools parsing this may get confused.cc @mknyszek @prattmic
The text was updated successfully, but these errors were encountered: