Skip to content

runtime/metrics: add metric to identify when finalizers are running slowly, holding up the queue #72948

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
mknyszek opened this issue Mar 19, 2025 · 1 comment
Open

runtime/metrics: add metric to identify when finalizers are running slowly, holding up the queue #72948

mknyszek opened this issue Mar 19, 2025 · 1 comment
Assignees
@mknyszek
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone
Backlog

Comments

@mknyszek
Copy link
Contributor

As per https://groups.google.com/g/golang-nuts/c/uL68-fxg2K4, this is a subtlety with finalizers (and cleanups) that doesn't really have any diagnostics. This issue proposes a new runtime/metrics metric to monitor when a finalizer or cleanup is likely to block the queue.

Taking a time measurement on each execution seems unfortunately expensive, but provided we're only trying to detect slow finalizers, it may be sufficient to take a sampling-based approach and expose a time histogram of finalizer (and cleanup) execution times.
@gopherbot gopherbot added the compiler/runtime Issues related to the Go compiler and/or runtime. label Mar 19, 2025
@gabyhelp gabyhelp added the Telemetry-Proposal Issues proposing new telemetry counters. label Mar 19, 2025
@mknyszek mknyszek self-assigned this Mar 19, 2025
@mknyszek mknyszek added NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. and removed Telemetry-Proposal Issues proposing new telemetry counters. labels Mar 19, 2025
@mknyszek mknyszek modified the milestones: Go1.25, Backlog Mar 19, 2025
@gopherbot
Copy link
Contributor

Change https://go.dev/cl/671437 mentions this issue: runtime: report finalizer and cleanup queue length with checkfinalizer>0

gopherbot pushed a commit that referenced this issue May 20, 2025
@mknyszek @gopherbot
runtime: report finalizer and cleanup queue length with checkfinalizer>0
0d42ceb 
This change adds tracking for approximate finalizer and cleanup queue
lengths. These lengths are reported once every GC cycle as a single line
printed to stderr when GODEBUG=checkfinalizer>0.

This change lays the groundwork for runtime/metrics metrics to produce
the same values.

For #72948.
For #72950.

Change-Id: I081721238a0fc4c7e5bee2dbaba6cfb4120d1a33
Reviewed-on: https://go-review.googlesource.com/c/go/+/671437
Reviewed-by: Michael Pratt <mpratt@google.com>
Auto-Submit: Michael Knyszek <mknyszek@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mknyszek mknyszek

Labels
compiler/runtime Issues related to the Go compiler and/or runtime. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
Go Compiler / Runtime
Status: Todo
Milestone
Backlog
Development

No branches or pull requests
3 participants
@mknyszek @gopherbot @gabyhelp