Fix memory Leak: use route template for request-metric path label - #115
Merged
Conversation
The request middleware used the raw r.URL.Path as the "path" label on the
axon_http_requests counter and axon_http_request_latency_seconds histogram.
Because prometheus/client_golang retains every series in memory permanently
(independent of scraping), routes that carry unique path segments — most
notably the webhook PathPrefix route /webhook/<id>/... — created a new,
never-freed counter + histogram per distinct URL. A heap profile attributed
~820MB / ~977MB to these two lines (newHistogram + MakeLabelPairs).
Use the matched mux route template (e.g. "/webhook/" or "/webhook/{id}")
instead of the raw path so all concrete URLs collapse to a single bounded
series. Falls back to a fixed "<unmatched>" sentinel when no route template
is resolvable, so the label can never become unbounded.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
shawnburke
enabled auto-merge (squash)
July 18, 2026 13:47
ashiramin
approved these changes
Jul 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A heap profile (
heap1.prof) showed ~820 MB of ~977 MB retained by Prometheus, attributed to two lines in the request middleware:The root cause is label-cardinality explosion, not a lack of scraping.
prometheus/client_golangkeeps every metric series in memory in the registry permanently, regardless of whether anything scrapes/metrics— scraping only reads values, it never frees series.The
pathlabel was set to the rawr.URL.Path. The webhook server registers withmux.PathPrefix("/webhook/"), so every/webhook/<id>/...URL matched and created its own permanent counter and histogram (each histogram is ~11 buckets + sum + count + label pairs, hence the largenewHistogram/MakeLabelPairsallocations). Unique webhook ids / resource ids → unbounded series growth → the observed ~1 GB.Fix
Use the matched mux route template for the
pathlabel instead of the raw path:/webhook/<any-id>→/webhook/(PathPrefix route)/webhook/{id}variable routes →/webhook/{id}All concrete URLs collapse to a single bounded series. Falls back to a fixed
"<unmatched>"sentinel if no route template is resolvable, so the label can never become unbounded. (Note: mux only runs middleware for matched routes, so 404 scans already never created series.)This bounds memory for every deployment, not just those that disable metrics.
Tests
Added
http_server_metrics_test.go:TestRequestMetricsCollapsePrefixRouteToTemplate— 25 distinct/webhook/<id>requests through aPathPrefixroute produce exactly one series labeled/webhook/on both the counter and the histogram. (Fails before the fix with 25 distinct series.)TestRequestMetricsCollapseVariableRouteToTemplate— same for a/webhook/{id}variable route, collapsing to/webhook/{id}.Full
./server/http/package passes;go build ./...andgo vetclean.🤖 Generated with Claude Code