Cardinality

Cardinality

A labelled metric creates one time series per distinct combination of label values, and each series is retained for the process's lifetime. Nothing evicts them.

So the number of series is bounded only by the number of distinct label values — and if a label value comes from the request, that bound is set by the client.

The failure

The request-duration histogram was once labelled with Request.URL.Path:

GET /users/1  →  http_request_duration_seconds{path="/users/1"}
GET /users/2  →  http_request_duration_seconds{path="/users/2"}
…

An unauthenticated client walking /users/1 through /users/10000000 creates ten million histograms, each with its own bucket array. Nothing evicts them. The /metrics response grows with them, so the scrape eventually times out too — losing the monitoring that would have shown the problem.

Two fixes, both needed

1. Label by route pattern

"/users/{id}" is one series regardless of how many identifiers exist, and the pattern comes from the route table rather than from the request. That removes the client's control entirely.

The framework's handled event carries RoutePattern for exactly this — already stripped of the router's BaseURL. Never label with Request.URL.Path.

A related bug this fixed: series were pre-created keyed by the route's pattern while requests created further series keyed by the URL path, so every parameterized route was double-counted — one empty series under /users/{id} plus one per distinct URL.

2. Cap the series count anyway

A route pattern is bounded, but a label drawn from a header, a query parameter or a tenant identifier is not — and this package cannot know which labels a caller will add.

const DefaultMaxSeries = 1000

1000 is generous for route patterns — an application with a thousand distinct endpoints is unusual — and small enough that hitting it means something is wrong rather than merely large.

What happens at the cap

Everything past it is recorded into one reserved series:

http_requests_total{path="__overflow__"} 48213

Not dropped. Dropping would make the totals silently wrong; folding them together keeps the sum honest and makes the overflow visible in the exposition itself.

The cap is what makes the failure mode "some series are missing" rather than "the process ran out of memory".

Detecting it

rex_metric_series_dropped_total{vector="http_requests_total"} 3

Alert on it:

rex_metric_series_dropped_total > 0

A non-zero value means a label is unbounded. Find which — the vector label names the metric — and either bound the label or stop using it.

Tuning

if b, ok := myVec.(metric.BoundedVec); ok {
	b.SetMaxSeries(5000) // 0 → the 1000 default; negative → no cap
	_ = b.DroppedSeries()
}

Every labelled vector implements BoundedVec.

Raising the cap is rarely the fix. If a vector is overflowing, the usual cause is a label that should not be a label. Ask what query the label enables: a label you never group or filter by is costing memory for nothing, and one whose values are client-supplied is a liability.

Disabling the cap (SetMaxSeries(-1)) is for a vector you can prove is bounded — an enum, a fixed list of backends — and even then the proof tends to expire.

A checklist for a new label

• Is the set of values enumerable in advance? If not, do not use it.
• Can a client influence it? If so, do not use it.
• Will I actually group by or filter on it?
• What is the product of this label's cardinality with the existing ones?

User ids, request ids, raw paths, full URLs, error messages, timestamps and email addresses are never labels. Put them in logs, joined to the metric by time and by the labels you do keep.