Troubleshooting

Troubleshooting

Prometheus rejects the whole scrape

Almost always a duplicate metric name producing two HELP/TYPE line pairs. Registration now refuses duplicates with metric.ErrDuplicateMetric — so:

if err := reg.Register(name, m); err != nil {
	return err // ← check it
}

If you discard that error, you get the old behaviour by another route: your metric is not registered and you will not know why.

A metric is missing from /metrics

• Registration failed and the error was discarded. See above.
• The series does not exist yet. With creates a series on first use; a counter that has never been incremented is absent. Pre-create with InitWith or a bare With call.
• Registered too early. The registry is placed in the container during the metrics extension's OnInitialize. Resolve it from your own extension's OnInitialize or later.

rex_metric_series_dropped_total is non-zero

A vector hit its cardinality cap. The vector label names it.

Find the unbounded label. It is almost always something client-derived — a raw path, a user id, a header value. Bound it or remove it; raising the cap is rarely the fix.

Everything past the cap is folded into __overflow__, so the totals stay honest, but the breakdown is gone for those series.

rex_events_dropped_total is non-zero

The event bus queue filled and events were discarded. A subscriber is too slow, or the traffic outgrew the queue.

Consequence: anything derived from events is now incomplete. Note that the HTTP metrics in this extension take Status, Duration and RoutePattern from the handled event, so a sustained non-zero drop count means those counters undercount. The in-flight gauge is unaffected — it is middleware.

The in-flight gauge is wrong

If it drifts upward and never recovers, something is maintaining it from events rather than from middleware. That was the old behaviour and it could not be correct: the increment and the decrement are separate events, dropped independently, and increments happen under precisely the load that fills the queue.

The current implementation is InFlightMiddleware, where the increment and the decrement are one call and its defer — nothing between them to drop, and the defer holds even if the handler panics.

Every URL is its own series

You are labelling by Request.URL.Path. Use the handled event's RoutePattern, which is the registered pattern with the BaseURL already stripped. See Cardinality.

A route shows no duration series until it gets traffic

It should be pre-created at registration. If it is not, the route was registered after the tables were built — which the framework now refuses — or the metrics extension is not subscribed. Check the startup log.

/metrics is slow or times out

• Too many series. Check rex_metric_series_dropped_total and the raw series count. A scrape that times out is usually cardinality, not I/O.
• ToOpenMetrics() in your own code. It builds the whole payload in memory. Use WriteOpenMetrics(w).

Histogram quantiles look wrong

Check the buckets. histogram_quantile interpolates within a bucket, so if 99% of observations land in one bucket, the quantile is an interpolation across that bucket's whole width and means very little.

Pick bounds around the latencies you care about:

metric.NewHistogramVecWithBuckets(labels,
	[]float64{.005, .01, .025, .05, .1, .25, .5, 1, 2.5, 5, 10})

Process metrics are stale

CollectionInterval is probably 0 — which disables the collector — most likely from a partial struct literal. Use metric.NewConfig(...).

Metrics are exposed publicly

/metrics on the default router exposes your route table, error rates and latency profile. Move it to a dedicated internal listener:

metric.WithMetricsRouter(rx.RouterConfig{Addr: "127.0.0.1:9090"})