Metrics are leaking in prometheus

[evkuzin]

I don't know why it was decided to implement metrics in Prometheus via go-kit but this now makes it pretty complicated to delete stale metrics.
We have waves of services registered and unregistered and every time fabio will have them stuck in metrics for ages. If we are not careful enough we can see prometheus suffering from Gigabytes of metrics per scrape :(

[tristanmorgan]

Can I ask for some more details on how to replicate this issue?
Would a simple web server that gets relaunched on a different port frequently show this? or would there have to be traffic to drive up the metrics to cause this issue?

[evkuzin]

@tristanmorgan Hi! Thank you for taking a look!
Sure.
You start local consul and add a service

cat test.json
{
    "service": {
        "name": "evgeny-test",
        "port": 9100,
        "tags": [
            "urlprefix-/metrics"
        ],
        "checks": [
          {
            "id": "trur",
            "name": "bool",
            "args": ["test", "-f", "./on"],
            "interval": "3s"
          }
        ]
    }
}

touch on file

touch ./on

and start fabio
I have this config:

grep -vE "^$|^#" /opt/fabio/fabio.properties
proxy.addr = :8081;proto=http,:9999;proto=prometheus
log.level = INFO
registry.consul.addr = 127.0.0.1:8500
registry.consul.kvpath = /evgeny/test
registry.consul.tagprefix = urlprefix-
registry.consul.register.addr = SOME_IP:9998
registry.consul.register.name = fabio_test
registry.consul.register.checkTLSSkipVerify = true
registry.consul.checksRequired = all
metrics.target = prometheus
metrics.prefix = fabio
ui.access = ro
ui.addr = :9998;proto=http
ui.color = light-green
ui.title = fabio

This will have you one service announced via fabio
You dont even need the real service on port 9100.
Now just do to make meteics

curl localhost:8081/metrics -s

and you will see them

curl localhost:9999/metrics -s | grep fabio_route_bucket
fabio_route_bucket{host="",path="/metrics",service="evgeny-test",target="http://127.0.0.1:9100/",le="0.005"} 5
fabio_route_bucket{host="",path="/metrics",service="evgeny-test",target="http://127.0.0.1:9100/",le="0.01"} 5
fabio_route_bucket{host="",path="/metrics",service="evgeny-test",target="http://127.0.0.1:9100/",le="0.025"} 5
fabio_route_bucket{host="",path="/metrics",service="evgeny-test",target="http://127.0.0.1:9100/",le="0.05"} 5
fabio_route_bucket{host="",path="/metrics",service="evgeny-test",target="http://127.0.0.1:9100/",le="0.1"} 5
fabio_route_bucket{host="",path="/metrics",service="evgeny-test",target="http://127.0.0.1:9100/",le="0.25"} 5
fabio_route_bucket{host="",path="/metrics",service="evgeny-test",target="http://127.0.0.1:9100/",le="0.5"} 5
fabio_route_bucket{host="",path="/metrics",service="evgeny-test",target="http://127.0.0.1:9100/",le="1"} 5
fabio_route_bucket{host="",path="/metrics",service="evgeny-test",target="http://127.0.0.1:9100/",le="2.5"} 5
fabio_route_bucket{host="",path="/metrics",service="evgeny-test",target="http://127.0.0.1:9100/",le="5"} 5
fabio_route_bucket{host="",path="/metrics",service="evgeny-test",target="http://127.0.0.1:9100/",le="10"} 5
fabio_route_bucket{host="",path="/metrics",service="evgeny-test",target="http://127.0.0.1:9100/",le="+Inf"} 5

and this is where the fun starts
Lets stop service via failed check

rm ./on

now I see route removed

2025/03/18 13:43:32 [INFO] Config updates
+ route add evgeny-test /metrics http://127.0.0.1:9100/
2025/03/18 13:43:49 [INFO] Config updates
- route add evgeny-test /metrics http://127.0.0.1:9100/
2025/03/18 13:47:11 [WARN] No route for localhost:8081/metrics

and I expect metrics to be removed too, but they are not

curl localhost:9999/metrics -s | grep fabio_route_bucket
fabio_route_bucket{host="",path="/metrics",service="evgeny-test",target="http://127.0.0.1:9100/",le="0.005"} 5
fabio_route_bucket{host="",path="/metrics",service="evgeny-test",target="http://127.0.0.1:9100/",le="0.01"} 5
fabio_route_bucket{host="",path="/metrics",service="evgeny-test",target="http://127.0.0.1:9100/",le="0.025"} 5
fabio_route_bucket{host="",path="/metrics",service="evgeny-test",target="http://127.0.0.1:9100/",le="0.05"} 5
fabio_route_bucket{host="",path="/metrics",service="evgeny-test",target="http://127.0.0.1:9100/",le="0.1"} 5
fabio_route_bucket{host="",path="/metrics",service="evgeny-test",target="http://127.0.0.1:9100/",le="0.25"} 5
fabio_route_bucket{host="",path="/metrics",service="evgeny-test",target="http://127.0.0.1:9100/",le="0.5"} 5
fabio_route_bucket{host="",path="/metrics",service="evgeny-test",target="http://127.0.0.1:9100/",le="1"} 5
fabio_route_bucket{host="",path="/metrics",service="evgeny-test",target="http://127.0.0.1:9100/",le="2.5"} 5
fabio_route_bucket{host="",path="/metrics",service="evgeny-test",target="http://127.0.0.1:9100/",le="5"} 5
fabio_route_bucket{host="",path="/metrics",service="evgeny-test",target="http://127.0.0.1:9100/",le="10"} 5
fabio_route_bucket{host="",path="/metrics",service="evgeny-test",target="http://127.0.0.1:9100/",le="+Inf"} 5

The only way to remove them will be to loop over each target and prometheus.Unregister(metric)
But we dont do that, we just replace one table with another.

And this makes the whole code leaky, because since metrics are the part of the Target structure, old tables wont be removed by the GC, because metric is in use.

[evkuzin]

actually you cant even unregister it because theer is no way you can do that with this package :( https://pkg.go.dev/github.com/go-kit/kit/metrics/prometheus@v0.9.0#NewHistogram

[evkuzin]

@tristanmorgan hi!
did you have a luck to replicate the issue by any chance?

[tristanmorgan]

Hi @evkuzin,
I have been able see the fabio_route_... metrics build up over time. I can also see that in a moderate/high churn scenario this will leak and I don't have a solution for it currently.

[evkuzin]

At least I get the code right

[bustersg]

@evkuzin Hi, I saw on another issue that you gotten 1.46mil ingestion with spike on a later version of victoriametrics.
Can you share a little how do you archive 1.46mil, kind of deployment, resources, config parameters, vmagent or promtheus remoteWrite etc. Thanks in advance.

[evkuzin]

@bustersg sorry, but I dont think its related to the issue.

[bustersg]

@evkuzin VictoriaMetrics/VictoriaMetrics#6211
saw your grafana graph Y axis in millions ingestion
i can only do 200k.
sorry to hijack the this thread but not sure how to message you directly.
do you have other means of communication? like email or slack. thanks