Scrape duration using prometheus is increasing continuously until restart

[Sparc0]

problem

We are using Prometheus to scrape metrics from the management server.
We see a issue that the scrape_duration_seconds metric for the cloudstack job is increasing in slow pace and after ~7-10 days the scrape duration is longer then the configured interval and we get no more metrics.
First time we saw this just doing a time curl towards the mangement server it took 40-50s for metrics to come back.

versions

CloudStack 4.22.0.0

The steps to reproduce the bug

1. Start scraping
2. See the scrape_duration_seconds{job="cloudstack"} increasing daily.

What to do about it?

1. Restart the management server solves the issue, temporarily