clustering: deal with gaps during cluster changes

[tpaschalis]

During cluster event changes, and when targets are being shuffled around, a target disappearing will make the underlying Prometheus code insert a staleness marker on the next scrape.

If no new scrapes (by the new target owner) has been made in the meantime, the staleness marker will appear as a gap in the graph.

Given how clustering is aiming to be a solution for dynamically scaling Grafana Agent, we should find a way to enhance staleness marker tracking to avoid gaps in graphs for series that continue their presence.

[github-actions]

This issue has not had any activity in the past 30 days, so the needs-attention label has been added to it.
If the opened issue is a bug, check to see if a newer release fixed your issue. If it is no longer relevant, please feel free to close this issue.
The needs-attention label signals to maintainers that something has fallen through the cracks. No action is needed by you; your issue will be kept open and you do not have to respond to this comment. The label will be removed the next time this job runs if there is new activity.
Thank you for your contributions!

[rfratto]

Hi there 👋

On April 9, 2024, Grafana Labs announced Grafana Alloy, the spirital successor to Grafana Agent and the final form of Grafana Agent flow mode. As a result, Grafana Agent has been deprecated and will only be receiving bug and security fixes until its end-of-life around November 1, 2025.

To make things easier for maintainers, we're in the process of migrating all issues tagged variant/flow to the Grafana Alloy repository to have a single home for tracking issues. This issue is likely something we'll want to address in both Grafana Alloy and Grafana Agent, so just because it's being moved doesn't mean we won't address the issue in Grafana Agent :)

[andres-garcia-webbeds]

Also happening to us.
Metrics are inconsistent