Graphs will have no data when zoomed in due to misalignment when $__rate_interval is used

[nobuto-m]

Bug Description

10 graphs in the node exporter dashboard are missing due to "no data" #23

However, 43 additional graphs will be missed when it's zoomed into say "Last 15 minutes". I think this is due to the misalignment in $__rate_interval. scrape interval in the data source config is not set and it has 15s as the default value. However, Prometheus in COS uses 1m as the global scrape_interval so $__rate_interval won't be properly calculated if I'm not mistaken.

ref: https://grafana.com/blog/2020/09/28/new-in-grafana-7.2-__rate_interval-for-prometheus-rate-queries-that-just-work/

[zoomed in to 15 min]

[after manually putting >1m (the scrape_interval) to the Min step]

[scrape interval is not set in the data source config]

To Reproduce

With MAAS provider, deploy the grafana-agent charm onto another charm, and relate it to COS. Then zoom into 15 min range in the Grafana node-exporter dashboard.

Environment

grafana-agent latest/edge 20

Relevant log output

N/A

Additional context

No response

[nobuto-m]

Moved to: canonical/prometheus-k8s-operator#543

[simskij]

@nobuto-m reopening this as it feels like a more appropriate place to have the conversation, as the agent subordinate charm won't have the option of using the scrape config charm inbetween.

[nobuto-m]

Hmm, what kind of actions are expected in the grafana-agent charm for this issue?

[marcusboden]

Hi, we're seeing the issue with one of our customers.
I've manually set the scrape_interval to 15s to test, which does the job but is not very permanent. It would be good to create a fix, similar to the k8s charm.