Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[bug] default resolution is too low #50

Closed
uhthomas opened this issue May 26, 2023 · 7 comments
Closed

[bug] default resolution is too low #50

uhthomas opened this issue May 26, 2023 · 7 comments
Assignees
Labels
bug Something isn't working

Comments

@uhthomas
Copy link
Contributor

Describe the bug

The default resolution of 30s is too low and renders some dashboards with "No Data". This is likely because I'm using Grafana Mimir, as opposed to a standard Prometheus install.

image

How to reproduce?

  1. Collect metrics with Grafana Mimir.
  2. Load the dashboard.

Expected behavior

Changing the resolution from 30s to 1m shows the data as expected.

image

Additional context

No response

@uhthomas uhthomas added the bug Something isn't working label May 26, 2023
uhthomas added a commit to uhthomas/grafana-dashboards-kubernetes that referenced this issue May 26, 2023
The default resolution of 30s is too low for some Prometheus metrics servers
and will result in no data returned for the query. An increase to 1m should fix
the issue.

Fixes dotdc#50
@dotdc
Copy link
Owner

dotdc commented May 29, 2023

I added the resolution variable in an attempt to see spikes in a very specific case.
I think it would make more sense to remove it completely and let $__rate_interval do its job.

In case anyone needs it, it's pretty easy to edit a panel to add it back.

@uhthomas
Copy link
Contributor Author

@dotdc In general, I would agree. Though I've seen the rate internal variable also not work and I don't think it's configurable by the user?

@uhthomas
Copy link
Contributor Author

For example, these two panels behave differently without an explicit resolution. They both use auto min resolution and the rate interval, but one renders okay and the other doesn't.

image

@dotdc
Copy link
Owner

dotdc commented May 30, 2023

The $__rate_interval is not configurable, but it should work properly if you remove the $resolution variable in Min step.

What is your Scrape Interval?
Do these panels work with a lower resolution set in Min step (1m)?

If it's the case, maybe the issue is tied to Grafana Mimir and/or your configuration. Maybe it's worth opening an issue there too.

I'll keep the PR as is for now, but I would rather delete the $resolution variable or set the default to 15s rather than 1m.

Doc: https://grafana.com/docs/grafana/latest/datasources/prometheus/template-variables/#use-__rate_interval

@uhthomas
Copy link
Contributor Author

uhthomas commented Jun 1, 2023

Hmm, Grafana agent is configured with a 15s scrape interval.

https://github.com/uhthomas/automata/blob/ed25c6b7a525182f1c626f1fb2feda99b268db65/k8s/unwind/grafana_agent/grafana_agent_list.cue#L22

Though, changing the value to 1m in Grafana for the datasource does seem to fix things! Thanks for the pointers and the help.

@dotdc
Copy link
Owner

dotdc commented Jun 1, 2023

Interesting...
Will add this to the "known issues" list. Feel free to share any further knowledge you may have on this issue in the future.

@dotdc dotdc closed this as completed Jun 1, 2023
dotdc added a commit that referenced this issue Jun 1, 2023
Signed-off-by: David Calvert <david@0xdc.me>
@dotdc
Copy link
Owner

dotdc commented Jun 1, 2023

If anyone has further knowledge on this issue, feel free to open a PR to update the documentation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants