Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mimir / Reads dashboard query scheduler queue duration breakout improvements #7098

Conversation

francoposa
Copy link
Member

What this PR does

Improves the Query Scheduler Time in Queue breakout.
Currently it has too many series to be helpful at a glance, and it's not particularly effective to have 99th percentile and 50th percentile/average on the same panel, because the graph is scaled to accommodate the 99th percentile, and the 50th %ile and average get squashed at the bottom:

Confusing:

Screenshot 2024-01-10 at 4 45 55 pm

The new better version breaks it out into a row with 3 panels, one each for 99th percentile, 50th percentile, and Average:

Better:

Screenshot from 2024-01-10 10-57-12

Also extended the jsonnet function to allow all these panels to be build from the same function.

Which issue(s) this PR fixes or relates to

Fixes #

Checklist

  • Tests updated.
  • Documentation added.
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX].
  • about-versioning.md updated with experimental features.

@francoposa francoposa requested a review from a team as a code owner January 10, 2024 19:03
Copy link
Contributor

@56quarters 56quarters left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

@charleskorn charleskorn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Comment on lines +1035 to +1044
legendFormat: '%sth Percentile: {{ %s }}' % [std.lstripChars(percentile, '0.'), labelBreakouts],
refId: 'A',
}
for percentile in percentiles
];
local averageTargets = [
{
expr: averageExprTmpl % [metricName, selector, labelBreakouts, multiplier, metricName, selector, labelBreakouts],
format: 'time_series',
legendFormat: 'Average: {{ %s }}' % [labelBreakouts],
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit] If we're only showing one percentile or just the average on a panel, the %sth Percentile: / Average: prefix is superfluous

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I figured I would leave it for the option as the func here can still be used show them both on the same chart if desired.

@francoposa francoposa merged commit 507d8b7 into main Jan 11, 2024
30 checks passed
@francoposa francoposa deleted the francoposa/mimir-reads-dashboard-query-scheduler-queue-duration-breakout-improvements branch January 11, 2024 00:46
@charleskorn
Copy link
Contributor

@francoposa just noticed this only applies to the "Reads" dashboard, but not the "Remote Ruler Reads" dashboard. Should we adjust the way these dashboards are built so we don't have to remember to make changes in both dashboards perhaps?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants