Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ruler: report failed eval on any 5xx status #2053

Merged

Conversation

ortuman
Copy link
Contributor

@ortuman ortuman commented Jun 8, 2022

Signed-off-by: Miguel Ángel Ortuño ortuman@gmail.com

What this PR does

When remote operational mode is enabled, ruler reports failed rule evaluation metric only when query-frontend returns a 500 status code.

This PR adapts it to consider all 5xx status codes.

Which issue(s) this PR fixes or relates to

Fixes N/A

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

@ortuman ortuman requested review from stevesg and pstibrany June 8, 2022 09:31
@ortuman ortuman force-pushed the ortuman/ruler/report-failed-eval-on-any-5xx-status-code branch from 667435e to 9dd4944 Compare June 8, 2022 10:24
Copy link
Contributor

@stevesg stevesg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is a useful change, it means we can consider increasing the RulerTooManyFailedQueries alert to critical in order to get coverage of issues with the ruler read path.

Copy link
Contributor

@replay replay left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This make sense to me, thanks!

Copy link
Collaborator

@pracucci pracucci left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM (modulo a comment about the CHANGELOG)

CHANGELOG.md Outdated Show resolved Hide resolved
Signed-off-by: Miguel Ángel Ortuño <ortuman@gmail.com>
Signed-off-by: Miguel Ángel Ortuño <ortuman@gmail.com>
@ortuman ortuman force-pushed the ortuman/ruler/report-failed-eval-on-any-5xx-status-code branch from 9dd4944 to ec1ca9f Compare June 9, 2022 07:09
Copy link
Collaborator

@pracucci pracucci left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@pracucci pracucci enabled auto-merge (squash) June 9, 2022 07:18
@pracucci pracucci merged commit e17cb56 into main Jun 9, 2022
@pracucci pracucci deleted the ortuman/ruler/report-failed-eval-on-any-5xx-status-code branch June 9, 2022 07:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants