Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reconcile errors "prometheus rule not found" #992

Closed
ArthurSens opened this issue Nov 27, 2023 · 2 comments · Fixed by #993
Closed

Reconcile errors "prometheus rule not found" #992

ArthurSens opened this issue Nov 27, 2023 · 2 comments · Fixed by #993

Comments

@ArthurSens
Copy link
Contributor

ArthurSens commented Nov 27, 2023

Quite often (read 100% of the time) when we create a new SLO, Pyrra sends error logs and increments the metric controller_runtime_reconcile_total{result="error"}, which consequentially triggers our SLO about Pyrra reconciliations 😅

Error logs
2023-11-27T11:57:48Z ERROR Reconciler error {"controller": "servicelevelobjective", "controllerGroup": "pyrra.dev", "controllerKind": "ServiceLevelObjective", "ServiceLevelObjective": {"name":"observability-thanos-stack-thanos-query-latency","namespace":"observability"}, "namespace": "observability", "name": "observability-thanos-stack-thanos-query-latency", "reconcileID": "3d51a06b-674b-4701-be57-9d0ac0cc1864", "error": "failed to create prometheus rule: PrometheusRule.monitoring.coreos.com \"observability-thanos-stack-thanos-query-latency\" not found"}

Ideally, Pyrra would not send error logs or increment error metrics when PrometheusRules aren't found, and would just create them as expected :)

Pyrra version: 0.7.1

@9SMTM6
Copy link

9SMTM6 commented Mar 4, 2024

has this been released? in that case perhaps this can be closed.

@metalmatze
Copy link
Member

Yes, it was part of v0.7.2 and all later patch releases.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants