Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Reject invalid PrometheusRule objects beforehand rather than failing the reconciliation #5191

Closed
simonpasquier opened this issue Nov 25, 2022 · 3 comments · Fixed by #5221
Closed

Reject invalid PrometheusRule objects beforehand rather than failing the reconciliation #5191

simonpasquier opened this issue Nov 25, 2022 · 3 comments · Fixed by #5221
Assignees
@slashpai
Labels
help wanted kind/feature

Comments

@simonpasquier
Copy link
Contributor

What is missing?

When the Thanos/Prometheus controllers detect an invalid PrometheusRule object (invalid PromQL expression for instance), they fail the reconciliation (and for Prometheus, the Reconciled condition will be set to False). Instead the controllers should reject the invalid object and proceed (just like we do for service/pod monitors and probes).

Why do we need it?

In self-service environments, blocking the reconciliation means that one single user can block all other users if they push a bad rule.

Environment

  • Prometheus Operator version: N/A

Anything else we need to know?:

Probably a good time to refactor the code handling the rules which is duplicated between pkg/prometheus and pkg/thanos.
@slashpai slashpai self-assigned this Nov 25, 2022
slashpai added a commit to slashpai/prometheus-operator that referenced this issue Dec 12, 2022
@slashpai
pkg/prometheus: Update selectRules method to be robust
f1ad186 
Fixes prometheus-operator#5191

Signed-off-by: Jayapriya Pai <slashpai9@gmail.com>
@slashpai slashpai mentioned this issue Dec 12, 2022
Merged
5 tasks
@simonpasquier simonpasquier mentioned this issue Dec 20, 2022
Closed
slashpai added a commit to slashpai/prometheus-operator that referenced this issue Dec 21, 2022
@slashpai
pkg/prometheus: Update selectRules method to be robust
9f21d50 
Fixes prometheus-operator#5191

Signed-off-by: Jayapriya Pai <slashpai9@gmail.com>
slashpai added a commit to slashpai/prometheus-operator that referenced this issue Dec 21, 2022
@slashpai
pkg/prometheus: Update selectRules method to be robust
001ddce 
Fixes prometheus-operator#5191

Signed-off-by: Jayapriya Pai <slashpai9@gmail.com>
slashpai added a commit to slashpai/prometheus-operator that referenced this issue Dec 27, 2022
@slashpai
pkg/prometheus: Update selectRules method to be robust
535734b 
pkg/thanos: Update selectRules method to be robust

Fixes prometheus-operator#5191

Signed-off-by: Jayapriya Pai <slashpai9@gmail.com>
@1it
Copy link

1it commented Jan 17, 2023

Erm… Sorry, this is a four years old problem, already. And still not going to be solved?

@simonpasquier
Copy link
Contributor Author

@1it you missed #5221 which is going to fix it :)

@1it
Copy link

1it commented Jan 17, 2023

Ah, OK. Thanks! I was about to start fixing this myself. 😅

slashpai added a commit to slashpai/prometheus-operator that referenced this issue Jan 19, 2023
@slashpai
pkg/operator: create Select method for rule selection
9636cd0 
reorganized rule selection code from prometheus
and thanos package to reduce duplicate code.

This change will also make sure
there is reconcillation failure on controller
when a bad PrometheusRule object is created.
Instead of a reconcillation failure controller
will just reject the bad PrometheusRule.

Fixes prometheus-operator#5191

Signed-off-by: Jayapriya Pai <slashpai9@gmail.com>
slashpai added a commit to slashpai/prometheus-operator that referenced this issue Jan 20, 2023
@slashpai
pkg/operator: create Select method for rule selection
d0c112f 
reorganized rule selection code from prometheus
and thanos package to reduce duplicate code.

This change will also make sure
there is no reconcillation failure on controller
when a bad PrometheusRule object is created.
Instead of a reconcillation failure controller
will just reject the bad PrometheusRule.

Fixes prometheus-operator#5191

Signed-off-by: Jayapriya Pai <slashpai9@gmail.com>
slashpai added a commit to slashpai/prometheus-operator that referenced this issue Jan 20, 2023
@slashpai
pkg/operator: create Select method for rule selection
df1eefe 
reorganized rule selection code from prometheus
and thanos package to reduce duplicate code.

This change will also make sure
there is no reconcillation failure on controller
when a bad PrometheusRule object is created.
Instead of a reconcillation failure controller
will just reject the bad PrometheusRule.

Fixes prometheus-operator#5191

Signed-off-by: Jayapriya Pai <slashpai9@gmail.com>
slashpai added a commit to slashpai/prometheus-operator that referenced this issue Jan 24, 2023
@slashpai
pkg/operator: create Select method for rule selection
98a2f7f 
reorganized rule selection code from prometheus
and thanos package to reduce duplicate code.

This change will also make sure
there is no reconcillation failure on controller
when a bad PrometheusRule object is created.
Instead of a reconcillation failure controller
will just reject the bad PrometheusRule.

Fixes prometheus-operator#5191

Signed-off-by: Jayapriya Pai <slashpai9@gmail.com>
slashpai added a commit to slashpai/prometheus-operator that referenced this issue Jan 30, 2023
@slashpai
pkg/operator: create Select method for rule selection
06ac5ac 
reorganized rule selection code from prometheus
and thanos package to reduce duplicate code.

This change will also make sure
there is no reconcillation failure on controller
when a bad PrometheusRule object is created.
Instead of a reconcillation failure controller
will just reject the bad PrometheusRule.

Fixes prometheus-operator#5191

Signed-off-by: Jayapriya Pai <slashpai9@gmail.com>
slashpai added a commit to slashpai/prometheus-operator that referenced this issue Jan 30, 2023
@slashpai
pkg/operator: create Select method for rule selection
7a66710 
reorganized rule selection code from prometheus
and thanos package to reduce duplicate code.

This change will also make sure
there is no reconcillation failure on controller
when a bad PrometheusRule object is created.
Instead of a reconcillation failure controller
will just reject the bad PrometheusRule.

Fixes prometheus-operator#5191

Signed-off-by: Jayapriya Pai <slashpai9@gmail.com>
slashpai added a commit to slashpai/prometheus-operator that referenced this issue Mar 7, 2023
@slashpai
pkg/operator: create Select method for rule selection
3184093 
reorganized rule selection code from prometheus
and thanos package to reduce duplicate code.

This change will also make sure
there is no reconcillation failure on controller
when a bad PrometheusRule object is created.
Instead of a reconcillation failure controller
will just reject the bad PrometheusRule.

Fixes prometheus-operator#5191

Signed-off-by: Jayapriya Pai <slashpai9@gmail.com>
slashpai added a commit to slashpai/prometheus-operator that referenced this issue Mar 9, 2023
@slashpai
pkg/operator: create Select method for rule selection
f70b86c 
reorganized rule selection code from prometheus
and thanos package to reduce duplicate code.

This change will also make sure
there is no reconcillation failure on controller
when a bad PrometheusRule object is created.
Instead of a reconcillation failure controller
will just reject the bad PrometheusRule.

Fixes prometheus-operator#5191

Signed-off-by: Jayapriya Pai <slashpai9@gmail.com>
slashpai added a commit to slashpai/prometheus-operator that referenced this issue Mar 9, 2023
@slashpai
pkg/operator: create Select method for rule selection
927b921 
reorganized rule selection code from prometheus
and thanos package to reduce duplicate code.

This change will also make sure
there is no reconcillation failure on controller
when a bad PrometheusRule object is created.
Instead of a reconcillation failure controller
will just reject the bad PrometheusRule.

Fixes prometheus-operator#5191

Signed-off-by: Jayapriya Pai <slashpai9@gmail.com>
slashpai added a commit to slashpai/prometheus-operator that referenced this issue Mar 9, 2023
@slashpai
pkg/operator: create Select method for rule selection
0af5c62 
reorganized rule selection code from prometheus
and thanos package to reduce duplicate code.

This change will also make sure
there is no reconcillation failure on controller
when a bad PrometheusRule object is created.
Instead of a reconcillation failure controller
will just reject the bad PrometheusRule.

Fixes prometheus-operator#5191

Signed-off-by: Jayapriya Pai <slashpai9@gmail.com>
slashpai added a commit to slashpai/prometheus-operator that referenced this issue Mar 9, 2023
@slashpai
pkg/operator: create Select method for rule selection
d522534 
reorganized rule selection code from prometheus
and thanos package to reduce duplicate code.

This change will also make sure
there is no reconcillation failure on controller
when a bad PrometheusRule object is created.
Instead of a reconcillation failure, controller
will just reject the bad PrometheusRule.

Fixes prometheus-operator#5191

Signed-off-by: Jayapriya Pai <slashpai9@gmail.com>
slashpai added a commit to slashpai/prometheus-operator that referenced this issue Mar 10, 2023
@slashpai
pkg/operator: create Select method for rule selection
dd18cc4 
reorganized rule selection code from prometheus
and thanos package to reduce duplicate code.

This change will also make sure
there is no reconcillation failure on controller
when a bad PrometheusRule object is created.
Instead of a reconcillation failure, controller
will just reject the bad PrometheusRule.

Fixes prometheus-operator#5191

Signed-off-by: Jayapriya Pai <slashpai9@gmail.com>
slashpai added a commit to slashpai/prometheus-operator that referenced this issue Mar 10, 2023
@slashpai
pkg/operator: create Select method for rule selection
aa92da4 
reorganized rule selection code from prometheus
and thanos package to reduce duplicate code.

This change will also make sure
there is no reconcillation failure on controller
when a bad PrometheusRule object is created.
Instead of a reconcillation failure, controller
will just reject the bad PrometheusRule.

Fixes prometheus-operator#5191

Signed-off-by: Jayapriya Pai <slashpai9@gmail.com>
ArthurSens pushed a commit to ArthurSens/prometheus-operator that referenced this issue Mar 10, 2023
@slashpai @ArthurSens
pkg/operator: create Select method for rule selection
a99a2f9 
reorganized rule selection code from prometheus
and thanos package to reduce duplicate code.

This change will also make sure
there is no reconcillation failure on controller
when a bad PrometheusRule object is created.
Instead of a reconcillation failure, controller
will just reject the bad PrometheusRule.

Fixes prometheus-operator#5191

Signed-off-by: Jayapriya Pai <slashpai9@gmail.com>
@sastorsl sastorsl mentioned this issue Mar 30, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@slashpai slashpai

Labels
help wanted kind/feature
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

pkg/operator: create Select method for rule selection slashpai/prometheus-operator
3 participants
@simonpasquier @slashpai @1it