New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
bug 2044622: staticpod pruner: check if the cert directory exists to avoid panic #1297
bug 2044622: staticpod pruner: check if the cert directory exists to avoid panic #1297
Conversation
@ingvagabund: This pull request references Bugzilla bug 2044622, which is valid. The bug has been updated to refer to the pull request using the external bug tracker. 6 validation(s) were run on this bug
Requesting review from QA contact: In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
@@ -305,6 +305,7 @@ func RunOperator(ctx context.Context, controllerContext *controllercmd.Controlle | |||
controllerContext.EventRecorder.WithComponentSuffix("invalid-webhook-certs"), | |||
"/var/run/configmaps/service-ca-bundle/service-ca.crt", | |||
metricscontroller.NewLegacyCNCertsMetricsSyncFunc( | |||
"", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@s-urbaniak you didn't leave instructions on how to update after the changes introduced in openshift/library-go#1289 :). Is it okay to pass an empty prefix?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@p0lyn0mial my bad, i was expecting to do this bump myself :-)
Two comments:
- Yes, it can be empty string. In this case, conditions will not be prefixed for status reporting. This is only important if you have multiple metrics controllers in the same operator or you want to give more context. here, empty string is fine.
- we're a bit smarter now with the detection what certificates are faulty. For this the query can return labels to differentiate. I gave a suggestion below.
@@ -14,7 +14,7 @@ require ( | |||
github.com/openshift/api v0.0.0-20210831091943-07e756545ac1 | |||
github.com/openshift/build-machinery-go v0.0.0-20211221164819-4024613928f1 | |||
github.com/openshift/client-go v0.0.0-20210831095141-e19a065e79f7 | |||
github.com/openshift/library-go v0.0.0-20220113084649-1a3940c8c9da | |||
github.com/openshift/library-go v0.0.0-20220125122342-ff51c8a74c7b |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ok, this points to the 4.9
branch
/retest-required |
pkg/operator/starter.go
Outdated
@@ -305,6 +305,7 @@ func RunOperator(ctx context.Context, controllerContext *controllercmd.Controlle | |||
controllerContext.EventRecorder.WithComponentSuffix("invalid-webhook-certs"), | |||
"/var/run/configmaps/service-ca-bundle/service-ca.crt", | |||
metricscontroller.NewLegacyCNCertsMetricsSyncFunc( | |||
"", | |||
`sum(apiserver_kube_aggregator_x509_missing_san_total{job="apiserver",namespace="default"}) + sum(apiserver_webhooks_x509_missing_san_total{job="apiserver",namespace="default"})`, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
To give the controller clues about what certificates are faulty let's modify the query to be:
sum by (type) (
label_replace(apiserver_webhooks_x509_missing_san_total{job="apiserver", namespace="default"}, "type", "webhooks", "", "")
or
label_replace(apiserver_kube_aggregator_x509_missing_san_total{job="apiserver", namespace="default"}, "type", "aggregation", "", ""))
This will yield results like:
{type="webhooks"}: 123
{type="aggregation"}: 456
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
as discussed OOB let's include a promtool unit test file for this query as well:
# This file declares a unit test for the cert detection query. Execute with:
# $ /bin/prometheus-2.x.y/promtool test rules missing_san_total.yaml
# Unit Testing: missing_san_total.yaml
# SUCCESS
#
# TODO(sur): add support for promtool in api-build-machinery.
evaluation_interval: 1m
tests:
- interval: 1m
input_series:
#
# api-server #1 metrics
#
- # apiserver #1 started observing invalid SANs at minute 4
series: 'apiserver_kube_aggregator_x509_missing_san_total{apiserver="kube-apiserver", endpoint="https", instance="10.0.0.1:6443", job="apiserver", namespace="default", service="kubernetes"}'
values: '0 0 0 0 1'
- series: 'apiserver_webhooks_x509_missing_san_total{apiserver="kube-apiserver", endpoint="https", instance="10.0.0.1:6443", job="apiserver", namespace="default", service="kubernetes"}'
values: '0 0 0 0 0'
#
# api-server #2 metrics
#
- series: 'apiserver_kube_aggregator_x509_missing_san_total{apiserver="kube-apiserver", endpoint="https", instance="10.0.0.2:6443", job="apiserver", namespace="default", service="kubernetes"}'
values: '0 0 0 0 0'
- # apiserver #2 started observing invalid SANs at minute 3
series: 'apiserver_webhooks_x509_missing_san_total{apiserver="kube-apiserver", endpoint="https", instance="10.0.0.2:6443", job="apiserver", namespace="default", service="kubernetes"}'
values: '0 0 0 1 1'
#
# api-server #3 metrics
#
- series: 'apiserver_kube_aggregator_x509_missing_san_total{apiserver="kube-apiserver", endpoint="https", instance="10.0.0.3:6443", job="apiserver", namespace="default", service="kubernetes"}'
values: '0 0 0 0 0'
- # apiserver #3 started observing invalid SANs at minute 4
series: 'apiserver_webhooks_x509_missing_san_total{apiserver="kube-apiserver", endpoint="https", instance="10.0.0.3:6443", job="apiserver", namespace="default", service="kubernetes"}'
values: '0 0 0 0 2'
#
# rogue job emitting this metrics, it must be ignored
#
- series: 'apiserver_webhooks_x509_missing_san_total{endpoint="https-metrics", instance="10.0.128.4:10250", job="unknonw", metrics_path="/metrics", namespace="foobar", node="ci-op-5fhw80bs-465b4-tfs28-worker-b-db2vd", service="foo"}'
values: '0 0 0 1 1'
promql_expr_test:
- eval_time: '0m'
expr: 'sum by (type) (
label_replace(apiserver_webhooks_x509_missing_san_total{job="apiserver", namespace="default"}, "type", "webhooks", "", "")
or
label_replace(apiserver_kube_aggregator_x509_missing_san_total{job="apiserver", namespace="default"}, "type", "aggregation", "", ""))'
exp_samples:
- labels: '{type="webhooks"}'
value: 0
- labels: '{type="aggregation"}'
value: 0
- eval_time: '3m'
expr: 'sum by (type) (
label_replace(apiserver_webhooks_x509_missing_san_total{job="apiserver", namespace="default"}, "type", "webhooks", "", "")
or
label_replace(apiserver_kube_aggregator_x509_missing_san_total{job="apiserver", namespace="default"}, "type", "aggregation", "", ""))'
exp_samples:
- labels: '{type="webhooks"}'
value: 1
- labels: '{type="aggregation"}'
value: 0
- eval_time: '4m'
expr: 'sum by (type) (
label_replace(apiserver_webhooks_x509_missing_san_total{job="apiserver", namespace="default"}, "type", "webhooks", "", "")
or
label_replace(apiserver_kube_aggregator_x509_missing_san_total{job="apiserver", namespace="default"}, "type", "aggregation", "", ""))'
exp_samples:
- labels: '{type="webhooks"}'
value: 3
- labels: '{type="aggregation"}'
value: 1
Modify the query to give the controller clues about what certificates are faulty
5f110b2
to
dd71f12
Compare
/lgtm |
@ingvagabund: The following test failed, say
Full PR test history. Your PR dashboard. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here. |
@ingvagabund: This pull request references Bugzilla bug 2044622, which is valid. 6 validation(s) were run on this bug
Requesting review from QA contact: In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
@ingvagabund: This pull request references Bugzilla bug 2044622, which is valid. 6 validation(s) were run on this bug
Requesting review from QA contact: In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
/approve |
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: ingvagabund, p0lyn0mial, sttts The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
/retest-required Please review the full test history for this PR and help us cut down flakes. |
/label cherry-pick-approved |
@ingvagabund: All pull requests linked via external trackers have merged:
Bugzilla bug 2044622 has been moved to the MODIFIED state. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
To bring openshift/library-go#1296