Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add dedicated serviceaccount for blackbox-exporter #5543

Merged
merged 1 commit into from Mar 9, 2022

Conversation

timebertt
Copy link
Member

How to categorize this PR?

/area security monitoring
/kind bug

What this PR does / why we need it:

Before, blackbox-exporter in the shoot was not specifying any serviceAccountName and was hence using the kube-system/default service account.
With #5422, the kube-system/default service account specifies automountServiceAccountToken=false and because the grm projected token volume webhook does not mount the default service account, blackbox-exporter runs without any service account token mount.

This causes the probe to fail with errors like this:

Logs for the probe:
ts=2022-03-09T08:24:52.8028413Z caller=main.go:304 module=http_kubernetes_service target=https://kubernetes.default.svc.cluster.local/healthz level=info msg="Beginning probe" probe=http timeout_seconds=9.5
ts=2022-03-09T08:24:52.803475Z caller=http.go:342 module=http_kubernetes_service target=https://kubernetes.default.svc.cluster.local/healthz level=info msg="Resolving target address" ip_protocol=ip4
ts=2022-03-09T08:24:52.8902673Z caller=http.go:342 module=http_kubernetes_service target=https://kubernetes.default.svc.cluster.local/healthz level=info msg="Resolved target address" ip=10.4.0.1
ts=2022-03-09T08:24:52.891821Z caller=main.go:119 module=http_kubernetes_service target=https://kubernetes.default.svc.cluster.local/healthz level=error msg="Error generating HTTP client" err="unable to load specified CA cert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt: open /var/run/secrets/kubernetes.io/serviceaccount/ca.crt: no such file or directory"
ts=2022-03-09T08:24:52.8920028Z caller=main.go:304 module=http_kubernetes_service target=https://kubernetes.default.svc.cluster.local/healthz level=error msg="Probe failed" duration_seconds=0.0889363

Hence, the API server availability from the shoot perspective was always displayed as "down".

This PR introduces a dedicated serviceaccount for blackbox-exporter, which is then mounted as a projected volume and used to authenticate against the API server.
With this, probes are working again.

Which issue(s) this PR fixes:
Part of #4878

Special notes for your reviewer:

/invite @rfranzke @kris94

Release note:

A bug has been fixed that caused the monitoring data to falsely display the API server as unavailable from shoots.

@timebertt timebertt requested a review from a team as a code owner March 9, 2022 11:28
@gardener-robot gardener-robot added area/monitoring Monitoring (including availability monitoring and alerting) related area/security Security related kind/bug Bug size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Mar 9, 2022
@timebertt
Copy link
Member Author

/priority 2
/needs cherry-pick

@gardener-robot gardener-robot added needs/cherry-pick priority/2 Priority (lower number equals higher priority) labels Mar 9, 2022
Copy link
Member

@rfranzke rfranzke left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@rfranzke
Copy link
Member

rfranzke commented Mar 9, 2022

/milestone v1.42

@gardener-robot gardener-robot added this to the v1.42 milestone Mar 9, 2022
Copy link
Contributor

@krgostev krgostev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@rfranzke rfranzke merged commit 71d0538 into gardener:master Mar 9, 2022
@timebertt timebertt deleted the fix-blackbox-exporter branch March 9, 2022 12:00
krgostev pushed a commit to krgostev/gardener that referenced this pull request Apr 21, 2022
krgostev pushed a commit to krgostev/gardener that referenced this pull request Jul 5, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/monitoring Monitoring (including availability monitoring and alerting) related area/security Security related kind/bug Bug priority/2 Priority (lower number equals higher priority) size/XS Denotes a PR that changes 0-9 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants