[GEP-19] Add sidecar to Plutono for fetching dashboard ConfigMaps dynamically
#9624
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Skipping CI for Draft Pull Request. |
gardener-prow
bot
added
do-not-merge/work-in-progress
46 tasks
rfranzke
force-pushed
the
plutono-sidecar
branch
from
1bb6cab
to
1d8f06c
Compare
gardener-prow
bot
removed
the
do-not-merge/work-in-progress
rfranzke
force-pushed
the
plutono-sidecar
branch
from
1d8f06c
to
20fecdd
Compare
|
/assign |
ialidzhikov
reviewed
Apr 23, 2024
There was a problem hiding this comment.
We started a review with @dimitar-kostadinov . We managed to review the 5/10 commits so far.
rfranzke
force-pushed
the
plutono-sidecar
branch
from
20fecdd
to
1a59d07
Compare
|
LGTM label has been added. Git tree hash: 10ea8b1ba35590734d854323a937b18b85e28cde
|
|
/approve |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: rfranzke The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
gardener-prow
bot
added
approved
Previously, there were two "folders" for dashboards, "garden" and "global". We don't follow this pattern for the seed or shoot plutono, so let's also drop it for the garden plutono for simplicity. Now, all dashboards are added in the same folder.
See https://github.com/kiwigrid/k8s-sidecar?tab=readme-ov-file#what for details how this works/what this does.
When the sidecar updates the dashboards in the file system, it needs to call an API of Plutono to make it reload the dashboards. This is only available in the admin API, and we need credentials to call it. This was disabled previously. Now, we enable it and generate a basic auth secret which is auto-rotated each `30d`. The API is only accessible within the cluster itself, and blocked via Ingress. Also, no other pod can talk to Plutono (protected via network policies), hence, access to this API is basically only possible from within the pod (by the sidecar) and from human operators with port-forwarding.
The sidecar is a controller that watches `configmaps`, so it needs access/permissions to do so.
The dashboard `ConfigMap` created by Gardener will be picked up by the sidecar and no longer directly mounted into the pod (next commit). For a few releases, when computing this `ConfigMap`, the old (now deprecated) extensions contract for providing dashboards will be respected, i.e., they will be collected and added to this `ConfigMap`. When an extension switches to the approach, this will no longer happen, but the sidecar will pick it up.
rfranzke
force-pushed
the
plutono-sidecar
branch
from
1a59d07
to
8761f07
Compare
|
New changes are detected. LGTM label has been removed. |
gardener-prow
bot
removed
the
needs-rebase
rfranzke
added
needs-rebase
gardener-prow
bot
removed
the
needs-rebase
|
/test pull-gardener-e2e-kind-ha-single-zone |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
How to categorize this PR?
/area monitoring
/kind enhancement
What this PR does / why we need it:
This PR implements this aspect of GEP-19. Today, when Plutono is deployed, all dashboards must be collected and mounted into the pod. With this PR, dashboards can be dynamically added/removed by labelling
ConfigMaps containing the dashboard JSON docs withdashboard.monitoring.gardener.cloud/{garden,seed,shoot}=true, without the necessity to restart the pod.The approach is to make use of https://github.com/kiwigrid/k8s-sidecar, a sidecar watching for such labelled
ConfigMaps, which writes the JSON docs to files in a shared folder (from which the Plutono container is picking up the dashboards). In order to make Plutono reload the dashboards when something changed, the sidecar calls the respective "reload" admin API.This is only possible with credentials, and the API was basically disabled previously. Now, we enable it and generate a basic auth secret which is auto-rotated each
30d. The API is only accessible within the cluster itself, and blocked via Ingress. Also, no other pod can talk to Plutono (protected via network policies), hence, access to this API is basically only possible from within the pod (by the sidecar) and from human operators with port-forwarding.Which issue(s) this PR fixes:
Part of #9065
Special notes for your reviewer:
/cc @ScheererJ
FYI @istvanballok @rickardsjp @vicwicker
Note
You can test it locally by running
make kind-operator-up operator-upand thenkubectl apply -f example/operator/20-garden.yamlforGardens, ormake kind-up gardener-upfor seeds, or an additionalkubectl apply -f example/provider-local/shoot.yamlforShoots.Release note: