[GEP-19] Monitoring Stack - Migrating to the `prometheus-operator` #6151

wyb1 · 2022-06-21T09:26:23Z

How to categorize this PR?

/area monitoring
/area documentation
/kind discussion

What this PR does / why we need it:
A proposal on how Gardener can migrate to the prometheus operator.

Which issue(s) this PR fixes:
Fixes #

Special notes for your reviewer:

Release note:

NONE

gardener-prow · 2022-06-21T09:26:27Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

gardener-prow · 2022-06-21T09:26:29Z

@wyb1: The label(s) area/documention cannot be applied, because the repository doesn't have them.

In response to this:

How to categorize this PR?

/area monitoring
/area documention
/kind discussion

What this PR does / why we need it:
A proposal on how Gardener can migrate to the prometheus operator.

Which issue(s) this PR fixes:
Fixes #

Special notes for your reviewer:

Release note:
NONE

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

docs/proposals/19-monitoring.md

timebertt · 2022-06-24T06:24:13Z

/assign
I definitely want to take a look and provide feedback, but will probably only manage to do so next week.

istvanballok

minor wording issues

docs/proposals/19-migrating-to-prometheus-operator.md.md

Co-authored-by: Wesley Bermbach <wesley.bermbach@sap.com> Co-authored-by: Istvan Zoltan Ballok <istvan.zoltan.ballok@sap.com>

wyb1 · 2022-06-27T06:51:39Z

Thanks for the reviews from everyone so far. Question: Should I create new commits instead of force pushing to make the changes easier to track?

docs/proposals/19-migrating-to-prometheus-operator.md.md

docs/proposals/19-monitoring.md

timebertt

Thanks a lot for this proposal, really looking forward to it!
I'm not through with my review, but I already left some questions and thoughts :)

docs/proposals/19-migrating-to-prometheus-operator.md.md

timebertt

Made it through the document and added some more comments.
Overall, it looks pretty good and I'm excited for this :)

In general, it would be good to precisely define all relevant contracts already in this GEP. This will make it easier to agree on something before jumping into the implementation.

Also, please open an umbrella issue with the concrete steps for implementing this proposal once this PR gets merged :)

docs/proposals/19-migrating-to-prometheus-operator.md.md

ialidzhikov · 2022-06-28T14:10:43Z

Thanks for addressing my comments/questions. I think on high level I don't have additional comments, lgtm.

wyb1 · 2022-06-30T12:57:34Z

Also, please open an umbrella issue with the concrete steps for implementing this proposal once this PR gets merged :)

We can use gardener/monitoring#14 as an umbrella issue. I will add items there.

…6293) * `garden` namespace deployment is only needed for second kind cluster In the first kind cluster, the `garden` namespace already exists because it runs the Gardener control plane. Without this, the second client-side apply removes the project labels from the `Namespace`. Co-Authored-By: Tim Ebert <tim.ebert@sap.com> * Enable `ShootVPAEnabledByDefault` admission plugin in local setup Co-Authored-By: Tim Ebert <tim.ebert@sap.com> * Simplify gardenlet bootstrap kubeconfig This makes it usable for locally created `ManagedSeed`s and follows the same pattern like in `example/gardener-local/gardenlet/values-kind2.yaml` Co-Authored-By: Tim Ebert <tim.ebert@sap.com> * Add documentation for `ManagedSeed`s Co-Authored-By: Tim Ebert <tim.ebert@sap.com> * Add example Kubernetes resources for local `ManagedSeed`s Co-Authored-By: Tim Ebert <tim.ebert@sap.com> * Register `node` webhook for shoot clusters Co-Authored-By: Tim Ebert <tim.ebert@sap.com> * Introduce seed-local container image registry We will use this registry as a mirror for shoots such that the images built by Skaffold are accessible from the shoot cluster in case it gets registered as `ManagedSeed`. This is needed because we don't want to push the images built by Skaffold to any official, publicly available registry. In the future, we might even be able to reuse this such that we can speed up the image pull processing times. Co-Authored-By: Tim Ebert <tim.ebert@sap.com> * Allow machine pods to talk to the registry Co-Authored-By: Tim Ebert <tim.ebert@sap.com> * Mutate `containerd` config to import additional configuration files This only applies to newly created nodes. Co-Authored-By: Tim Ebert <tim.ebert@sap.com> * Bump `machine-controller-manager-provider-local` image This includes gardener-attic/machine-controller-manager-provider-local@f2c9319 which allows machine pods to talk to the seed API server. In the local setup, the seed API server is also the garden API server and the gardenlet needs to talk to it to register the `Seed`. Co-Authored-By: Tim Ebert <tim.ebert@sap.com> * Mount `backup-path` volume with `DirectoryOrCreate` mode in `provider-local` This will create the directory if it does not exist which is the case for shoot clusters registered as `ManagedSeed`s. Co-Authored-By: Tim Ebert <tim.ebert@sap.com> * Explicitly delete `Ingress`es in seed deletion This effectively fixes #6062. It's actually a work around since not all seed system components (like the monitoring stack) are deployed via `ManagedResource`s yet. Hence, `gardener-resource-manager` does not clean this up for us and we have to delete the resources manually. We only do it for `Ingress`es now to fix above mentioned bug since the deployment of the monitoring stack is anyways planned to be refactored with [GEP-19](#6151). Co-Authored-By: Tim Ebert <tim.ebert@sap.com> * Only delete seed system resources after all other resources are deleted Otherwise, we might delete important system resources like `PriorityClass`es which can cause extensions to not come up anymore (e.g. after being scaled by VPA). This can result in a deadlock during seed deletion. Co-Authored-By: Tim Ebert <tim.ebert@sap.com> * Make `defaultShoot` function reuseable in `e2e` test packages Co-Authored-By: Tim Ebert <tim.ebert@sap.com> * Add e2e test for `ManagedSeed`s Co-Authored-By: Tim Ebert <tim.ebert@sap.com> * Drop testmachinery-based `ManagedSeed` test This is no longer valuable now that we have an e2e test which can run on each PR and periodically on `master` branch. Co-Authored-By: Tim Ebert <tim.ebert@sap.com> * Address PR review feedback * Address PR review feedback Co-authored-by: Tim Ebert <tim.ebert@sap.com>

timebertt

/lgtm
/hold cancel

gardener-prow · 2022-07-13T05:56:52Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: timebertt

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [timebertt]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

docs/proposals/19-migrating-to-prometheus-operator.md

istvanballok · 2022-07-13T07:39:29Z

/hold
(typo in the link)
otherwise, looks good to me 🎉

docs/proposals/19-migrating-to-prometheus-operator.md

istvanballok · 2022-07-13T12:10:19Z

/lgtm
/hold cancel
🎉

timebertt · 2022-07-13T13:44:39Z

/override pull-gardener-e2e-kind

gardener-prow · 2022-07-13T13:44:42Z

@timebertt: Overrode contexts on behalf of timebertt: pull-gardener-e2e-kind

In response to this:

/override pull-gardener-e2e-kind

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

* 'master' of github.com:gardener/gardener: (51 commits) Switch extension controller to `logr` and streamline/cleanup logs (gardener#6332) Switch `./test/...` packages to `logr` and drop `github.com/sirupsen/logrus` dependency (gardener#6316) Only check shoot conditions during hibernation integration test (gardener#6325) Add dashboard for monitoring conntrack race failures. (gardener#6329) Reconcile quota before rbac (gardener#6326) Update istio to v1.14.1 (gardener#6271) Update gardenlet's base image to alpine:3.16.0 (gardener#6321) Update envoy proxy to v1.21.4 (gardener#6320) Deploy the metrics server to the kind cluster (gardener#6301) Fix tools download for aarch64 (arm64) (gardener#6314) update with latest CA releases (gardener#6295) Add missing unit tests for the predicates provided by the extensions library (gardener#6249) [GEP-19] Monitoring Stack - Migrating to the `prometheus-operator` (gardener#6151) Revert "Recreate DWD deployment if needed" (gardener#6307) Update to golang 1.18.4 (gardener#6300) Cleaned up imports in vpn-seed-server (gardener#6315) Prepare next Dev Cycle v1.52.0-dev Release v1.51.0 Add pre/post reconciliation/deletion hooks for the Worker resource (gardener#6290) Update the supported values in the usage text of the `--leader-election-resource-lock` flag (gardener#6304) ...

gardener-prow bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 21, 2022

gardener-prow bot requested review from DockToFuture and ialidzhikov June 21, 2022 09:26

gardener-prow bot added area/monitoring Monitoring (including availability monitoring and alerting) related kind/discussion Discussion (engaging others in deciding about multiple options) labels Jun 21, 2022

gardener-prow bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. cla: yes Indicates the PR's author has signed the cla-assistant.io CLA. labels Jun 21, 2022

rfranzke changed the title ~~Monitoring GEP~~ [GEP-19] Monitoring Stack - Migrating to the prometheus-operator Jun 21, 2022

rfranzke reviewed Jun 21, 2022

View reviewed changes

wyb1 force-pushed the gep-monitoring branch from 08bb3c4 to 464d3a4 Compare June 21, 2022 12:27

rfranzke self-assigned this Jun 21, 2022

ialidzhikov reviewed Jun 22, 2022

View reviewed changes

istvanballok mentioned this pull request Jun 23, 2022

☂️ Use the prometheus-operator to manage Gardener's monitoring stacks gardener/monitoring#14

Open

10 tasks

gardener-prow bot assigned timebertt Jun 24, 2022

wyb1 force-pushed the gep-monitoring branch from 464d3a4 to cf2e28d Compare June 24, 2022 13:06

istvanballok reviewed Jun 24, 2022

View reviewed changes

Monitoring GEP

411651c

Co-authored-by: Wesley Bermbach <wesley.bermbach@sap.com> Co-authored-by: Istvan Zoltan Ballok <istvan.zoltan.ballok@sap.com>

wyb1 force-pushed the gep-monitoring branch from cf2e28d to 411651c Compare June 27, 2022 06:48

timebertt reviewed Jun 28, 2022

View reviewed changes

docs/proposals/19-migrating-to-prometheus-operator.md.md Outdated Show resolved Hide resolved

docs/proposals/19-migrating-to-prometheus-operator.md.md Outdated Show resolved Hide resolved

docs/proposals/19-monitoring.md Outdated Show resolved Hide resolved

timebertt reviewed Jun 28, 2022

View reviewed changes

wyb1 added 3 commits June 30, 2022 15:25

Add reviewers

54d371c

Improve wording

4429eb4

Grafana sidecar contract

50a2d09

Fix minor nits, typos, and indentation

c482dac

gardener-prow bot removed the lgtm Indicates that a PR is ready to be merged. label Jul 12, 2022

Remove duplicate file extension

d571fe6

Add proposal steps to ToC

80963d2

timebertt approved these changes Jul 13, 2022

View reviewed changes

gardener-prow bot added lgtm Indicates that a PR is ready to be merged. and removed do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. labels Jul 13, 2022

timebertt removed this from the v1.51 milestone Jul 13, 2022

istvanballok reviewed Jul 13, 2022

View reviewed changes

docs/proposals/19-migrating-to-prometheus-operator.md Outdated Show resolved Hide resolved

docs/proposals/19-migrating-to-prometheus-operator.md Outdated Show resolved Hide resolved

gardener-prow bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 13, 2022

*Monitors will be in the shoot namespace

29d4adb

gardener-prow bot removed the lgtm Indicates that a PR is ready to be merged. label Jul 13, 2022

Change grafana -> Grafana

0faf844

istvanballok reviewed Jul 13, 2022

View reviewed changes

docs/proposals/19-migrating-to-prometheus-operator.md Outdated Show resolved Hide resolved

docs/proposals/19-migrating-to-prometheus-operator.md Outdated Show resolved Hide resolved

Fix leftovers from ignoreNamespaceSelectors features

248cf7c

istvanballok reviewed Jul 13, 2022

View reviewed changes

wyb1 added 2 commits July 13, 2022 14:04

Remove references to the monitoring ns

d67c70e

Use less ambiguous language for the monitoring-target

bf07cbf

gardener-prow bot added lgtm Indicates that a PR is ready to be merged. and removed do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. labels Jul 13, 2022

gardener-prow bot merged commit e3a1300 into gardener:master Jul 13, 2022

wyb1 deleted the gep-monitoring branch July 13, 2022 14:00

rfranzke mentioned this pull request Oct 14, 2022

[GEP-21] logging extension #6823

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[GEP-19] Monitoring Stack - Migrating to the `prometheus-operator` #6151

[GEP-19] Monitoring Stack - Migrating to the `prometheus-operator` #6151

wyb1 commented Jun 21, 2022 •

edited

Loading

gardener-prow bot commented Jun 21, 2022

gardener-prow bot commented Jun 21, 2022

timebertt commented Jun 24, 2022

istvanballok left a comment

wyb1 commented Jun 27, 2022

timebertt left a comment

timebertt left a comment

ialidzhikov commented Jun 28, 2022

wyb1 commented Jun 30, 2022

timebertt left a comment

gardener-prow bot commented Jul 13, 2022

istvanballok commented Jul 13, 2022 •

edited

Loading

istvanballok commented Jul 13, 2022

timebertt commented Jul 13, 2022

gardener-prow bot commented Jul 13, 2022

[GEP-19] Monitoring Stack - Migrating to the prometheus-operator #6151

[GEP-19] Monitoring Stack - Migrating to the prometheus-operator #6151

Conversation

wyb1 commented Jun 21, 2022 • edited Loading

gardener-prow bot commented Jun 21, 2022

gardener-prow bot commented Jun 21, 2022

timebertt commented Jun 24, 2022

istvanballok left a comment

Choose a reason for hiding this comment

wyb1 commented Jun 27, 2022

timebertt left a comment

Choose a reason for hiding this comment

timebertt left a comment

Choose a reason for hiding this comment

ialidzhikov commented Jun 28, 2022

wyb1 commented Jun 30, 2022

timebertt left a comment

Choose a reason for hiding this comment

gardener-prow bot commented Jul 13, 2022

istvanballok commented Jul 13, 2022 • edited Loading

istvanballok commented Jul 13, 2022

timebertt commented Jul 13, 2022

gardener-prow bot commented Jul 13, 2022

[GEP-19] Monitoring Stack - Migrating to the `prometheus-operator` #6151

[GEP-19] Monitoring Stack - Migrating to the `prometheus-operator` #6151

wyb1 commented Jun 21, 2022 •

edited

Loading

istvanballok commented Jul 13, 2022 •

edited

Loading