Refactor Gardener components to native controller-runtime components

[rfranzke]

How to categorize this issue?

/area open-source dev-productivity scalability
/kind cleanup technical-debt

What would you like to be added:

• gardener-scheduler, gardener-controller-manager and gardenlet should be refactored such that they become native controller-runtime components.
• gardener-resource-manager, gardener-admission-controller, gardener-seed-admission-controller are already native controller-runtime components.
• gardener-apiserver is special anyways because it's neither a controller nor a webhook server.

Why is this needed:

• Less code
• Less duplication/hand-crafted logic
• Easier maintenance
• Metrics out-of-the-box
• Streamlined logging

Steps

• all controllers should implement the reconcile.Reconciler interface
  • Cleanup duplicated watches in gardenlet (1) #4503
  • Refactor shoot controller to reconcile.Reconciler #4891
• remove current dummy controller metrics
  • ... in gardener-scheduler: [scheduler] Introduce controller-runtime manager and switch Shoot controller #4320
  • ... in gardener-controller-manager, gardenlet: Remove dummy controller metrics #4913
• check whether we can remove active worker counting in our controllers (dummy graceful termination)
• streamline logging – replace logrus with logr
  • Prepare switch to logr #4925
  • Add logging developer guideline #5175
  • add logcheck tool to ensure even number of args for Info, Error and WithValues calls, etc.: Add logcheck tool #5442
  • scheduler
    • [scheduler] Introduce controller-runtime manager and switch Shoot controller #4320
  • controller-manager
    • inject logr logger into reconcile context to make it retrievable by logf.FromContext(ctx): Switch controller-manager to logr (1) #5057
    • Switch controller-manager to logr (2) #5268
    • Switch controller-manager to logr (3) #5290
    • Switch controller-manager to logr (4) #5488
    • Switch controller-manager to logr (5) #6253
  • gardenlet
    • Switch gardenlet to logr (1) #6259
    • Switch gardenlet to logr (2) #6260
    • Switch gardenlet to logr (3) #6264
    • Switch gardenlet to logr (4) #6265
    • Switch gardenlet to logr (5) #6266
    • Switch gardenlet to logr (6) #6267
    • Switch gardenlet to logr (7) #6278
    • Switch gardenlet to logr (8) #6282
  • ./test/... packages
    • Switch ./test/... packages to logr and drop github.com/sirupsen/logrus dependency #6316
  • Configurable log level/format everywhere #5191
  • Switch extension controller to logr and streamline/cleanup logs #6332
  • Use debug log level in all tests #6380
• introduce c-r Manager, including leader election, client/cache creation, enhance healthz/readyz endpoints to include checks for informer/watch healthiness
  • controller-manager
    • [controller-manager] Introduce controller-runtime manager and switch CloudProfile controller #6333
  • scheduler
    • [scheduler] Introduce controller-runtime manager and switch Shoot controller #4320
  • gardenlet
    • [gardenlet] Introduce controller-runtime manager #6688
• add native c-r controllers to manager including predicates, event handlers, etc. as needed (can be done step by step):
  • controller-manager
    • [controller-manager] Switch Bastion controller to controller-runtime and introduce common indexers #6358
    • [controller-manager] Switch CertificateSigningRequest controller to controller-runtime #6615
    • [controller-manager] Introduce controller-runtime manager and switch CloudProfile controller #6333
      • [controller-manager] Add documentation and integration test for CloudProfile controller #6360
    • [controller-manager] Switch ControllerDeployment controller to controller-runtime #6462
    • [controller-manager] Switch ControllerRegistration controller to controller-runtime #6629
    • [controller-manager] Switch Event controller to controller-runtime #6550
    • [controller-manager] Switch ExposureClass controller to controller-runtime #6423
    • [controller-manager] Switch ManagedSeedSet controller to controller-runtime #6742
    • [controller-manager] Drop Plant API and controller #6348
    • [controller-manager] Switch main Project controller to controller-runtime #6645
      • [controller-manager] Switch remaining Project controllers to controller-runtime #6667
    • [controller-manager] Switch Quota controller to controller-runtime #6391
    • [controller-manager] Switch SecretBinding controller to controller-runtime #6621
    • [controller-manager] Switch Seed controller to controller-runtime #6636
    • [controller-manager] Switch Shoot controller to controller-runtime #6620
  • scheduler
    • [scheduler] Introduce controller-runtime manager and switch Shoot controller #4320
  • gardenlet (introduce filtered cache with cache.Options.SelectorsByObject for watched objects)
    • [gardenlet] Switch BackupBucket controller to controller-runtime #6837
    • [gardenlet] Switch BackupEntry controller to controller-runtime #6926
    • [gardenlet] Switch Bastion controller to controller-runtime #6839
    • [gardenlet] Switch ControllerInstallation controller to controller-runtime #6733
    • [gardenlet] Switch extensions controller to controller-runtime #6801
    • [gardenlet] Switch ManagedSeed controller to controller-runtime #7108
    • [gardenlet] Switch NetworkPolicy controller to controller-runtime #6991
    • [gardenlet] Switch Seed controller to controller-runtime #6836
    • [gardenlet] Switch Shoot migration controller to controller-runtime #7062
      • [gardenlet] Switch Shoot main controller to controller-runtime #7082
      • [gardenlet] Refactor handling of Shoot main controller special cases #7189
      • [gardenlet] Switch Shoot care controller to controller-runtime #7206
    • [gardenlet] Switch shootsecret controller to controller-runtime #6819
• expose controller-runtime metrics (related to ☂️-issue for gardener component metrics/alerts/dashboards #2815)
  • Scrape extension pods in seed gardener-extension-provider-aws#428 (gives a small preview at what metrics we can get from that)
  • ... in gardener-scheduler: [scheduler] Introduce controller-runtime manager and switch Shoot controller #4320
  • ... in gardener-controller-manager: [controller-manager] Introduce controller-runtime manager and switch CloudProfile controller #6333
  • ... in gardenlet: [gardenlet] Introduce controller-runtime manager #6688
  • adapt seed-prometheus scrape config to scrape gardenlet with http scheme: Fix gardenlet metrics scraping by seed-prometheus #6979
  • Add dashboards for controller-runtime metrics #7180
• streamlining and reduction of development complexity
  • Streamline gardener-admission-controller entrypoint #6706
  • Streamline gardener-controller-manager controllers after refactorings #6699
  • Streamline gardener-scheduler entrypoint and controllers #6690
  • Streamline gardener-seed-admission-controller entrypoint #6708
  • Streamline gardener-resource-manager entrypoint and introduce component configuration #6865
  • Streamline webhook handler registration in gardener-admission-controller #6873
  • Merge gardener-seed-admission-controller logic into gardener-resource-manager #7053
• Support HTTPS for /healthz, /readyz, /metrics and /debug endpoints #2829
• controller-manager does not respect config.server.https #4605
• add timeout for initial cache syncs in controller/component constructors (similar to Time limit for cache sync on startup and controller reconciliations gardener-attic/gardener-resource-manager#102) or make sure that c-r does this out of the box -> Adds timeouts for reconciler contexts #7147
• use predicate.GenerationChangedPredicate instead of comparing metadata.generation and status.observedGeneration and stop returning nil after adding finalizers in order to prevent duplicated reconciliations (basically implement option b) from Harmonize finalizer addition across all controllers #4730 where it is not done yet) -> Harmonize the remaining finalizer additions of controllers #7056

[timebertt]

All controllers should implement the reconcile.Reconciler interface (already done for GSCHED, GCM)

I'm currently working on this for the remaining gardenlet controllers as part of #2822

[timebertt]

All controllers should implement the reconcile.Reconciler interface (already done for GSCHED, GCM)

I'm currently working on this for the remaining gardenlet controllers as part of #2822

It's mostly done with #4503.
Though, the shoot controller should be further refactored into individual reconcile.Reconcilers instead of using reconcile.Funcs.

[rfranzke]

/assign @timebertt
/roadmap internal
/milestone 2021-Q4

[timebertt]

/in-progress

[timebertt]

refactor remaining controllers to reconcile.Reconciler interface (extensions controller: controllerinstallation/shootstate)

I tried to accomplish this, but I faced a few difficulties with the current implementation/architecture of the package:

• the controller actually contains 2 controllers: a controllerinstallation controller and a shootstate controller, but both controllers are initialized and built from the shared artifacts map
• the controllerinstallation reconcilers share a common state (kindToRequiredTypes) which is not so easy to do if they are split into dedicated structs

Given, that this controller package anyways needs significant restructuring work (actually it almost needs to be rebuilt from scratch) in order to make it work with native c-r constructs, I decided to not refactor this right now to reconcile.Reconciler. This provides only little to no value right now over reconcile.Func, as it cannot be simply added as a native controller to a c-r manager, so it needs to be touched again anyways.
Once we are able to leverage all the c-r goodies to get rid of all our custom coding, we can rebuild it in a clean way.

So this will be done in a dedicated PR as part of step

add native c-r controllers to manager including predicates, event handlers, etc. as needed (can be done step by step)

[timebertt]

check whether we can remove active worker counting in our controllers (dummy graceful termination)

I look into this and played around with it a little bit. The result:

• current implementation doesn't prevent the process from exiting, if items are left in the queue or workers have not finished yet
• too much effort to fix current implementation
• could remove it now in favor of graceful termination feature, that we will get with native c-r controllers
• but it also doesn't hurt, so let's keep it for now and simply remove it together in individual refactorings of controllers

Hence, I marked the step as completed for now.
cc @rfranzke

[gardener-ci-robot]

The Gardener project currently lacks enough contributors to adequately respond to all issues and PRs.
This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close

/lifecycle stale

[rfranzke]

/remove-lifecycle stale

[timebertt]

Now that #7147 is merged, we can close this issue, correct? :)

[shafeeqes]

Yes 🚀
/close

[gardener-prow]

@shafeeqes: Closing this issue.

In response to this:

Yes 🚀
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.