Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[gardenlet] Introduce controller-runtime manager #6688

Merged
merged 30 commits into from
Sep 23, 2022

Conversation

rfranzke
Copy link
Member

How to categorize this PR?

/area dev-productivity scalability
/kind enhancement

What this PR does / why we need it:
Introduce controller-runtime manager in gardenlet.
No controllers have been refactored yet.

Which issue(s) this PR fixes:
Part of #4251

Special notes for your reviewer:
Similar to #6333.

Release note:

`gardenlet`s component configuration API has been changed in the following breaking ways:
- `.server.http` has been split into `server.{healthProbes,metrics}` (health endpoints and metrics are now served on different ports)
- `.server.https` has been removed
`gardenlet` serves health endpoints and metrics on different ports now. Adapt your scrape configs accordingly to port `metrics`.

@gardener-prow gardener-prow bot added area/dev-productivity Developer productivity related (how to improve development) area/scalability Scalability related kind/enhancement Enhancement, improvement, extension cla: yes Indicates the PR's author has signed the cla-assistant.io CLA. labels Sep 19, 2022
@gardener-prow gardener-prow bot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Sep 19, 2022
@acumino acumino self-assigned this Sep 19, 2022
@oliver-goetz
Copy link
Member

/assign @oliver-goetz

pkg/client/kubernetes/client.go Outdated Show resolved Hide resolved
cmd/gardenlet/app/app.go Show resolved Hide resolved
Copy link
Member

@oliver-goetz oliver-goetz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I went through it until commit Make kubernetes.FallbackClient public and found only minor things so far.

These tests don't belong to the `controller` packages but rather to the `imagevector` package.
Pointer types are removed now just like in the component configs for GCM, scheduler, GAC, etc.
The config structure is similar to admission-controller and controller-manager now and fits to the `manager.Options` structure as well.
So that it can be reused in `gardenlet` later on
Previously, this was logged in the gardenlet's `app.go`, but the seed controller is the only entity which requires the client certificate object. So, better move this here instead of transporting the information through many different intermediate instances.
The health manager in gardenlet will be registered as `manager.Runnable` in the future.
The client map for shoots in gardenlet will be registered as `manager.Runnable` in the future.
This function will be ran as `manager.Runnable`, so make sure that it blocks as long as it is actively waiting (previously the function was starting a goroutine and returning immediately, but this doesn't play nicely with `manager.Runnable` way of running code).
So that garden client is no longer fetched from client map
The gardenlet will have the garden and seed clients already available when creating the client map, so we can pass them right away.
Typically, runnables added to the a manager are all started in parallel, hence uncontrolled. In the gardenlet case, we first want to perform a few bootstrap tasks (like validating the seed config or fetching a client certificate for the garden cluster) before we start the actual runnables (controllers, etc.).
The introduced `Runner` is capable of achieving this goal by first starting all bootstrap runnables and only adding the actual runnables once all bootstrap runnables completed successfully.
Earlier, this client was used when instantiating a new `kubernetes.Interface`, however the gardenlet will now instantiate the clients using the controller-runtime's manager. Hence, we have to make this function public so that we can use provide it to the `manager.Options.NewClientFunc`.
We will be only using the HTTP servers that already come with the controller-runtime's `Manager`, so we don't need this package anymore.
We don't need the garden or seed client maps anymore, and we allso don't need the delegating client map. Only shoot client maps will remain.
The will be provided with the `cluster.Cluster` or client objects from the factory when the controllers are instantiated.
Also, there will be no ClientMap anymore for garden/seed, so related coding can be cleaned up/deleted.
This is to have aligned names of the clients across code and components. Also, "ClientSet` hints that it's not `client.Client` but `kubernetes.Interface`.
This is to have aligned names of the clients across code and components. Also, "ClientSet` hints that it's not `client.Client` but `kubernetes.Interface`.
In the future, there will be no client maps for garden or seed clusters anymore (no longer needed). Hence, let's make sure that this client map is for shoot clusters only.
This is to have aligned names of the clients across code and components. Also, "Client` hints that it's `client.Client` and not `kubernetes.Interface`.
The gardenlet will have the seed client always available, no need to initialize it anymore.
@gardener-prow
Copy link
Contributor

gardener-prow bot commented Sep 20, 2022

@rfranzke: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
pull-gardener-check-vulnerabilities 11c7f90 link false /test pull-gardener-check-vulnerabilities

Full PR test history. Your PR dashboard. Command help for this repository.
Please help us cut down on flakes by linking this test failure to an open flake report or filing a new flake report if you can't find an existing one. Also see our testing guideline for how to avoid and hunt flakes.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

Copy link
Member

@acumino acumino left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@gardener-prow gardener-prow bot added the lgtm Indicates that a PR is ready to be merged. label Sep 20, 2022
Copy link
Member

@oliver-goetz oliver-goetz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@gardener-prow
Copy link
Contributor

gardener-prow bot commented Sep 20, 2022

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: oliver-goetz

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@gardener-prow gardener-prow bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Sep 20, 2022
@gardener-prow gardener-prow bot merged commit 036fa06 into gardener:master Sep 23, 2022
@rfranzke rfranzke deleted the enh/cr-gardenlet branch September 23, 2022 13:08
@rfranzke rfranzke changed the title Introduce controller-runtime manager in gardenlet [gardenlet] Introduce controller-runtime manager Sep 23, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/dev-productivity Developer productivity related (how to improve development) area/scalability Scalability related cla: yes Indicates the PR's author has signed the cla-assistant.io CLA. component/gardener Gardener kind/enhancement Enhancement, improvement, extension lgtm Indicates that a PR is ready to be merged. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants