[controller-manager] Switch Shoot controller to controller-runtime

[rfranzke]

How to categorize this PR?

/area dev-productivity scalability
/kind enhancement

What this PR does / why we need it:
Refactor the Shoot controller to controller-runtime. It consists of several reconcilers, so there are some empty separator commits to allow easy navigation through the PR.

Which issue(s) this PR fixes:
Part of #4251

Special notes for your reviewer:
Generally, we want to follow this cookbook while refactoring existing controllers:

1. Add documentation
2. Add integration test based on envtest
3. Switch controller to controller-runtime

Release note:

NONE

[rfranzke]

@rfranzke: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
pull-gardener-e2e-kind 4259ef1 link true /test pull-gardener-e2e-kind
Full PR test history. Your PR dashboard. Command help for this repository. Please help us cut down on flakes by linking this test failure to an open flake report or filing a new flake report if you can't find an existing one. Also see our testing guideline for how to avoid and hunt flakes.

This failed because of

    {"level":"info","ts":"2022-09-06T04:57:48.374Z","logger":"test","msg":"Last Operation","shoot":"garden-local/e2e-rotate","lastOperation":"&LastOperation{Description:Waiting until the Kubernetes API server can connect to the Shoot workers,LastUpdateTime:2022-09-06 04:44:38 +0000 UTC,Progress:87,State:Processing,Type:Create,}"}

which is unrelated to the PR. There are no logs for the vpn-shoot pod in the shoot's kube-system namespace: https://gcsweb.gardener.cloud/gcs/gardener-prow/pr-logs/pull/gardener_gardener/6620/pull-gardener-e2e-kind/1567006984274710528/artifacts/shoot--local--e2e-rotate/machine-shoot--local--e2e-rotate-local-64967-h95ls/pods/ since it wasn't deployed at all.
The vpn-shoot is only deployed after vpn-seed-server, however this failed with

2022-09-06T04:39:23.513553545Z stderr F {"level":"error","ts":"2022-09-06T04:39:23.513Z","logger":"controller.shoot.shoot","msg":"Error","name":"e2e-rotate","namespace":"garden-local","operation":"reconcile","flow":"Shoot cluster reconciliation","task":"Deploying vpn-seed-server","error":"retry failed with context deadline exceeded, last error: Internal error occurred: failed calling webhook \"controlplaneexposure.local.extensions.gardener.cloud\": failed to call webhook: Post \"https://gardener-extension-provider-local.extension-provider-local-98vjx.svc:443/controlplaneexposure?timeout=10s\": dial tcp 10.2.166.84:443: connect: connection refused","stacktrace":"runtime.goexit\n\truntime/asm_amd64.s:1594"}

Probably the extension was shortly unavailable due to scaling/restart.
We should definitely add a dependency for the "deploy vpn-shoot" task before we wait for the VPN connection to be established. I will open a separate PR.

/test pull-gardener-e2e-kind

[timebertt]

/assign

[timebertt]

From a joint review with @acumino (made it until commit Update documentation for quota reconciler).

[timebertt]

Humongous but nice PR 😄

[rfranzke]

Awesome, thanks @timebertt, highly appreciated! I have rebased the PR and addressed all your comments (except for one which is still unresolved) - please have another look :)

[timebertt]

Awesome, one remaining comment :)

[timebertt]

Thanks for addressing all my suggestions!
/lgtm

[gardener-prow]

New changes are detected. LGTM label has been removed.

[rfranzke]

@rfranzke can you rebase to get this in?

rebased now, adding the labels to make Prow merge this :)

/lgtm
/approve

[gardener-prow]

@rfranzke: you cannot LGTM your own PR.

In response to this:

@rfranzke can you rebase to get this in?

rebased now, adding the labels to make Prow merge this :)

/lgtm
/approve

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[gardener-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: rfranzke

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [rfranzke]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment