[gardenlet] Refactor handling of Shoot main controller special cases

[timebertt]

How to categorize this PR?

/area dev-productivity quality
/kind technical-debt

What this PR does / why we need it:

Previously, special cases for confined and jittered shoot reconciliations (see docs) were handled by a few lines of boolean variable combinations, and the reconciliation due tracker.
We faced some bugs and regressions in this logic because it was not covered by tests at all, was not readable, and was not documented at all.
However, this logic is quite important for the stability of large-scale gardener installations.

This PR refactors this logic to make it more readable and well-documented and adds full unit test coverage.
Additionally, we can now use the helpers in both the shoot main reconciler and its event handler.
This allows us to use AddAfter in the event handler, instead of performing a reconciliation that will exit early with RequeueAfter (and allows us to get rid of the reconciliation due tracker).

Co-Authored-By @rfranzke

Which issue(s) this PR fixes:
Follow-up on #7082
Part of #4251

Special notes for your reviewer:

Release note:

NONE

[rfranzke]

/assign

[timuthy]

/assign

[timebertt]

This PR should go into the v1.62.0 release. Otherwise, the reconcileInMaintenanceOnly=false case is partially broken.
cc @oliver-goetz
/milestone v1.62

[gardener-prow]

@timebertt: You must be a member of the gardener/gardener-maintainers GitHub team to set the milestone. If you believe you should be able to issue the /milestone command, please contact your and have them propose you as an additional delegate for this responsibility.

In response to this:

This PR should go into the v1.62.0 release. Otherwise, the reconcileInMaintenanceOnly=false case is partially broken.
cc @oliver-goetz
/milestone v1.62

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[rfranzke]

/assign @ary1992

[rfranzke]

Thanks, we did a joint review with @ary1992

[rfranzke]

Thank you @timebertt for the joint work on this, enjoyed it.

/lgtm
/approve

[timebertt]

Thanks for the fast review, PTAL :)

[gardener-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: rfranzke

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [rfranzke]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[timebertt]

/retest-required

[timuthy]

/unassign
for now since there are enough reviewers and I won't make it in time.

[timebertt]

Sorry folks, something went wrong during the rebase. It should be fixed now.

[acumino]

/lgtm

[gardener-prow]

@timebertt: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-gardener-apidiff	75833b7	link	false	/test pull-gardener-apidiff

Full PR test history. Your PR dashboard. Command help for this repository.
Please help us cut down on flakes by linking this test failure to an open flake report or filing a new flake report if you can't find an existing one. Also see our testing guideline for how to avoid and hunt flakes.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[rfranzke]

/lgtm

[timuthy]

Kudos @timebertt and @rfranzke for touching this spaceship 👨‍🚀 Nice PR!