feat: set leader election when single controller deployments are configured

[pmalek]

What this PR does / why we need it:

This PR ensures we enable leader election when Admin API service discovery is enabled.

Which issue this PR fixes:

Fixes #3528

Special notes for your reviewer:

When implemented this way (the same we used to do it for other configurations, dbless or not) KIC's deployment has only 1 ready Pod - the leader - and all the other ones are not ready, because the readiness probe set to

kubernetes-ingress-controller/internal/manager/run.go

Lines 171 to 178 in 22f6bad

	if err := mgr.AddReadyzCheck("check", func(_ *http.Request) error {
	if !synchronizer.IsReady() {
	return errors.New("synchronizer not yet configured")
	}
	return nil
	}); err != nil {
	return fmt.Errorf("unable to setup readyz: %w", err)
	}

is not responding since the dataplane synchronizer is not ready yet.

Is that desired? Or do we want non-leader pods to be ready as well?

The readyz handler was changed to allow all Pods from KIC's Deployment to be marked as ready. The leader will be marked as ready when the dataplane synchronizer is ready whereas the other Pods will be ready regardless (given that it responds to readiness probe requests).

PR Readiness Checklist:

Complete these before marking the PR as ready to review:

• the CHANGELOG.md release notes have been updated to reflect any significant (and particularly user-facing) changes introduced by this PR

[codecov]

Codecov Report

Base: 72.3% // Head: 72.4% // Increases project coverage by +0.0% 🎉

Coverage data is based on head (1065014) compared to base (7cb15ee).
Patch coverage: 0.0% of modified lines in pull request are covered.

❗ Current head 1065014 differs from pull request most recent head 3b4a0df. Consider uploading reports for the commit 3b4a0df to get more accurate results

Additional details and impacted files

@@          Coverage Diff          @@
##            main   #3529   +/-   ##
=====================================
  Coverage   72.3%   72.4%           
=====================================
  Files        121     120    -1     
  Lines      14357   14321   -36     
=====================================
- Hits       10394   10380   -14     
+ Misses      3295    3279   -16     
+ Partials     668     662    -6     

Impacted Files	Coverage Δ
internal/manager/setup.go	46.4% <0.0%> (-1.2%)	⬇️
internal/metrics/prometheus.go	100.0% <0.0%> (ø)
internal/dataplane/deckerrors/conflict.go
internal/dataplane/kong_client.go	85.1% <0.0%> (+0.1%)	⬆️
internal/dataplane/parser/parser.go	91.5% <0.0%> (+0.8%)	⬆️
...ternal/controllers/gateway/httproute_controller.go	71.6% <0.0%> (+1.8%)	⬆️
...nternal/controllers/gateway/tlsroute_controller.go	73.4% <0.0%> (+2.0%)	⬆️
...nternal/controllers/gateway/udproute_controller.go	73.7% <0.0%> (+2.1%)	⬆️
internal/dataplane/sendconfig/sendconfig.go	83.3% <0.0%> (+6.0%)	⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

[pmalek]

I've tweaked the readyz handler, please see the description.

[czeslavo]

The logic seems correct to me. 👍