feat: use /status/ready as readiness probe of gateway for gateway discovery

[czeslavo]

What this PR does / why we need it:

Changes the Gateway's Pod readiness probe from /status to /status/ready. /status/ready returns 200 after receiving the first non-empty configuration. That allows to make Gateway pods to serve proxy traffic only after they're initially configured by KIC.

In KIC's Gateway discovery, instead of relying on EndpointSlices Endpoints' being ready, we use the discovered endpoints despite their readiness (we only discard terminating ones) and verify their readiness on our own with the use of the Admin API's /status in the ReadinessChecker. It requires running a periodic readiness reconciliation loop to check whether the endpoints that were ready should be moved to the pending list and vice versa.

Fundamental changes that were made in this PR:

• KongAdminAPIServiceReconciler watches Admin API service endpoints and pushes all but terminating ones to the AdminAPIClientsManager (via Notify method)
• AdminAPIClientsManager accepts the discovered endpoints and verifies their readiness with use of a new ReadinessChecker first:
  • if they're not ready - they're kept on a pending list
  • if they're ready - they're kept on an active list (that is used to return GatewayClients() to the upper layers)
• AdminAPIClientsManager is responsible for running a readiness reconciliation loop in which it uses ReadinessChecker to verify if:
  • the clients kept on the active list became not ready; if yes, move them to the pending list
  • the clients kept on the pending list became ready; if yes, move them to the active list

Which issue this PR fixes:

Closes #3499.

Special notes for your reviewer:

PR Readiness Checklist:

Complete these before marking the PR as ready to review:

• the CHANGELOG.md release notes have been updated to reflect any significant (and particularly user-facing) changes introduced by this PR

[team-k8s-bot]

E2E (targeted) tests with KIND-based clusters were started at https://github.com/Kong/kubernetes-ingress-controller/actions/runs/5591322167

[codecov]

Codecov Report

Patch coverage: 93.1% and project coverage change: +0.2 🎉

Comparison is base (00cf5d1) 65.7% compared to head (92ffd58) 65.9%.

Additional details and impacted files

@@           Coverage Diff           @@
##            main   #4368     +/-   ##
=======================================
+ Coverage   65.7%   65.9%   +0.2%     
=======================================
  Files        157     158      +1     
  Lines      18159   18274    +115     
=======================================
+ Hits       11939   12052    +113     
- Misses      5481    5484      +3     
+ Partials     739     738      -1     

Impacted Files	Coverage Δ
internal/adminapi/client.go	47.0% <0.0%> (-1.5%)	⬇️
internal/manager/run.go	53.2% <50.0%> (+0.1%)	⬆️
internal/clients/readiness.go	89.8% <89.8%> (ø)
internal/adminapi/endpoints.go	84.2% <100.0%> (ø)
internal/clients/manager.go	97.7% <100.0%> (+2.9%)	⬆️
...ntrollers/configuration/kongadminapi_controller.go	62.7% <100.0%> (+0.4%)	⬆️

... and 2 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

[team-k8s-bot]

E2E (targeted) tests with KIND-based clusters were started at https://github.com/Kong/kubernetes-ingress-controller/actions/runs/5591805351

[rainest]

Do we expect this to have any affect on interaction with older versions? I don't see any testing with 3.0 (proxy pods come up fine and get config, no controller errors for the most part) but have a minor bugbear that there should be some odd behavior on versions where the endpoint doesn't exist.

There were some errors on scale up about failing to post config, but posts were successful after a few seconds, so it seems like we maybe try and push config before the instance is fully ready, but that's benign (and happens with 3.3 too, so it's not an older version behavior difference).

[czeslavo]

Do we expect this to have any affect on interaction with older versions? I don't see any testing with 3.0 (proxy pods come up fine and get config, no controller errors for the most part) but have a minor bugbear that there should be some odd behavior on versions where the endpoint doesn't exist.

I don't think so. The behavior of the controller will be compatible with any version of Gateway we supported so far. Despite the readiness endpoint used for Gateway's readinessProbe, it will detect all the endpoints behind an Admin API Service (with an exception for terminating ones) and will ensure their readiness by calling Admin API's /status endpoint. For Gateway 3.0 with /status configured as readinessProbe, it will work the same way which should be fine (we may say the effort for readiness checking will be doubled in a way, but it won't make it break).

There were some errors on scale up about failing to post config, but posts were successful after a few seconds, so it seems like we maybe try and push config before the instance is fully ready, but that's benign (and happens with 3.3 too, so it's not an older version behavior difference).

That's interesting - I was able to reproduce it on scale up as you mentioned. The initial readiness check passes (we succeed to create a client and it returns no error on /status) but later in the config push loop sometimes it happens that it refuses the connection on the first and/or second config push. Eventually, it always gets to the right state.

Perhaps this could be solved by making the checks made by KIC to keep track of success calls and consider clients ready only after they reach some success threshold (and similarly - we could have such a threshold for failures). I'd say that if we'd like to implement this, we can do that in a separate PR, WDYT?

[rainest]

Perhaps this could be solved by making the checks made by KIC to keep track of success calls and consider clients ready only after they reach some success threshold (and similarly - we could have such a threshold for failures). I'd say that if we'd like to implement this, we can do that in a separate PR, WDYT?

Yeah, it's not a concern for what this PR is trying to do, nor was it introduced by the changes here. I just noticed it when checking for these changes' backwards compatibility. Backwards compatibility for this is indeed fine.