fix the race condition in pod cache and endpoint resolver

[oliviassss]

Issue

#3299

Description

The pod informer and endpoints informer works independently. In some edge case, the pod is created and exists in endpoints, but when we find the podIP in endpoints, the pod is not propagated in our informer cache yet. But in the code above we simply ignored it and never retries. (and since this pod has a readinessGate, it never became ready in endpoints so no further reconcile is triggered).

For now, We can fixe this by setting containsPotentialReadyEndpoints=true if the pod in endpoints is not found in pod cache yet (so there will be a retry later).

In the future, we'll add a pod event watcher to address this issue nicely, i.e. this TODO

Checklist

• Added tests that cover your change (if possible)
• Added/modified documentation as required (such as the README.md, or the docs directory)
• Manually tested
• Made sure the title of the PR is a good description that can go into the release notes

BONUS POINTS checklist: complete for good vibes and maybe prizes?! 🤯

• Backfilled missing tests for code in same general area 🎉
• Refactored something and made the world a better place 🌟

[codecov-commenter]

Codecov Report

Patch coverage: 100.00% and no project coverage change.

Comparison is base (d1b8fbb) 54.76% compared to head (e071573) 54.76%.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #3305   +/-   ##
=======================================
  Coverage   54.76%   54.76%           
=======================================
  Files         148      148           
  Lines        8601     8602    +1     
=======================================
+ Hits         4710     4711    +1     
  Misses       3559     3559           
  Partials      332      332           

Files Changed	Coverage Δ
pkg/backend/endpoint_resolver.go	86.50% <100.00%> (+0.08%)	⬆️

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[M00nF1sh]

/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: M00nF1sh, oliviassss

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [M00nF1sh]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[oliviassss]

/retest