Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bgpv1: filter terminating backends from endpoint selection #32536

Merged

Conversation

harsimran-pabla
Copy link
Contributor

@harsimran-pabla harsimran-pabla commented May 14, 2024

Filtering out backends which are terminating when creating local endpoint state. This will result in quicker route withdrawal if local backends go into terminating state, without waiting for graceful shutdown period of the backend pod.

Fixes: #32487

bgp: service eTP=local, withdraw route when last backend on the node goes in terminating state

Filtering out backends which are terminating when creating local
endpoint state. This will result in quicker route withdrawal if local
backends go into terminating state, without waiting for graceful
shutdown period of the pods.

Signed-off-by: harsimran pabla <hpabla@isovalent.com>
@harsimran-pabla harsimran-pabla added release-note/bug This PR fixes an issue in a previous release of Cilium. area/bgp needs-backport/1.15 This PR / issue needs backporting to the v1.15 branch labels May 14, 2024
@harsimran-pabla harsimran-pabla requested a review from a team as a code owner May 14, 2024 18:39
Copy link
Contributor

@rastislavs rastislavs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks!

@rastislavs
Copy link
Contributor

/test

Copy link
Member

@YutaroHayakawa YutaroHayakawa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@bewing
Copy link
Contributor

bewing commented May 15, 2024

While this is definitely an improvement (and does solve the immediate problem!), would it make more sense to enumerate EndpointSlice.endpoints.conditions and EndpointSlice.endpoints.nodeName to utilize the same logic the service controller does when determining if traffic should be directed to a node/pod?

Edit:
A quick test of a pod with a misconfigured readinessProbe shows that it won't be considered a backend pod, so the above might be overkill.

@harsimran-pabla
Copy link
Contributor Author

Hi @bewing, I am not sure if I understand your concern.

would it make more sense to enumerate EndpointSlice.endpoints.conditions and EndpointSlice.endpoints.nodeName to utilize the same logic the service controller does when determining if traffic should be directed to a node/pod?

We are using cilium internal Endpoints representation. Do you have some concern that we should be using an alternative approach ?

@bewing
Copy link
Contributor

bewing commented May 15, 2024

My concern was that endpoint didn't have the same data available as EndpointSlice, which has conditions.ready explicitly tracking the readiness of the endpoint to receive traffic.

ready
ready indicates that this endpoint is prepared to receive traffic, according
to whatever system is managing the endpoint. A nil value indicates an
unknown state. In most cases consumers should interpret this unknown state
as ready. For compatibility reasons, ready should never be "true" for
terminating endpoints, except when the normal readiness behavior is being
explicitly overridden, for example when the associated Service has set the
publishNotReadyAddresses flag.

However, given that testing shows that a pod failing its readiness check doesn't contribute to a node's BGP advertisement, the two approaches are probably functionally identical

@harsimran-pabla harsimran-pabla removed the request for review from danehans May 15, 2024 19:25
@harsimran-pabla
Copy link
Contributor Author

harsimran-pabla commented May 15, 2024

@bewing I see what you mean. I believe terminating setting of Cilium ep backend is only set when conditions.terminating is set. So, there might be some other scenarios in which backend is not in terminating state but also not ready to receive traffic. If there is such a case, we should expand Cilium endpoint backend to expose ready field.

I think we can continue with this approach at this time, since there is clear case of excluding terminating pod from backend selection in case of eTP=local BGP advertisements. We can revisit if we need to add ready bool in backend structure of cilium ep.

Edit: If we check here, I think terminating state should cover all cases.

@harsimran-pabla harsimran-pabla added the ready-to-merge This PR has passed all tests and received consensus from code owners to merge. label May 15, 2024
@julianwiedmann julianwiedmann added this pull request to the merge queue May 16, 2024
Merged via the queue into cilium:main with commit 4aa6a7f May 16, 2024
66 checks passed
@YutaroHayakawa YutaroHayakawa mentioned this pull request May 23, 2024
15 tasks
@YutaroHayakawa YutaroHayakawa added backport-pending/1.15 The backport for Cilium 1.15.x for this PR is in progress. and removed needs-backport/1.15 This PR / issue needs backporting to the v1.15 branch labels May 23, 2024
@github-actions github-actions bot added backport-done/1.15 The backport for Cilium 1.15.x for this PR is done. and removed backport-pending/1.15 The backport for Cilium 1.15.x for this PR is in progress. labels May 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/bgp backport-done/1.15 The backport for Cilium 1.15.x for this PR is done. ready-to-merge This PR has passed all tests and received consensus from code owners to merge. release-note/bug This PR fixes an issue in a previous release of Cilium.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Delayed NLRI withdrawl for ingress-nginx endpoint removal
5 participants