Proxy Terminating Endpoints

[andrewsykim]

Enhancement Description

• One-line enhancement description (can be used as a release note): Enable zero downtime deployments for Services with ExternalTrafficPolicy=Local
• Kubernetes Enhancement Proposal: (link to kubernetes/enhancements file, if none yet, link to PR): https://github.com/kubernetes/enhancements/tree/master/keps/sig-network/1669-proxy-terminating-endpoints
• Primary contact (assignee): @andrewsykim
• Responsible SIGs: Network
• Enhancement target (which target equals to which milestone):
  • Alpha release target v1.22
  • Beta release target v1.26
  • Stable release target v1.28
    • Code: promote ProxyTerminatingEndpoints to GA kubernetes#117718

Please to keep this description up to date. This will help the Enhancement Team track efficiently the evolution of the enhancement

[andrewsykim]

/sig network

[johnbelamaric]

@andrewsykim Based on the discussion it looks like you are still planning this for 1.19? If so I will make sure we track it in the release team.

[andrewsykim]

@andrewsykim Based on the discussion it looks like you are still planning this for 1.19? If so I will make sure we track it in the release team.

I know there's a lot of other efforts in-flight in SIG Network so no worries if this slips v1.19. Assuming we're happy with what's proposed, I intend to work on this for v1.19

[johnbelamaric]

We are trying to reduce content and risk in v1.19 so if you can defer to v1.20 that would be best.

[johnbelamaric]

@andrewsykim Hi Andrew, tomorrow is Enhancements Freeze for 1.19, what's the decision? 1.19 or 1.20?

[andrewsykim]

Happy to work on this for v1.19, KEP just needs approval/review at this point. If it slips to v1.20 that is okay with me as well :).

[johnbelamaric]

Enhancements freeze is upon us, deferring to 1.20.

/milestone v1.20

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[andrewsykim]

/remove-lifecycle stale

[kikisdeliveryservice]

Hi @andrewsykim

Enhancements Lead here, do you still intend to target this for GA in 1.20?

Thanks!
Kirsten

[kikisdeliveryservice]

Hi @andrewsykim

Pinging again, is this to be included in 1.20?

Thanks
Kirsten

[kikisdeliveryservice]

Hi @andrewsykim @thockin

Another reminder: Enhancements Freeze is next Tuesday October 6th. If you intend for this enhancement to be included, please let me know. This KEP's graduation criteria says it's dependent on #1672 , but #1672 currently does not have any KEP associated with it. So if this were to proceed into 1.20, you would need to update the graduation criteria.

I also note that this KEP seems to be missing the PRR.

Thanks,
Kirsten

[andrewsykim]

Sorry @kikisdeliveryservice this KEP was merged in v1.19, forgot to update the description with a link

[kikisdeliveryservice]

Hi @andrewsykim

Thanks for the update. A little confusion bc the graduation criteria says:

The graduation criteria of this KEP will largely depend on the graduation status of the EndpointSlice API. Once the terminating field is added to EndpointSlice API, this change in behavior will kick-in as soon as kube-proxy consumes EndpointSlice.

Is that correct? What do you expect to delivery in this release exactly? Can you update the KEP to provide some clarity?

Thanks!
Kirsten

[kikisdeliveryservice]

@andrewsykim

As a reminder Enhancements Freeze is October 6th, so you will need to update your KEP by that time to lay out the alpha (this release deliverable), alpha->beta, etc... graduation criteria as requested above.

Thanks!
Kirsten

[andrewsykim]

Updated alpha graduation criteria here #2064

[kikisdeliveryservice]

@andrewsykim This is great and much clearer!! 👍

[kikisdeliveryservice]

Hi @andrewsykim

Ping!

As a reminder your PR (#2064 ) needs to merge by EOD PST tomorrow October 6th to be included in the 1.20 Release. After that time you will need to request an exception.

Best,
Kirsten

[kikisdeliveryservice]

#2064 merged! Updating tracking sheet now. 😄

[kikisdeliveryservice]

Hey @andrewsykim

Sincethis Enhancement is scheduled to be in 1.20, please keep in mind the important upcoming dates:
Friday, Nov 6th: Week 8 - Docs Placeholder PR deadline
Thursday, Nov 12th: Week 9 - Code Freeze

As a reminder, please link all of your k/k PR as well as docs PR to this issue so we can track them.

Thanks!
Kirsten

[SergeyKanzhelev]

Basically, there is a contract on the pod lifecycle and other features have to depend on that contract, in this case terminating endpoints is relying on the kubelet to handle the pod lifecycle, but there are many more features that depend on that, not being terminating endpoints

terminating endpoints is the first feature I know that needs readiness probes running on a terminating pod. Before this feature running those was pointless - there were no traffic to the Pod anyway. Thus I think KEP-1672 or this KEP may be a good driver to actually add this test =).

[aojea]

Before this feature running those was pointless - there were no traffic to the Pod anyway.

I will add the tests, but not because of this feature, I do not agree on that assumption of "there were no traffic to the Pod anyway" , Services and Endpoints are the main users of the Pod status to send traffic but not the only ones, the ingress-gce controller per example uses the Pods directly and check the Pod status way before this feature.

[aojea]

lol, I remember this bug, this was present also without this feature, and I already added a test that reproduced it that Ryan incorporated in the fix
kubernetes/kubernetes#110191 (comment)

Swetha and I also talked in kubecon about it

[robert-heinzmann-logmein]

Just to understand better: ProxyTerminatingEndpoints (and thus downtime free node and pod rotation) depend on kubernetes/kubernetes#110309 to be fixed, correct ?

[aojea]

Just to understand better: ProxyTerminatingEndpoints (and thus downtime free node and pod rotation) depend on kubernetes/kubernetes#110309 to be fixed, correct ?

noep, that was fixed a long time ago, ProxyTerminatingEndpoints works fine since 1.26 and have e2e test covering that feature WITH KUBE-PROXY. Other implementations should run the e2e tests provided by kubernetes to verify they are complaint.

https://testgrid.k8s.io/google-gce#gci-gce-ingress&include-filter-by-regex=should%20not%20have%20connectivity%20disruption%20during%20rolling%20update%20with%20externalTrafficPolicy

grep -r ProxyTerminatingEndpoints test/
test/e2e/network/service.go:    ginkgo.It("should fail health check node port if there are only terminating endpoints [Feature:ProxyTerminatingEndpoints]", func(ctx context.Context) {
test/e2e/network/service.go:    ginkgo.It("should fallback to terminating endpoints when there are no ready endpoints with internalTrafficPolicy=Cluster [Feature:ProxyTerminatingEndpoints]", func(ctx context.Context) {
test/e2e/network/service.go:    ginkgo.It("should fallback to local terminating endpoints when there are no ready endpoints with internalTrafficPolicy=Local [Feature:ProxyTerminatingEndpoints]", func(ctx context.Context) {
test/e2e/network/service.go:    ginkgo.It("should fallback to terminating endpoints when there are no ready endpoints with externallTrafficPolicy=Cluster [Feature:ProxyTerminatingEndpoints]", func(ctx context.Context) {
test/e2e/network/service.go:    ginkgo.It("should fallback to local terminating endpoints when there are no ready endpoints with externalTrafficPolicy=Local [Feature:ProxyTerminatingEndpoints]", func(ctx context.Context) {

[thockin]

Next: Cleanup and close in 1.30

[ruheenaansari34]

Hello @andrewsykim 👋, Enhancements team here.

Just checking in as we approach enhancements freeze on 1:00 UTC on Friday 16th June 2023.

This enhancement is targeting stage stable for 1.28 (correct me, if otherwise)

Here’s where this enhancement currently stands:

• KEP readme using the latest template has been merged into the k/enhancements repo.
• KEP status is marked as implementable for latest-milestone: 1.28
• KEP readme has a updated detailed test plan section filled out
• KEP readme has up-to-date graduation criteria.
• KEP has a production readiness review that has been completed and merged into k/enhancements.

With all the KEP requirements in place and merged into k/enhancements, this enhancement is all good for the upcoming enhancements freeze. 🚀

The status of this enhancement is marked as tracked. Please keep the issue description up-to-date with the appropriate stages as well. Thank you!

[VibhorChinda]

Hello @andrewsykim @bowei 👋, 1.28 Docs Shadow here.

Does this enhancement work planned for 1.28 require any new docs or modification to existing docs?

If so, please follows the steps here to open a PR against dev-1.28 branch in the k/website repo. This PR can be just a placeholder at this time and must be created before Thursday 20th July 2023.

Also, take a look at Documenting for a release to get yourself familiarize with the docs requirement for the release.

Thank you!

[Rishit-dagli]

Hey @andrewsykim , could you please create a docs PR even if it is a draft PR with no content yet against dev-1.28 branch in the k/website repo. The deadline to create this draft PR is Thursday 20th July 2023.

[ruheenaansari34]

Hey again @andrewsykim 👋
Just checking in as we approach Code freeze at 01:00 UTC Friday, 19th July 2023.

I don't see any code (k/k) update PR(s) in the issue description so if there are any k/k related PR(s) that we should be tracking for this KEP please link them in the issue description above.

As always, we are here to help if any questions come up. Thanks!

[VibhorChinda]

Hey @andrewsykim , could you please create a docs PR even if it is a draft PR with no content yet against dev-1.28 branch in the k/website repo. The deadline to create this draft PR is Thursday 20th July 2023.
Thanks.

[Atharva-Shinde]

Hey @andrewsykim 👋 Enhancements Lead here,
With kubernetes/kubernetes#117718 merged as per the issue description, this enhancement is now tracked for v1.28 Code Freeze. Thanks!

[Rishit-dagli]

Hey @andrewsykim , I apologize for the ping again but we are already past the deadline for having a draft docs PR make on kubernetes/website , could you please do this as soon as possible, it could be a PR with no content yet too.

[VibhorChinda]

Hello @andrewsykim 👋 please take a look at Documenting for a release - PR Ready for Review to get your PR ready for review before Tuesday 25th July 2023. Thank you!

[Rishit-dagli]

Hey @kubernetes/sig-network-leads (sorry for another ping after https://kubernetes.slack.com/archives/C09QYUH5W/p1689977384967339),

I'm Rishit from the 1.28 Release Team. We had this KEP which is past the deadline for having a draft documentation PR for the 1.28 release. As indicated here the deadline to create a draft docs PR on kubernetes/website for the KEP was 20th July. Could you please take a look at this and populate the docs PR was 25th July?

[andrewsykim]

Opened a PR to update feature gate reference in kubernetes/website#42229. Other docs were added in previous releases.

[npolshakova]

Hey @andrewsykim 👋 1.29 Enhancements Lead here, is there any more work planned for this KEP? Also is the milestone accurate (are you targeting v1.30 instead of v1.29?)

[npolshakova]

/remove-label lead-opted-in

[aojea]

This KEP is finished, last step is to completely remove the feature tag from the code, that is why it targets 2 releases after GA