WIP: Support EndpointSlice in sdn and test handling terminating endpoints #271

smarterclayton · 2021-03-04T18:36:37Z

86ceaca (Clayton Coleman, 5 minutes ago)
DO NOT MERGE: Force EndpointSliceProxying on

bf1fd9b (Clayton Coleman, 12 minutes ago)
DO NOT MERGE: UPSTREAM: 97238: Handle terminating endpoints

This is a test cherry-pick based on the current vendor state containing
upstream 97238, which allows the proxier to handle terminating endpoints.
This is not sufficient by itself because we need to test endpoint slices,
but ensures the right code is in place.

openshift-ci-robot · 2021-03-04T18:36:49Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: smarterclayton
To complete the pull request process, please assign rcarrillocruz after the PR has been reviewed.
You can assign the PR to them by writing /assign @rcarrillocruz in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

smarterclayton · 2021-03-04T20:46:43Z

@Miciah this i where i'm testing it

smarterclayton · 2021-03-05T05:32:21Z

E0304 23:42:19.517393    2983 reflector.go:138] k8s.io/client-go@v0.18.6/tools/cache/reflector.go:167: Failed to watch *v1beta1.EndpointSlice: failed to list *v1beta1.EndpointSlice: endpointslices.discovery.k8s.io is forbidden: User "system:serviceaccount:openshift-sdn:sdn" cannot list resource "endpointslices" in API group "discovery.k8s.io" at the cluster scope

openshift/cluster-network-operator#1003

smarterclayton · 2021-03-05T06:38:00Z

Panic from https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/release-openshift-origin-installer-launch-gcp/1367709628095270912

E0305 06:11:58.595222   21469 runtime.go:78] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
goroutine 311 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic(0x1c0e9c0, 0x2e01a50)
	k8s.io/apimachinery@v0.18.6/pkg/util/runtime/runtime.go:74 +0x95
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
	k8s.io/apimachinery@v0.18.6/pkg/util/runtime/runtime.go:48 +0x89
panic(0x1c0e9c0, 0x2e01a50)
	runtime/panic.go:969 +0x1b9
k8s.io/kubernetes/pkg/proxy.(*EndpointSliceCache).updatePending(0x0, 0xc0006e1000, 0x0, 0xc000608200)
	k8s.io/kubernetes@v1.20.0-rc.0/pkg/proxy/endpointslicecache.go:167 +0x22d
k8s.io/kubernetes/pkg/proxy.(*EndpointChangeTracker).EndpointSliceUpdate(0xc000437f80, 0xc0006e1000, 0x0, 0xc000520600)
	k8s.io/kubernetes@v1.20.0-rc.0/pkg/proxy/endpoints.go:260 +0x29f
k8s.io/kubernetes/pkg/proxy/iptables.(*Proxier).OnEndpointSliceAdd(0xc000873c80, 0xc0006e1000)
	k8s.io/kubernetes@v1.20.0-rc.0/pkg/proxy/iptables/proxier.go:628 +0x3d
github.com/openshift/sdn/pkg/network/proxyimpl/hybrid.(*HybridProxier).OnEndpointSliceAdd(0xc0006073b0, 0xc0006e1000)
	github.com/openshift/sdn/pkg/network/proxyimpl/hybrid/proxy.go:268 +0x522
github.com/openshift/sdn/pkg/network/proxy.(*OsdnProxy).OnEndpointSliceAdd(0xc0007d0360, 0xc0006e1000)
	github.com/openshift/sdn/pkg/network/proxy/proxy.go:332 +0x142
k8s.io/kubernetes/pkg/proxy/config.(*EndpointSliceConfig).handleAddEndpointSlice(0xc00050f120, 0x1e79cc0, 0xc0006e1000)
	k8s.io/kubernetes@v1.20.0-rc.0/pkg/proxy/config/config.go:244 +0x13b
k8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnAdd(...)
	k8s.io/client-go@v0.18.6/tools/cache/controller.go:231

EDIT: Feature gate enablement was in wrong place

smarterclayton · 2021-03-06T01:09:23Z

https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/release-openshift-origin-installer-launch-gcp/1367988272344600576

clean install it looks like

smarterclayton · 2021-03-06T01:46:41Z

well, test went green except for an unidling test (i may have broken it just because of the use of the slice for annotations instead of endpoints). will kick some upgrade jobs and see how it plays out as well as a network stress test

This is a test cherry-pick based on the current vendor state containing upstream 97238, which allows the proxier to handle terminating endpoints. This is not sufficient by itself because we need to test endpoint slices, but ensures the right code is in place.

smarterclayton · 2021-03-06T17:31:17Z

Ok, got one run of e2e two runs of upgrade with endpoint slices on, but the termination gate off, and behavior was similar to endpoints except for the idling bug. Now testing with termination gate on in https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/release-openshift-origin-installer-launch-aws/1368251013236002816 as an upgrade and https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/release-openshift-origin-installer-launch-aws/1368251449502339072 as an e2e

openshift-ci · 2021-03-06T21:27:52Z

@smarterclayton: The following tests failed, say /retest to rerun all failed tests:

Test name	Commit	Details	Rerun command
ci/prow/verify-deps	`c686cc0`	link	`/test verify-deps`
ci/prow/e2e-gcp	`c686cc0`	link	`/test e2e-gcp`
ci/prow/e2e-aws	`c686cc0`	link	`/test e2e-aws`
ci/prow/e2e-aws-upgrade	`c686cc0`	link	`/test e2e-aws-upgrade`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

smarterclayton · 2021-03-06T21:49:18Z

Passes on those (for the factors that matter) without EndpointSliceTerminating set on the server side (so we're safe to roll this out first to Kube-proxy).

aojea · 2021-03-09T21:54:53Z

pkg/network/proxyimpl/hybrid/proxy.go

 	// we track all endpoints in the unidling endpoints handler so that we can succesfully
 	// detect when a service become unidling
 	klog.V(6).Infof("hybrid proxy: (always) add ep %s/%s in unidling proxy", endpoints.Namespace, endpoints.Name)
-	p.unidlingProxy.OnEndpointsAdd(endpoints)
+	p.unidlingProxy.OnEndpointSliceAdd(endpoints)


I'm not familiar at all with the sdn/proxy implementation, maybe this information is redundant, but can be multiple slices for the same service, and each slice can have duplicate endpoints, kube-proxy uses a cache
https://github.com/kubernetes/kubernetes/blob/2bcbc527a760106ec89647fcf6852f37c804f4ed/pkg/proxy/endpointslicecache.go#L43-L49

the limit of endpoints per slice is 100, so if you have more than 100 endpoints, let's say 110 for service X you'll receive two slices Y1 and Y2, maybe with 100 and 10 endpoints each

We really should have an e2e test in upstream that creates a service with > 100 endpoints then to exercise this. Is there one you know of I can crib?

Actually, for idling, wondering whether we even need to support > three or four endpoints. The only time the user space proxy should be in play is on an idle service which has no endpoints.

her we even need to support > three or four endpoints.

as I said, I'm not familiar with this code, just raising some points that I think may be taking into consideration, if that is the case, it seems we should't worry about this

We really should have an e2e test in upstream that creates a service with > 100 endpoints

this is well tested upstream

The unidling proxy can just ignore the endpointslices and work with the endpoints like it always did. Endpoints objects always contain the full set of endpoints, even in cases where the EndpointSlice controller would start splitting things up; it just means that code working with the Endpoints objects doesn't get the efficiency wins that code working with the EndpointSlice objects would get.

danwinship

Idling is broken because this flips us to only using EndpointSlice, but the userspace proxy (which is used by idling) doesn't support EndpointSlice.

danwinship · 2021-03-11T15:37:19Z

vendor/k8s.io/kubernetes/cmd/kube-proxy/app/server.go

@@ -245,6 +245,11 @@ func (o *Options) Complete() error {
 		return err
 	}

+	// DO NOT MERGE: hack endpoint slice on


You need to revert 12b21f8 from #227.
(That will result in EndpointSlice and EndpointSliceProxying being enabled since they're enabled by default.) For EndpointSliceTerminatingCondition we should eventually handle that like other feature gates, but CNO doesn't watch the FeatureGate resource yet (https://issues.redhat.com/browse/SDN-1325).

oh, wait, no, Aniket already reverted that a while back; we'll need to revert the relevant part of openshift/cluster-network-operator#905

danwinship · 2021-03-11T15:53:59Z

pkg/network/proxyimpl/hybrid/proxy.go

-	proxyconfig.NoopEndpointSliceHandler
+	// TODO implement https://github.com/kubernetes/enhancements/pull/640
+	proxyconfig.NoopNodeHandler
+	NoopEndpointsHandler


The HybridProxier can't no-op Endpoints handling; it has to pass EndpointSlice events down to the iptables proxier and Endpoints events to the userspace proxier. And since OsdnProxy acts as a filter on top of HybridProxier, it needs to also pass both sets of events down to the proxier it's wrapping.

I updated userspace proxier to use EndpointSlice, I thought we were already going to have to switch to use Service instead of Endpoints for idling.

Oops, I see what you mean. Why wasn't userspace proxier updated? Just no one signed up for it?

Why wasn't userspace proxier updated? Just no one signed up for it?

Upstream doesn't care about the userspace proxier any more (Tim would probably have already deleted it if OCP wasn't using it for unidling) and Red Hat had thought we weren't going to have to use EndpointSlice in openshift-sdn, so we didn't care about updating it either.

At any rate, I think we don't actually need to update userspace to use EndpointSlice; we just need to make HybridProxier and OsdnProxy pass both endpoint events and endpointslice events down to their wrapper proxiers, and then eventually the iptables proxy will act on the endpointslice events and the userspace proxy will act on the endpoint events.

danwinship · 2021-05-03T15:13:06Z

FYI #296 is a more complete EndpointSlice PR

openshift-ci · 2021-05-03T15:13:13Z

@smarterclayton: PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-bot · 2021-08-01T20:33:43Z

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

danwinship · 2021-08-05T13:12:20Z

/close
sdn now uses EndpointSlices, and is rebased to 1.22-rc.0, which has 97238

openshift-ci · 2021-08-05T13:12:28Z

@danwinship: Closed this PR.

In response to this:

/close
sdn now uses EndpointSlices, and is rebased to 1.22-rc.0, which has 97238

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Mar 4, 2021

openshift-ci-robot requested review from alexanderConstantinescu and rcarrillocruz March 4, 2021 18:36

smarterclayton force-pushed the proxy_2 branch from 81fa8ae to 60146b6 Compare March 4, 2021 20:38

smarterclayton force-pushed the proxy_2 branch from 60146b6 to fe49bfd Compare March 4, 2021 23:22

smarterclayton mentioned this pull request Mar 5, 2021

sdn: Add list/watch RBAC for endpointslices to openshift-sdn openshift/cluster-network-operator#1003

Merged

smarterclayton force-pushed the proxy_2 branch 2 times, most recently from 16d1b86 to c918927 Compare March 5, 2021 23:54

smarterclayton added 3 commits March 6, 2021 12:23

DO NOT MERGE: Force EndpointSliceProxying on

a4e1845

WIP: Support EndpointSlice in sdn

c686cc0

smarterclayton force-pushed the proxy_2 branch from c918927 to c686cc0 Compare March 6, 2021 17:23

aojea reviewed Mar 9, 2021

View reviewed changes

danwinship suggested changes Mar 11, 2021

View reviewed changes

This was referenced Mar 31, 2021

WIP: UPSTREAM 92968: Add Terminating Condition to EndpointSlice openshift/kubernetes#431

Closed

WIP: UPSTREAM 96371: kube-proxy: fall back to node-local terminating endpoints for external traffic #223

Closed

Miciah mentioned this pull request Apr 11, 2021

WIP: Enable "EndpointSliceTerminatingCondition" feature openshift/api#773

Closed

openshift-ci bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 3, 2021

openshift-ci bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Aug 1, 2021

openshift-ci bot closed this Aug 5, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: Support EndpointSlice in sdn and test handling terminating endpoints #271

WIP: Support EndpointSlice in sdn and test handling terminating endpoints #271

smarterclayton commented Mar 4, 2021

openshift-ci-robot commented Mar 4, 2021

smarterclayton commented Mar 4, 2021

smarterclayton commented Mar 5, 2021

smarterclayton commented Mar 5, 2021 •

edited

smarterclayton commented Mar 6, 2021

smarterclayton commented Mar 6, 2021

smarterclayton commented Mar 6, 2021

openshift-ci bot commented Mar 6, 2021

smarterclayton commented Mar 6, 2021

aojea Mar 9, 2021

aojea Mar 9, 2021

smarterclayton Mar 18, 2021

smarterclayton Mar 18, 2021

aojea Mar 18, 2021

danwinship Apr 7, 2021

danwinship left a comment

danwinship Mar 11, 2021

danwinship Mar 11, 2021

danwinship Mar 11, 2021

smarterclayton Mar 18, 2021

smarterclayton Mar 18, 2021

danwinship Apr 7, 2021

danwinship commented May 3, 2021

openshift-ci bot commented May 3, 2021

openshift-bot commented Aug 1, 2021

danwinship commented Aug 5, 2021

openshift-ci bot commented Aug 5, 2021

WIP: Support EndpointSlice in sdn and test handling terminating endpoints #271

WIP: Support EndpointSlice in sdn and test handling terminating endpoints #271

Conversation

smarterclayton commented Mar 4, 2021

openshift-ci-robot commented Mar 4, 2021

smarterclayton commented Mar 4, 2021

smarterclayton commented Mar 5, 2021

smarterclayton commented Mar 5, 2021 • edited

smarterclayton commented Mar 6, 2021

smarterclayton commented Mar 6, 2021

smarterclayton commented Mar 6, 2021

openshift-ci bot commented Mar 6, 2021

smarterclayton commented Mar 6, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

danwinship left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

danwinship commented May 3, 2021

openshift-ci bot commented May 3, 2021

openshift-bot commented Aug 1, 2021

danwinship commented Aug 5, 2021

openshift-ci bot commented Aug 5, 2021

smarterclayton commented Mar 5, 2021 •

edited