Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix endpoint selection for externalTrafficPolicy=local #4170

Merged
merged 1 commit into from
Apr 15, 2024

Conversation

ricky-rav
Copy link
Contributor

@ricky-rav ricky-rav commented Feb 20, 2024

In #4072 the following edge case was not handled correctly in ovnkube-controller code.

Fix the case for all endpoints terminating on a node when traffic policy is local:

"When the traffic policy is "Local" and all endpoints are terminating within a single node, then traffic should be routed to any terminating endpoint that is ready on that node."

https://github.com/kubernetes/enhancements/blob/master/keps/sig-network/1669-proxy-terminating-endpoints/README.md#example-all-endpoints-terminating-on-a-node-when-traffic-policy-is-local

The endpoint selection logic in the services controller is now entirely implemented in getEndpointsForService, which computes for a given service and each service port all its cluster-wide endpoints and per-node local endpoints. We first apply a cluster-wide vs local endpoint selection and only then we apply readiness-based filtering with getEligibleEndpointAddresses.

Added unit tests for the new logic.

@coveralls
Copy link

coveralls commented Feb 20, 2024

Coverage Status

coverage: 52.411% (+0.07%) from 52.346%
when pulling cba7633 on ricky-rav:kep-1669-bis_upstream
into ebdc80f on ovn-org:master.

@ricky-rav
Copy link
Contributor Author

ricky-rav commented Feb 20, 2024

I'll have to add a unit test for this particular case to Test_buildPerNodeLBs. I'll do that tomorrow morning.
Done!

@ricky-rav ricky-rav force-pushed the kep-1669-bis_upstream branch 2 times, most recently from 85b0db1 to 6dbc864 Compare February 22, 2024 19:14
@ricky-rav
Copy link
Contributor Author

I've added some unit tests to go-controller/pkg/ovn/controller/services/lb_config_test.go that would have caught the issue I'm fixing here (ETP=local and terminating endpoints). @tssurya since you've done the work on ETP=local, would you be able to review those tests? Thanks!

go-controller/pkg/util/kube_util.go Outdated Show resolved Hide resolved
go-controller/pkg/util/kube.go Outdated Show resolved Hide resolved
go-controller/pkg/util/kube.go Outdated Show resolved Hide resolved
go-controller/pkg/util/kube.go Outdated Show resolved Hide resolved
go-controller/pkg/util/kube.go Outdated Show resolved Hide resolved
go-controller/pkg/util/kube.go Outdated Show resolved Hide resolved
go-controller/pkg/ovn/controller/services/utils.go Outdated Show resolved Hide resolved
go-controller/pkg/util/kube.go Outdated Show resolved Hide resolved
go-controller/pkg/ovn/controller/services/lb_config.go Outdated Show resolved Hide resolved
@ricky-rav ricky-rav force-pushed the kep-1669-bis_upstream branch 6 times, most recently from de5f1a5 to 0c4ab97 Compare March 4, 2024 11:34
go-controller/pkg/util/kube.go Outdated Show resolved Hide resolved
go-controller/pkg/util/kube.go Outdated Show resolved Hide resolved
go-controller/pkg/util/kube.go Outdated Show resolved Hide resolved
go-controller/pkg/util/kube.go Outdated Show resolved Hide resolved
go-controller/pkg/testing/kube.go Outdated Show resolved Hide resolved
go-controller/pkg/ovn/controller/services/utils.go Outdated Show resolved Hide resolved
@ricky-rav ricky-rav force-pushed the kep-1669-bis_upstream branch 3 times, most recently from dde092e to 96dc373 Compare March 8, 2024 16:50
@ricky-rav
Copy link
Contributor Author

/retest

@ovn-robot
Copy link
Collaborator

Oops, something went wrong:

Must have admin rights to Repository.

@ricky-rav ricky-rav force-pushed the kep-1669-bis_upstream branch 2 times, most recently from d395fdf to 2793a33 Compare March 11, 2024 08:08
@tssurya tssurya added kind/bug All issues that are bugs and PRs opened to fix bugs area/gateway Issues related to node gateway code labels Mar 12, 2024
@ricky-rav ricky-rav force-pushed the kep-1669-bis_upstream branch 2 times, most recently from e04df18 to dc52cb0 Compare March 28, 2024 19:28
@ricky-rav
Copy link
Contributor Author

ricky-rav commented Mar 28, 2024

I've pushed a new getEndpointsForService as suggested in slack by Jaime. It reduces by more than half the 200 lines of code of my original implementation. Thanks, Jaime!
To reduce memory consumption, the per-node endpoints map is limited to the nodes in the local OVN zone.

Copy link
Contributor

@jcaamano jcaamano left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was a bit afraid of taking this route. Can it wait?

Yup

Copy link
Contributor

@jcaamano jcaamano left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good idea! I made getNodeSwitchTargetIPs handle v4 and v6 at the same time. Does it look ok?
Also, do you prefer a free method because we're not really modifying c *lbConfig?

I preferred the free method because it didn't make sense to pass as arguments data that is already available through lbConfig. I thought about keeping the receiver and removing the extra arguments but something that I don't remember know didn't make sense as well. So I suggested the free method.

Copy link
Contributor

@jcaamano jcaamano left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jcaamano as discussed, I implemented the new logic in one function, getEndpointsForService, that gets called once for a service and whose output is then parsed by the rest of the services controller to retrieve cluster-wide and per-node endpoint addresses. PTAL :)

Looks good, added some comments to polish. CI looks bad though, hopefuly something else.

@ricky-rav
Copy link
Contributor Author

ricky-rav commented Apr 2, 2024

CI looks bad though, hopefuly something else.

I broke something... let me try to fix it :)

@ricky-rav ricky-rav force-pushed the kep-1669-bis_upstream branch 4 times, most recently from b2891e4 to bfbfaaf Compare April 4, 2024 07:32
Copy link
Contributor

@jcaamano jcaamano left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Everything looks good @ricky-rav.

The smal nit, up to you to act on it, and just that failing lane.

Also, woudl you be interested in running this dowsntream to check on the disruption results before merging here?

}
klog.V(5).Infof("Cluster endpoints for %s/%s are: %v", service.Namespace, service.Name, portToLBEndpoints)

if requiresLocalEndpoints {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: this is a superfluous check, no problem on iterating on an empty map

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

right, I'll move the check to the log line further below... no need to print the local endpoints if we don't need them in the first place.

@ricky-rav
Copy link
Contributor Author

Also, would you be interested in running this dowsntream to check on the disruption results before merging here?

Yes! Let's see how it goes here: openshift/ovn-kubernetes#2109

}

if requiresLocalEndpoints {
klog.V(5).Infof("Local endpoints for %s/%s are: %v", service.Namespace, service.Name, portToNodeToLBEndpoints)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: double spacing in this log and the one above

@ricky-rav
Copy link
Contributor Author

ricky-rav commented Apr 9, 2024

Our downstream CI fails on the e2e-metal-ipi-ovn-ipv6 job on tests related to services and endpoints:

  • [sig-network] Networking Granular Checks: Services should function for endpoint-Service: http [Suite:openshift/conformance/parallel] [Suite:k8s]
  • [sig-network] Networking Granular Checks: Services should function for endpoint-Service: udp [Suite:openshift/conformance/parallel] [Suite:k8s]
  • [sig-network] Services should be able to connect to terminating and unready endpoints if PublishNotReadyAddresses is true [Suite:openshift/conformance/parallel] [Suite:k8s]
  • [sig-network] Networking Granular Checks: Services should function for multiple endpoint-Services with same selector [Suite:openshift/conformance/parallel] [Suite:k8s]
  • [sig-network] Services should be able to update service type to NodePort listening on same port number but different protocols [Suite:openshift/conformance/parallel] [Suite:k8s]
  • [sig-network] Services should fallback to terminating endpoints when there are no ready endpoints with externallTrafficPolicy=Cluster [Suite:openshift/conformance/parallel] [Suite:k8s]

https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/openshift_ovn-kubernetes/2109/pull-ci-openshift-ovn-kubernetes-master-e2e-metal-ipi-ovn-ipv6/1777348680731332608

These tests are either not run in our upstream CI for IPv6 jobs or never run at all :(

I'm trying to reproduce these failures on KIND and figure out how the tests get skipped...

@ricky-rav ricky-rav force-pushed the kep-1669-bis_upstream branch 2 times, most recently from cba7633 to f423e57 Compare April 9, 2024 16:52
@ricky-rav
Copy link
Contributor Author

ricky-rav commented Apr 12, 2024

Downstream CI is back, but e2e-metal-ipi-ovn-ipv6 is still failing:
https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/openshift_ovn-kubernetes/2109/pull-ci-openshift-ovn-kubernetes-master-e2e-metal-ipi-ovn-ipv6/1778683430230298624

I found a bug in makeNodeSwitchTargetIPs: openshift/ovn-kubernetes@a57c6b6
Trying out the fix now and added unit tests for it.
openshift/ovn-kubernetes#2109

@ricky-rav
Copy link
Contributor Author

Fix the case for "all endpoints terminating on a node when traffic policy is local":
"When the traffic policy is "Local" and all endpoints are terminating within a single node, then traffic should be routed to any terminating endpoint that is ready on that node."
https://github.com/kubernetes/enhancements/blob/master/keps/sig-network/1669-proxy-terminating-endpoints/README.md#example-all-endpoints-terminating-on-a-node-when-traffic-policy-is-local

The endpoint selection logic in the services controller is now entirely implemented in getEndpointsForService, which computes for a given service and each service port all its cluster-wide endpoints and per-node local endpoints. We first apply a cluster-wide vs local endpoint selection and only then we apply readiness-based filtering with getEligibleEndpointAddresses.

Added unit tests for the new logic.

Signed-off-by: Riccardo Ravaioli <rravaiol@redhat.com>
@ricky-rav
Copy link
Contributor Author

Failure in CI is on [FAIL] Services of type NodePort [It] should listen on each host addresses, which is a known issue #4275

@jcaamano jcaamano merged commit a3c8570 into ovn-org:master Apr 15, 2024
37 of 38 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/gateway Issues related to node gateway code core-kubernetes Issues related to core kubernetes like pods, services, endpoints, endpointslices feature/services&endpoints All issues related to the Servces/Endpoints API kind/bug All issues that are bugs and PRs opened to fix bugs
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

None yet

5 participants