kube-proxy only looks at Addresses[0] in EndpointSlice

[danwinship]

What happened?

An EndpointSlice can contain up to 1000 Endpoints, which can each contain up to 100 Addresses. Or at least, that's what the docs say, but actually, kube-proxy (or more specifically, pkg/proxy/endpointslicecache.go) only looks at the first Address in each Endpoint:

		endpointInfo := newBaseEndpointInfo(endpoint.Addresses[0], nodeName, zone, portNum, isLocal,
			endpoint.Ready, endpoint.Serving, endpoint.Terminating, endpoint.ZoneHints)

(This is compatible with how EndpointSliceController works, since it would only generate multi-Address Endpoints if a pod had multiple PodIPs of the same address family, which is not allowed. But it's not compatible with manually-generated EndpointSlices.)

What did you expect to happen?

Either all Addresses are used, or else the docs state that only Addresses[0] is used and the field is an array solely for historical reasons.

[k8s-ci-robot]

@danwinship: This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[danwinship]

/sig network
/priority important-longterm

/cc @robscott

[aojea]

All addresses in the slice must be of the same type, do you mean creating an Slice with different address types in the Addresses field?

kubernetes/staging/src/k8s.io/api/discovery/v1/types.go

Lines 35 to 42 in 376b215

	// addressType specifies the type of address carried by this EndpointSlice.
	// All addresses in this slice must be the same type. This field is
	// immutable after creation. The following address types are currently
	// supported:
	// * IPv4: Represents an IPv4 Address.
	// * IPv6: Represents an IPv6 Address.
	// * FQDN: Represents a Fully Qualified Domain Name.
	AddressType AddressType `json:"addressType" protobuf:"bytes,4,rep,name=addressType"`

kubernetes/staging/src/k8s.io/api/discovery/v1/types.go

Lines 70 to 77 in 376b215

	type Endpoint struct {
	// addresses of this endpoint. The contents of this field are interpreted
	// according to the corresponding EndpointSlice addressType field. Consumers
	// must handle different types of addresses in the context of their own
	// capabilities. This must contain at least one address but no more than
	// 100.
	// +listType=set
	Addresses []string `json:"addresses" protobuf:"bytes,1,rep,name=addresses"`

[robscott]

I'm not sure if/how kube-proxy should handle an endpoint with more than 1 address. I think the goal of kube-proxy is to give each endpoint an equivalent probability of being selected. To extend that logic here, I think we'd need to take the probability assigned to an endpoint and divide it by the number of addresses attached to that endpoint. So if an endpoint has a 50% probability of being selected, and it provides 50 addresses, each should end up with a 1% probability of being selected. Of course with the way we need to do descending probabilities, the logic may get a bit complicated.

This isn't as clear as it should be, but I don't think kube-proxy or any consumer of the EndpointSlice API needs to commit to supporting every value specified by that API. For example, many may filter by address type, Service, or some other means. I think it's also valid for a consumer to say they only support a single address per family per endpoint.

So all of that to say, I'd support either documenting that kube-proxy only selects the first address in an endpoint, or updating kube-proxy to support more than one address per endpoint. Given the number of changes currently in flight for kube-proxy, documentation seems like a safer initial approach to me.

[cyclinder]

/assign

[cyclinder]

can I have this？

editing only the first address(Address[0]) in Endpoint.Address to be used. The use of an array in this field is based on historical reasons. Isn't that right ？

[robscott]

@cyclinder thanks for volunteering to help with this one! I know @danwinship has some big kube-proxy changes/fixes in flight right now, so it would be good to be sure you don't conflict with him. Otherwise, as long as @danwinship isn't already working on this, I think you're free to have it.

[danwinship]

editing only the first address(Address[0]) in Endpoint.Address to be used. The use of an array in this field is based on historical reasons. Isn't that right ？

I don't think we agreed that the array is for historical reasons only. We aren't currently using it, but we aren't ready to say that we won't use it in the future at this point.

So, the docs should warn that kube-proxy currently ignores additional addresses, but it shouldn't imply that it's actually wrong to have additional addresses

Otherwise, as long as @danwinship isn't already working on this, I think you're free to have it.

I'm not