EndpointSlice Controllers should use multiple slices for Services with many ports

[robscott]

What happened:

The EndpointSlice controllers are unaware of the limit on the number of EndpointSlice ports. When a Service has more than 100 ports, the EndpointSlice controllers try to create/update EndpointSlices as usual but fail.

What you expected to happen:

The EndpointSlice controller should probably have some support for subsetting EndpointSlices into groups of 100 ports in these instances. Alternatively we could remove the EndpointSlice validation for this but that would result in yet another unbounded list in k8s APIs.

How to reproduce it (as minimally and precisely as possible):

Create a Service with >100 ports.

/cc @swetharepakula @thockin

[robscott]

/sig network
/triage accepted

[aojea]

interesting, is this limit present on the normal Endpoints controller too?

[robscott]

No, this is unique to the EndpointSlice API. We were trying to ensure there were no unbounded lists in the EndpointSlice API, but Service does have an unbounded list here. I didn't properly handle that in the controller. The simplest solution would be to loosen the EndpointSlice API validation but I don't think that's something we want to do. I'm guessing that having >100 ports on a Service is relatively uncommon, but it is still technically something Services support.

I wonder if we could say that EndpointSlices would only track up to 100 ports for a Service and then drop anything beyond that? That number may be too low but it does seem like we need to have some kind of upper limit here unlike the unbounded list that is currently possible on Service.

/cc @thockin

[freehan]

I would rather just drop the validation. The port number is actually bounded to 0-65535

[robscott]

@thockin How do you feel about increasing the limit of ports to 65536 in EndpointSlice validation?

[thockin]

The port number is bounded to 64k * 3 of possible protocols (3 today).

In theory, you could craft a YAML that just fails - 192Ki endpoints, assume 32 bytes each * 100 Endpoints is > 4x the max size of a resource, not even counting the endpoints themselves

Now, I am SURE that nobody is really trying to expose 192 thousand ports. But how many is enough? If we limit it to 1k, there will be someone doing VOIP who needs 4k.

If we set a limit of 16K ports per slice, is that enough? 32K puts us at the 1MiB mark. 64K overflows a resource.

If we get rid of the limit, what we end up with is non-deterministic failures. Can we pick a (seemingly) ridiculous number like 16K ports and then fix this issue (that the EPslice controller REALLY needs to add a dimension to re-slice on # of ports)?

Do we think 16K is going to be enough? Do we have data to support or refute that? Will it be enough to get us through to a controller which slices ports?

If we put no limit, we will not be able to easily go back on that, but it is compatible with Endpoints (sadly).

[aojea]

Do we think 16K is going to be enough? Do we have data to support or refute that? Will it be enough to get us through to a controller which slices ports?

16k covers all the current iana registered ports

curl -o- https://www.iana.org/assignments/service-names-port-numbers/service-names-port-numbers.txt | grep -E "tcp|udp|sctp" | wc

  12843   96609 2171379