[loadbalancingexporter] `Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice`

[skandragon]

Component(s)

exporter/loadbalancing

What happened?

Description

Kubernetes 1.33+ has marked the corev1.Endpoints usage as deprecated, and the warning above is spamming us every 4 seconds in a collector running on k8s v1.33

Steps to Reproduce

Run the load balancer with k8s resolution on k8s 1.33

Expected Result

No warnings :)

Actual Result

Warnings.

Collector version

• gomod: github.com/open-telemetry/opentelemetry-collector-contrib/exporter/loadbalancingexporter v0.128.0

Environment information

Environment

Kubernets 1.33 in AWS EKS, although a locally installed 1.33 on bare metal also has this warning.

OpenTelemetry Collector configuration

loadbalancing/metrics:
    protocol:
      otlp:
        timeout: 10s
        tls:
          insecure: true
    resolver:
      k8s:
        ports:
          - 14317
        service: ${env:HEADLESS_SERVICE_NAME}.${env:POD_NAMESPACE}
    routing_key: streamID

Log output

Additional context

No response

[github-actions]

Pinging code owners:

• exporter/loadbalancing: @rlankfo

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[skandragon]

After looking at this a bit, I think the path forward is to have the config accept an "address family" that is either IPv4 (default) or IPv6, and then find that endpoint slice in the list of slices returned. We also need to query based on a label on the endpoint slice, as it is no longer directly associated with the service but its name is dynamic.

This feels like it would be the minimal change. I think the "use name" method is still workable, as there is a FQDN family available, but none of my clusters is anything setting that. It could be added manually or by some non-standard controller, but I suspect that usage is minimal enough to ignore for the first round of changes.

I think the proper way to handle the port is to also look at the EndpointSlice and see if the port we want is in the list or not. If it is, we can use that set of addresses, and if not we should look for another one that might have what we want. This generally should not happen, as the endpoint slice should have all the ports in the service, but it's probably good as a safety check.

I am happy to make a go at this. EndpointSlice has been around for several years now, so while it will break pre-1.21 Kubernetes, that's been around since April 08, 2021, so I think we can ignore that issue.

[ChristianCiach]

and then find that endpoint slice in the list of slices returned.

You probably mean the right thing, but I would like to clarify: As far as I know there can me multiple EndpointSlices for the same service and the same "IP version", which is the whole point of EndpointSlices, as compared to legacy Endpoints. So we probably should merge all .endpoints.addresses of all matching EndpointSlices for a service.

[rlankfo]

Thanks @skandragon for reporting this! I've opened a draft PR to start this transition; please take a look at the current approach and let me know if there are any concerns.

After looking at this a bit, I think the path forward is to have the config accept an "address family" that is either IPv4 (default) or IPv6, and then find that endpoint slice in the list of slices returned.

I agree here and IPv4 should be the default. Given that the exporter only supports IPv4 today, my initial thought is to transition to EndpointSlices for IPv4 address types only for now, and follow up with support for IPv6 and FQDN in a later PR.

We also need to query based on a label on the endpoint slice, as it is no longer directly associated with the service but its name is dynamic.

This is correct. I added a label selector based on the service name to lookup the associated endpoint slices, for example:

➜  ~ kubectl get endpointslices -n observability -l kubernetes.io/service-name=backends-collector
NAME                       ADDRESSTYPE   PORTS   ENDPOINTS                                         AGE
backends-collector-4zbhp   IPv4          4317    10.244.0.66,10.244.0.64,10.244.0.68 + 2 more...   3h29m

This feels like it would be the minimal change. I think the "use name" method is still workable, as there is a FQDN family available, but none of my clusters is anything setting that. It could be added manually or by some non-standard controller, but I suspect that usage is minimal enough to ignore for the first round of changes.

Agreed on this point as well. I had a hard time finding any EndpointSlices with FQDN AddressType. Just to clarify, by "use name" method are you referring to the return_hostnames configuration option on this resolver? I noticed in most cases neither my Endpoints nor EndpointSlices typically have the hostname set. This results in an error that could probably be handled more gracefully.

I think the proper way to handle the port is to also look at the EndpointSlice and see if the port we want is in the list or not. If it is, we can use that set of addresses, and if not we should look for another one that might have what we want. This generally should not happen, as the endpoint slice should have all the ports in the service, but it's probably good as a safety check.

Essentially this is what I did in the draft PR, and as @ChristianCiach pointed out, there can be multiple EndpointSlices for the same service, port, address type combo.

I am happy to make a go at this.

My apologies if I jumped the gun here! There's still quite a bit of work to do on the PR and I'd be happy to collaborate on this with you. Please let me know. Thanks again 🙏

[rlankfo]

/label -waiting-for-code-owners