Kubernetes SD fails to discover Ingress in Kubernetes v1.22

[tksm]

As of Prometheus v2.29.1, Kubernetes SD uses the networking.k8s.io/v1beta1 Ingress API that is deprecated in Kubernetes v1.19. The networking.k8s.io/v1beta1 Ingress API was removed in Kubernetes v1.22, so Kubernetes SD fails to discover Ingress in Kubernetes v1.22.

In Kubernetes v1.22.0 and Prometheus v2.29.1, I confirm the following log.

level=error ts=2021-08-12T07:06:09.600Z caller=klog.go:116 component=k8s_client_runtime func=ErrorDepth msg="pkg/mod/k8s.io/client-go@v0.21.3/tools/cache/reflector.go:167: Failed to watch *v1beta1.Ingress: failed to list *v1beta1.Ingress: the server could not find the requested resource"

The networking.k8s.io/v1 Ingress API is available since Kubernetes v1.19. If Kubernetes SD only supports Kubernetes v1.19 or later, I think we can simply update to the networking.k8s.io/v1 Ingress API.

If Kubernetes SD supports Kubernetes version less than v1.19, we need to use the different Ingress API based on the Kubernetes version.

[roidelapluie]

Thank you! Are you aware about how other pieces of software deal with this? It seems that if we update the k8s library, we might not be able to support versions below 1.19 anymore.

cc @paulfantom @lilic @simonpasquier @brancz @metalmatze to gather broader input.

[LeviHarrison]

We could use https://pkg.go.dev/k8s.io/client-go@v0.22.0/discovery#ServerSupportsVersion to check if the server supports networking.k8s.io/v1, and if not use the beta.

The latest version of the client supports both APIs https://pkg.go.dev/k8s.io/client-go@v0.22.0/kubernetes#Clientset.NetworkingV1.

[tksm]

In the case of Endpoint and EndpointSlice, CoreDNS's Kubernetes plugin determines which endpoint object type to watch (Endpoint or EndpointSlice) based on the Kubernetes version and the supportability.

• https://github.com/coredns/coredns/blob/v1.8.4/plugin/kubernetes/kubernetes.go#L260-L269
• https://github.com/coredns/coredns/blob/v1.8.4/plugin/kubernetes/kubernetes.go#L304

[paulfantom]

AFAIK kubernetes client library should still have networking.k8s.io/v1beta1 definition till 0.25 (k8s 1.25) so implementing logic from coredns example makes sense for the next year or so. After that time API should be dropped from the client library and prometheus will need to drop support for Ingresses pre k8s 1.19 to keep the client library up-to-date.