[receiver/k8scluster] Check resource support before setting up watchers

[dmitryax]

This change adds additional validation using discovery API before setting up watchers on particular k8s resources. Otherwise, if k8s API doesn't support a k8s resource, setting up watchers breaks the receiver.

This change also adds support of batch/v1beta1 CronJob resources.

This makes the receiver able to work with k8s server versions 1.19 and 1.20 which are still available on some cloud providers.

Fixes: #9414

[mx-psi]

@open-telemetry/collector-contrib-approvers is there anyone with more knowledge of Kubernetes internals than me that can review this? I don't feel confident enough to be able to give a thorough review here

[bogdandrutu]

cc @dashpole

[jpkrohling]

This looks good to me, but it would be good to decide when to remove the deprecated batch API from this code.

[dashpole]

Here is a different idea: Can we feature-gate the use of these new k8s APIs? Right now, since most users are on versions without support for the new APIs, they need to stay on the old ones, and we should use the old ones by default. In ~6 months, most users should be on the new ones (but many lag behind), and we should change the default to use the new ones.

Edit: alternatively, we can just stick with the older APIs.

I'm not sure using discovery APIs is a good idea. It isn't a common pattern in k8s to check something only once. APIs can also be updated without restarting the collector.

[jpkrohling]

Can we feature-gate the use of these new k8s APIs?

The batch v1 API has been released a couple of versions ago, hasn't it?

[dashpole]

The batch v1 API has been released a couple of versions ago, hasn't it?

It was, but the k8s community has a fairly wide support window for older versions, since most managed k8s offerings lag ~3 versions behind what is just released.

[dmitryax]

@dashpole thanks for your feedback. We can use feature gates instead of discovery API. Ideally I wanted to make sure that if the informer API is not available for some resource, the receiver would keep working providing metrics for other resources. Do you have any idea how that can be achieved without using discovery API?

[dashpole]

Do you have any idea how that can be achieved without using discovery API?

The discovery API is definitely the right way of doing it. It would probably be acceptable to use the discovery API if the receiver fails if the discovery API isn't available (i.e. don't just log the error--return it)

[dmitryax]

The discovery API is definitely the right way of doing it. It would probably be acceptable to use the discovery API if the receiver fails if the discovery API isn't available (i.e. don't just log the error--return it)

I like this approach. I'll change it accordingly

[dashpole]

this looks reasonable to me.