upcoming releases of k8s/OpenShift are removing some APIs - need to review Kiali server/operator code

[jmazzitelli]

The operator SDK folks are reminding everyone about the following removals of deprecated k8s APIs. We need to go through our stuff and make sure we do what is necessary to remove anything in our code and in our operator so things don't break in the upcoming new releases of k8s and OpenShift.

See: https://kubernetes.io/docs/reference/using-api/deprecation-guide/

• APis removed from k8s 1.25/Openshift 4.12:

Group	Version	Resource	How to upgrade
batch	v1beta1	cronjobs	use v1 instead of v1beta1. v1 has been available since k8s v1.21/Openshift 4.8. No notable changes
discovery.k8s.io	v1beta1	endpointslices	use v1 instead of v1beta1. v1 has been available since k8s v1.21/Openshift 4.8. See the guide to check the notable changes
events.k8s.io	v1beta1	events	use v1 instead of v1beta1. v1 has been available since k8s v1.19/Openshift 4.6. See the guide to check the notable changes
autoscaling	v2beta1	horizontalpodautoscalers	use v2 instead of v2beta1. v2 has been available since k8s v1.23/Openshift 4.10. No notable changes
policy	v1beta1	poddisruptionbudgets	use v1 instead of v1beta1. v1 has been available since k8s v1.21/Openshift 4.8. See the guide to check the notable changes.
policy	v1beta1	podsecuritypolicies	Migrated to 3rd-party admission webhooks.
node.k8s.io	v1beta1	runtimeclasses	use v1 instead of v1beta1. v1 has been available since k8s v1.20/Openshift 4.7. No notable changes

• APis removed from k8s 1.26/Openshift 4.13:

Group	Version	Resource	How to upgrade
autoscaling	v2beta2	horizontalpodautoscalers	use v2 instead of v2beta2. v2 has been available since k8s v1.23/Openshift 4.10. No notable changes
flowcontrol.apiserver.k8s.io	v1beta1	flowschemas	use v1beta2 instead of v1beta1. v1beta2 has been available since k8s v1.23/Openshift 4.10. No notable changes
flowcontrol.apiserver.k8s.io	v1beta1	prioritylevelconfigurations	use v1beta2 instead of v1beta1. v1beta2 has been available since k8s v1.23/Openshift 4.10. No notable changes

[jmazzitelli]

UPDATE: As of May 16, 2022, I believe this is the only one left we have to address.

Group	Version	Resource	How to upgrade
batch	v1beta1	cronjobs	use v1 instead of v1beta1. v1 has been available since k8s v1.21/Openshift 4.8. No notable changes

For CronJob, see all references to batch_v1beta1.CronJob - https://github.com/kiali/kiali/search?q=batch_v1beta1

[jmazzitelli]

UPDATE: the HPA change is done in this PR and has been merged: kiali/kiali-operator#536

Group	Version	Resource	How to upgrade
autoscaling	v2beta2	horizontalpodautoscalers	use v2 instead of v2beta2. v2 has been available since k8s v1.23/Openshift 4.10. No notable changes.

Only other one that appears relevant for Kiali is the HPA (horizontal pod autoscaler) - it is used by the operator to create an HPA object.

Here's the k8s docs on the new v2: https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/

Here's the touch points in the operator codebase that use HPA - some of these (not all) need to change:

• https://github.com/kiali/kiali-operator/blob/v1.50.0/roles/default/kiali-deploy/templates/openshift/hpa.yaml
• https://github.com/kiali/kiali-operator/blob/v1.50.0/roles/default/kiali-deploy/templates/kubernetes/hpa.yaml
• https://github.com/kiali/kiali-operator/blob/v1.50.0/roles/default/kiali-remove/defaults/main.yml#L7
• https://github.com/kiali/kiali-operator/blob/v1.50.0/crd-docs/crd/kiali.io_kialis.yaml#L273-L293
• https://github.com/kiali/kiali-operator/blob/v1.50.0/roles/default/kiali-deploy/defaults/main.yml#L63-L65
• https://github.com/kiali/kiali-operator/blob/v1.50.0/crd-docs/cr/kiali.io_v1alpha1_kiali.yaml#L104-L110

[AllexVeldman]

Be aware that with the operator v1.51.0 using autoscaling/v2 in a cluster older than k8s 1.23 (most cloud providers are at 1.22.x) this results in a Kiali CR with a Failed status and no HPA created:

Failed to find exact match for autoscaling/v2.HorizontalPodAutoscaler by [kind, name, singularName, shortNames]

[jmazzitelli]

Yes. This issue should have been mentioned in the 1.51 release notes (if not, we should make sure we add it).
If you are telling Kiali to create a HPA and you hit this, you simply just need to add the api_version to your Kiali CR:

    hpa:
      api_version: "autoscaling/v2beta2"

That used to be the default but no longer is. So just explicitly define it and it will work as before.

(UPDATE: didn't make the release notes - I'm adding it now: kiali/kiali.io#534)

[AllexVeldman]

That worked, thanks!
Although It's odd I have to set the api_version while I left .spec.deployment.hpa.spec empty so no HPA is created.

[jmazzitelli]

@AllexVeldman Hmm... so.. you are saying you were getting that original error when BOTH spec.deployment.hpa.api_version and spec.deployment.hpa.spec were empty/not defined??

Or are you saying you left spec.deployment.hpa.spec empty/undefined and it didn't create the HPA even though you specified the api_version? Because that IS expected - the hpa.spec controls whether the HPA is created or not. The api_version just indicates WHAT api version the HPA should be declared as. But without a hpa.spec, Kiali will not create anything.

However, if it is the former, then we need to find out why. None of our testing is showing that. I wonder if it has to do with removing? Maybe we should do some conditional checking to set the default depending on the k8s version?

So - @AllexVeldman can you provide a more succinct description of WHEN you are seeing errors and WHAT workaround fixes it for you? I'm trying to determine if there is an edge case here that we missed.

[AllexVeldman]

The original error happend when there was no hpa section in my Kiali resource at all.

Creates error:

apiVersion: kiali.io/v1alpha1
kind: Kiali
metadata:
  name: kiali
  namespace: istio-system
spec:
  auth:
    strategy: "token"
  deployment:
    ingress_enabled: false
    view_only_mode: true
    accessible_namespaces: "**"
    ...
  istio_namespace: "istio-system"
  external_services:
    grafana:
      enabled: true
      ...

Does not create the error:

apiVersion: kiali.io/v1alpha1
kind: Kiali
metadata:
  name: kiali
  namespace: istio-system
spec:
  auth:
    strategy: "token"
  deployment:
    ingress_enabled: false
    view_only_mode: true
    accessible_namespaces: "**"
    ...
    hpa:
      api_version: "autoscaling/v2beta2"
  istio_namespace: "istio-system"
  external_services:
    grafana:
      enabled: true
      ...

[jmazzitelli]

@AllexVeldman OK, it must be because we are testing on a cluster newer enough to at least have v2 that we don't see this. Can you provide me a little bit more of the operator logs. Maybe snip a couple of the ansible task logs before and after the error? I would like to try to figure out where this is happening in the code. At the moment, I do not have a cluster that at least does not support hpa v2, so I can't test that right this second. But thanks for at least confirming the workaround. I think this also warrants another github issue (bug issue).

UPDATE: I have a feeling it is this: https://github.com/kiali/kiali-operator/blob/v1.51/roles/default/kiali-deploy/tasks/kubernetes/k8s-main.yml#L28-L37

[jmazzitelli]

I think this also warrants another github issue (bug issue).

Bug issue created - let's continue the discussion there as needed: #5115

[jshaughn]

@jmazzitelli , OS 4.12 is the next release (k8s 1.25), anything left to worry about? Or can we close this?

[jmazzitelli]

all issues relevant to kiali have been addressed. We can close this.