PSP Rollout is broken

[dschunack]

Hi,

the last Chart update breaks the rollout of the PSP. API Version v1 is not available on AWS EKS 1.22, 1.23 and 1.24.

Error: resource mapping not found for name: "cluster-autoscaler-aws-cluster-autoscaler" namespace: "" from "": no matches for kind "PodSecurityPolicy" in version "policy/v1"

autoscaler/charts/cluster-autoscaler/templates/_helpers.tpl

Lines 73 to 74 in a484713

	{{- else if semverCompare ">1.21-0" $kubeTargetVersion -}}
	{{- print "policy/v1" -}}

EKS 1.22

kubectl api-resources --api-group='policy'   
NAME                   SHORTNAMES   APIVERSION       NAMESPACED   KIND
poddisruptionbudgets   pdb          policy/v1        true         PodDisruptionBudget
podsecuritypolicies    psp          policy/v1beta1   false        PodSecurityPolicy

kubectl version 
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.3", GitCommit:"816c97ab8cff8a1c72eccca1026f7820e93e0d25", GitTreeState:"clean", BuildDate:"2022-01-25T21:25:17Z", GoVersion:"go1.17.6", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"22+", GitVersion:"v1.22.16-eks-ffeb93d", GitCommit:"52e500d139bdef42fbc4540c357f0565c7867a81", GitTreeState:"clean", BuildDate:"2022-11-29T18:41:42Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}

EKS 1.24

kubectl api-resources --api-group='policy'   
NAME                   SHORTNAMES   APIVERSION       NAMESPACED   KIND
poddisruptionbudgets   pdb          policy/v1        true         PodDisruptionBudget
podsecuritypolicies    psp          policy/v1beta1   false        PodSecurityPolicy

kubectl version                       
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.3", GitCommit:"816c97ab8cff8a1c72eccca1026f7820e93e0d25", GitTreeState:"clean", BuildDate:"2022-01-25T21:25:17Z", GoVersion:"go1.17.6", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"24+", GitVersion:"v1.24.8-eks-ffeb93d", GitCommit:"abb98ec0631dfe573ec5eae40dc48fd8f2017424", GitTreeState:"clean", BuildDate:"2022-11-29T18:45:03Z", GoVersion:"go1.18.8", Compiler:"gc", Platform:"linux/amd64"}

v1 doesn't exist in the API Documentation for PSP [API ref 1.24] (https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#podsecuritypolicy-v1beta1-policy) and PSP are removed in 1.25. It make sense to revert #5357 and #5500 .

autoscaler/charts/cluster-autoscaler/templates/_helpers.tpl

Lines 66 to 76 in b57d917

	{{/*
	Return the appropriate apiVersion for podsecuritypolicy.
	*/}}
	{{- define "podsecuritypolicy.apiVersion" -}}
	{{- $kubeTargetVersion := default .Capabilities.KubeVersion.GitVersion .Values.kubeTargetVersionOverride }}
	{{- if semverCompare "<1.10-0" $kubeTargetVersion -}}
	{{- print "extensions/v1beta1" -}}
	{{- else -}}
	{{- print "policy/v1beta1" -}}
	{{- end -}}
	{{- end -}}

I will create a PR to fix this in the next minutes.

[gjtempleton]

#5480 should resolve this once merged.

[gjtempleton]

#5480 has now been merged, if you can confirm that's fixed this, we can close this off. Thanks for raising it.

[llamahunter]

tried to update to 1.25 today, and got this:

Error: resource mapping not found for name: "aws-cluster-autoscaler" namespace: "" from "": no matches for kind "PodSecurityPolicy" in version "policy/v1beta1"
ensure CRDs are installed first

Using helm chart 9.28.0 and image tag v1.25.1

[josecsotomorales]

Upgraded to k8s 1.25 ... getting this issue with helm chart 9.28.0 as well

[josecsotomorales]

This is the error I'm getting on my side: no matches for kind "PodDisruptionBudget" in version "policy/v1beta1"

[josecsotomorales]

From my research it's a Helm issue actually, tested helm template and it produces the expected policy, more details here: helm/helm#7219

[4sudiptodas]

Upgraded to aws k8s 1.27 and getting this issue with helm chart 9.28.0, unable to build kubernetes objects from current release manifest: resource mapping not found for name: "aws-cluster-autoscaler" namespace: "" from "": no matches for kind "PodDisruptionBudget" in version "policy/v1beta1" ensure CRDs are installed first

[JCBSLMN]

i'm having the same issue as @4sudiptodas

[xiaodong-xie]

Got inspirations from this comment: helm/helm#7219 (comment)

This works for me:

helm plugin install https://github.com/helm/helm-mapkubeapis
helm mapkubeapis -n kube-system cluster-autoscaler

Then upgrading cluster-autoscaler using helm upgrade --install succeeded.

[jd-sandk]

Same issue with Helm Chart: 9.29.1 when using Terraform to apply - we're were getting Error:

╷
│ Error: unable to build kubernetes objects from current release manifest: resource mapping not found for name: "cluster-autoscaler-aws-cluster-autoscaler" namespace: "" from "": no matches for kind "PodDisruptionBudget" in version "policy/v1beta1"
│ ensure CRDs are installed first
│ 
│   on modules/cluster/helm.tf line 45, in resource "helm_release" "cluster-autoscaler":
│   45: resource "helm_release" "cluster-autoscaler" {
│ 
╵

Exited with code exit status 1
CircleCI received exit code 1

We've even tried to explicitly set the following in the Helm Chart:

  set {
    name  = "kubeTargetVersionOverride"
    value = "1.27.0"
    type  = "string"
  }

As we are on v1.27 and due to seeing:

{{- define "podDisruptionBudget.apiVersion" -}}
{{- $kubeTargetVersion := default .Capabilities.KubeVersion.GitVersion .Values.kubeTargetVersionOverride }}
{{- if semverCompare "<1.21-0" $kubeTargetVersion -}}
{{- print "policy/v1beta1" -}}
{{- else -}}
{{- print "policy/v1" -}}
{{- end -}}
{{- end -}}

https://github.com/kubernetes/autoscaler/blob/master/charts/cluster-autoscaler/templates/_helpers.tpl#L78-L88

Also later based upon our looking of the Template code (https://github.com/kubernetes/autoscaler/blob/master/charts/cluster-autoscaler/templates/podsecuritypolicy.yaml#L1C3-L1C3) a member of our team thought the following might help trying to set the following on the Helm Chart:

  set {
    name  = "rbac.create"
    value = "true"
    type  = "string"
  }

  set {
    name  = "rbac.pspEnabled"
    value = "true"
    type  = "string"
  }

but then that also only leads to another similiar issue:

╷
│ Error: resource mapping not found for name: "cluster-autoscaler-aws-cluster-autoscaler" namespace: "" from "": no matches for kind "PodSecurityPolicy" in version "policy/v1beta1"
│ ensure CRDs are installed first
│ 
│   on modules/cluster/helm.tf line 45, in resource "helm_release" "cluster-autoscaler":
│   45: resource "helm_release" "cluster-autoscaler" {
│ 
╵

Exited with code exit status 1
CircleCI received exit code 1

Why is the Helm chart trying to force us to usev1beta1, when we're on EKS Cluster/Node Group (Kubernetes) Version: 1.27, when it should be trying to use v1 instead.

We have those Kinds available in our Cluster:

# kubectl api-resources --api-group='policy'   
NAME                   SHORTNAMES   APIVERSION   NAMESPACED   KIND
poddisruptionbudgets   pdb          policy/v1    true         PodDisruptionBudget

😞

[Idan-Lazar]

@jd-sandk Do you find any solutions?