Installing Kubecost on K8s v1.25 clusters fails due to PodSecurityPolicy

[michaelmdresser]

To reproduce

Create a K8s v1.25 cluster with K3d:

k3d cluster create --image rancher/k3s:v1.25.3-rc3-k3s1

Confirm version:

kubectl version --short

Client Version: v1.25.3
Kustomize Version: v4.5.7
Server Version: v1.25.3-rc3+k3s1

Try to install the latest version of Kubecost (v1.98.0-rc.3):

helm upgrade \
    -i \
    --create-namespace kubecost \
    kubecost/cost-analyzer \
    --namespace kubecost \
    --version "1.98.0-rc.3"

Error: unable to build kubernetes objects from release manifest: unable to recognize "": no matches for kind "PodSecurityPolicy" in version "policy/v1beta1"

Expected behavior

Kubecost installs successfully on the K8s v1.25 cluster without any modification to default values.

Workaround

This can be worked around by disabling PSPs. Unfortunately, Grafana also must be disabled to remove its PSP.

helm upgrade \
    -i \
    --create-namespace kubecost \
    kubecost/cost-analyzer \
    --namespace kubecost \
    --version "1.98.0-rc.3" \
    --set "podSecurityPolicy.enabled=false" \
    --set "global.grafana.enabled=false" \
    --set "global.grafana.proxy=false"

Note that the Grafana proxy option has to be disabled because otherwise the Kubecost frontend container errors out due to the Grafana upstream being unavailable.

Fix ideas

• Disable Kubecost PSPs by default on K8s version 1.25+, possibly replacing with PodSecurity admission controller if desired.
• Update the Grafana subchart with the same logic as above. Current logic:

  cost-analyzer-helm-chart/cost-analyzer/charts/grafana/templates/podsecuritypolicy.yaml

  Lines 1 to 4 in 56b880c

  	{{ if .Values.global.grafana.enabled }}
  	{{- if .Values.rbac.pspEnabled }}
  	apiVersion: policy/v1beta1
  	kind: PodSecurityPolicy

[michaelmdresser]

• Related to Update depricated kubernetes APIs #1493

[lgmorand]

+1. was coming to create a bug too.

[AjayTripathy]

My suggestion is we document and make clear the workaround for v1.98 and implement it seamlessly in v1.99 since we probably need to give folks adequate warning before turning off PSPs, and most people on cloud providers are not yet on v1.25.

[michaelmdresser]

We've also had a request that we place PSPs with SecurityContext constraints where appropriate, but I'm not sure how applicable they are -- I'm not familiar with our existing PSPs. Just noting for the record!

Ref: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/

cc @srpomeroy

[michaelmdresser]

There is also an alternative workaround that (might?) not disable Grafana in the process:

helm upgrade \
    -i \
    --create-namespace kubecost \
    kubecost/cost-analyzer \
    --namespace kubecost \
    --version "1.98.0-rc.3" \
    --set "podSecurityPolicy.enabled=false" \
    --set "networkCosts.podSecurityPolicy.enabled=false" \
    --set "prometheus.podSecurityPolicy.enabled=false" \
    --set "grafana.rbac.pspEnabled=false"

Haven't tested personally yet.

[fungiboletus]

Haven't tested personally yet.

I have and it works well. Thanks.

[zioproto]

Installation of Kubecost on Kubernetes 1.25 still fails when installing without Helm:

kubectl apply -f https://raw.githubusercontent.com/kubecost/cost-analyzer-helm-chart/master/kubecost.yaml --namespace kubecost

[AjayTripathy]

Thanks for this heads up @zioproto , I think we generate that file from a template where we assume we're on helm less than 1.25. Let's modify the template.