no.scale.down.node.pod.kube.system.unmovable with helm in default configuration on GKE

[jsoref]

Which component: controller / helm chart

Describe the bug

The helm chart doesn't define a PDB, but does default to installing into kube-system which results in the Kubernetes Cluster Autoscaler (e.g. GKE) getting upset when it wants to scale down a node containing sealed-secrets.

https://cloud.google.com/kubernetes-engine/docs/how-to/cluster-autoscaler-visibility

"no.scale.down.node.pod.kube.system.unmovable"

• Pod is blocking scale down because it's a non-DaemonSet, non-mirrored, Pod without a PodDisruptionBudget in the kube-system namespace. Parameters: Name of the blocking pod.

• Follow the instructions in the Kubernetes Cluster Autoscaler FAQ to set a PodDisruptionBudget to enable cluster autoscaler to move Pods in the kube-system namespace.

no.scale.down.node.pod.kube.system.unmovable log event

{
  "insertId": "...",
  "jsonPayload": {
    "noDecisionStatus": {
      "measureTime": "...",
      "noScaleDown": {
        "nodesTotalCount": 2,
        "nodes": [
          {
            "node": {
              "mig": {
                "nodepool": "default",
                "name": "gke-default-cluster-default-...-grp",
                "zone": "us-central1-b"
              },
              "cpuRatio": 31,
              "memRatio": 34,
              "name": "gke-default-cluster-default-...-..."
            },
            "reason": {
              "parameters": [
                "sealed-secrets-...-..."
              ],
              "messageId": "no.scale.down.node.pod.kube.system.unmovable"
            }
          },
          ...
        ]
      }
    }
  },
  "resource": {
    "type": "k8s_cluster",
    "labels": {
      "project_id": "...-...-...",
      "cluster_name": "default-cluster",
      "location": "us-central1"
    }
  },
  "timestamp": "2023-09-11T17:26:35.912644608Z",
  "logName": "projects/...-...-.../logs/container.googleapis.com%2Fcluster-autoscaler-visibility",
  "receiveTimestamp": "2023-09-11T17:26:36.109845771Z"
}

To Reproduce
Steps to reproduce the behavior:

1. Create a GKE cluster
2. Use helm to install sealed-secrets
3. Poke the cluster enough such that the autoscaler decides it wants to move the sealed-secret pod
4. Check GKE cluster logs

Expected behavior
The helm chart should be able to create a PDB (probably allowing the pod to be deleted, but with its configuration being configurable by values)

Version of Kubernetes:

• Output of kubectl version:

Server Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.6-gke.1700", GitCommit:"050932de5e9f5f588e3e912efcaee2616df0a8fc", GitTreeState:"clean", BuildDate:"2023-07-17T09:24:53Z", GoVersion:"go1.19.10 X:boringcrypto", Compiler:"gc", Platform:"linux/amd64"}

Additional context
https://www.google.com/search?q=%22no.scale.down.node.pod.kube.system.unmovable%22&oq=%22no.scale.down.node.pod.kube.system.unmovable%22&aqs=chrome..69i57j0i30j0i5i30.258j0j7&sourceid=chrome&ie=UTF-8

[alemorcuq]

Hi, @jsoref. Thanks for reporting this, we will look into it.

In the meantime, you can use the chart parameter extraDeploy to deploy a PDB alongisde the chart.