[Bug] kyverno may not be able to handle 3.5k policy resources with 2 rules each, total of 7k rules

[aborrero]

Kyverno Version

1.10.6

Description

We are working on migrating away from PodSecurityPolicy (PSP) in order to migrate off kubernetes 1.24 into 1.25.

For that task we selected kyverno, and deployed kyverno namespaced policy resources that we considered would be equivalent to what PSP were doing.

Our kubenertes cluster is a platform-as-a-service, and we create a namespace for each user to deploy their own workloads, and therefore we work with namespaced Policy resources. We deploy kubernetes in bare-metal (via kubeadm) in virtual machines in our own datacenter.

We deploy kyverno via helm, using the 3.0.9 chart, which deploys kyverno v1.10.7.

We initially tried to deploy 3.5k policy resources, each with 2 rules (mutate, validate), therefore a total of 7k rules.
The different kyverno components had problems with the default CPU/MEM resource requests and limits. The pods were constantly getting OOM-killed, etc. We decided to lift the resource limits entirely. This change made kyverno apparently stable.

Then, I detected some of the kyverno policies had not been correctly created because we were using a name too long (the resource name was templated with the user name). So I decided to rename each policy and each rule with the same string.

When running the rename script (delete the old policy resource, create a new policy resource which is the same, but with different name), the cluster experimented an extremely high load, to the point of becoming unusable.

We needed to perform a number of rescue operations, including completely disabling kyverno as documented here: https://kyverno.io/docs/troubleshooting/#api-server-is-blocked

In our system, a kyverno policy resource looks like this:

apiVersion: kyverno.io/v1
kind: Policy
metadata:
  annotations:
    kyverno.io/kubernetes-version: "1.24"
    kyverno.io/kyverno-version: 1.10.7
    policies.kyverno.io/category: Toolforge pod policy
    policies.kyverno.io/description: Toolforge tool account pod security mutation
      and validation
    policies.kyverno.io/subject: Pod
    policies.kyverno.io/title: Toolforge arturo-test-tool pod policy
    toolforge.org/kyverno_pod_policy_version: "2"
  creationTimestamp: "2024-06-12T16:45:35Z"
  generation: 1
  name: toolforge-kyverno-pod-policy
  namespace: tool-arturo-test-tool
  resourceVersion: "1997638015"
  uid: 4c10361a-65b2-4141-a5b0-011dc3d6d36d
spec:
  background: true
  rules:
  - match:
      all:
      - resources:
          kinds:
          - Pod
    mutate:
      patchStrategicMerge:
        spec:
          containers:
          - (name): '*'
            securityContext:
              allowPrivilegeEscalation: false
              capabilities:
                drop:
                - ALL
              privileged: false
              runAsNonRoot: true
          hostIPC: false
          hostNetwork: false
          hostPID: false
          securityContext:
            fsGroup: 54005
            runAsGroup: 54005
            runAsUser: 54005
            seccompProfile:
              type: RuntimeDefault
    name: toolforge-mutate-pod-policy
  - match:
      any:
      - resources:
          kinds:
          - Pod
    name: toolforge-validate-pod-policy
    validate:
      message: pod security configuration must be correct
      pattern:
        spec:
          =(ephemerealContainers):
          - =(securityContext):
              =(allowPrivilegeEscalation): false
              =(capabilities):
                drop:
                - ALL
              =(privileged): false
              =(runAsGroup): 54005
              =(runAsNonRoot): true
              =(runAsUser): 54005
              =(seccompProfile):
                type: RuntimeDefault
              x(seLinuxOptions): null
          =(hostIPC): false
          =(hostNetwork): false
          =(hostPID): false
          =(initContainers):
          - =(securityContext):
              =(allowPrivilegeEscalation): false
              =(capabilities):
                drop:
                - ALL
              =(privileged): false
              =(runAsGroup): 54005
              =(runAsNonRoot): true
              =(runAsUser): 54005
              =(seccompProfile):
                type: RuntimeDefault
              x(seLinuxOptions): null
          =(workingDir): /data/project/arturo-test-tool
          containers:
          - =(securityContext):
              =(allowPrivilegeEscalation): false
              =(capabilities):
                drop:
                - ALL
              =(privileged): false
              =(runAsGroup): 54005
              =(runAsNonRoot): true
              =(runAsUser): 54005
              =(seccompProfile):
                type: RuntimeDefault
              x(seLinuxOptions): null
          securityContext:
            =(runAsNonRoot): true
            =(seccompProfile):
              type: RuntimeDefault
            =(supplementalGroups): 1-65535
            fsGroup: 54005
            runAsGroup: 54005
            runAsUser: 54005
  validationFailureAction: Audit

I cannot tell exactly what happened. I've seen other reports about Kyverno's UpdateRequests flooding the k8s API Server, which may be the case here, because the k8s control plane nodes experimented an extremely high load during this incident.

Also, reading the documentation about resource limits https://kyverno.io/docs/installation/scaling/ I have some doubts if kyverno was ever tested to work in an environment with 7k rules.

So my questions are:

• is this failure scenario known already? is this exactly the same as other similar and already opened issues like 8668 and 8668
• are we doing something obviously wrong?
• is kyverno able to handle 3.5k policy resources with 2 rules each, totaling 7k rules?
• do you have any special recommendations / hints about how to accommodate for our PSP-replacement scenario? Maybe, to split the policies into 2 separate things, or make a single huge cluster-level policy to achieve the same?
• do you think the issue is somewhere else? like, the api-server needs to be scaled up, or something.

Slack discussion

No response

Troubleshooting

• I have read and followed the documentation AND the troubleshooting guide.
• I have searched other issues in this repository and mine is not recorded.

[welcome]

Thanks for opening your first issue here! Be sure to follow the issue template!

[aborrero]

I was able to reproduce the problem, let me share how can you reproduce yourself.

For Wikimedia Toolforge, we use a local kubernetes setup for development that we call lima-kilo, which is basically a kind installation inside a lima-vm.

1. install lima-vm in your system https://lima-vm.io/
2. download lima-kilo git repo: https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo
3. start the environment with the start-devenv.sh script. This will create the VM and install Wikimedia Toolforge in it. It can take 30 minutes to finish.
4. after this step, you have a working kubectl to interface with the cluster.
5. create kyverno load. For example, the helpers/toolforge_kyverno_load_many_resources.sh script creates 4000 policy resources.
6. see the cluster crash

NOTE: this setup is very alive, and some of the steps will fetch data from other repos dynamically, so this setup may be different in, let's say, 3 months period

I have played with different settings, in particular:

• the webhook calls constantly time out, because kyverno doesn't seem to be able to evaluate the 4k policy resources in under the max 30 seconds that k8s allows for the webhook call.
• having higher CPU/MEM resources requests and limits for kyverno. I believe a higher number makes this issue go even worse, because kyverno has higher power to hammer the k8s API with the different daemons.
• having less CPU/MEM resouces requests and limits for kyverno. This result in kyverno pods constantly getting restarted by kubernetes because OOMkilled, failing liveness probes and such. Given the fail-closed setting in our policies, this also has a strong impact on the function of kubernetes itself.

Do you have any recommendations for this setup? Maybe we are making some obvious mistake when crafting our policy resources.

I'm wondering if we could benefit from different approaches:

• having the policy resources being smaller, i.e, doing less things in a single policy rule
• doing only validation instead of mutation+validation.
• completely stopping some of the daemons could help, i.e, if we can consider the report controller optional, we could stop it if kyverno operations wont be impacted

[realshuting]

Hi @aborrero - can you help us clarify a few things?

1. How many replicas have you configured for the admission controller? With such a big cluster, you may want to scale it out to distribute requests across all instances.
2. Your namespaced policies are applied upon admission requests, are there active admission events?

Since you have namespaced policies only, each with a mutate and a validate rule. Upon one single admission request there should be only two rules applied, I don't think it will cause the timeout issue.

You mentioned the cluster crashed after policies were installed, can you further elaborate? Did the admission controller pod crash (OOM?) causing the cluster to crash or something else?

5. create kyverno load. For example, the helpers/toolforge_kyverno_load_many_resources.sh script creates 4000 policy resources.
6. see the cluster crash

And yes, if you don't need reporting at this point, you can disable the report controller during deployment via https://github.com/kyverno/kyverno/blob/main/charts/kyverno/values.yaml#L2115C3-L2115C10, and turn off admission reports via https://github.com/kyverno/kyverno/blob/main/charts/kyverno/values.yaml#L603.

[aborrero]

Hey, @realshuting thanks for the follow up!

Hi @aborrero - can you help us clarify a few things?

1. How many replicas have you configured for the admission controller? With such a big cluster, you may want to scale it out to distribute requests across all instances.

I was using 3 replicas. I'd be happy to scale this further if you think this can help. I'd be happy to use 5, 7, 9 or whatever number of replicas.

2. Your namespaced policies are applied upon admission requests, are there active admission events?

Yes, there are pods constantly being created on the cluster. This is the expected usage of the cluster.

Since you have namespaced policies only, each with a mutate and a validate rule. Upon one single admission request there should be only two rules applied, I don't think it will cause the timeout issue.

You mentioned the cluster crashed after policies were installed, can you further elaborate? Did the admission controller pod crash (OOM?) causing the cluster to crash or something else?

Yes, I see all kyverno pods getting OOMkilled, not only the admission controller pods, but all the others as well.

I know pretty much nothing about the kyverno internals, so I can't tell what is going on inside. I can only tell that the cluster becomes unavailable, and I could see several of the k8s components crashing, including the api-server and the controller-manager. Maybe etcd gets overloaded, thus cascading into everything else crashing?

This should be reproducible with the method I described in the earlier comment.

And yes, if you don't need reporting at this point, you can disable the report controller during deployment via https://github.com/kyverno/kyverno/blob/main/charts/kyverno/values.yaml#L2115C3-L2115C10, and turn off admission reports via https://github.com/kyverno/kyverno/blob/main/charts/kyverno/values.yaml#L603.

Ok, I'll try this and the other recommendations, and report back.

[aborrero]

Tried again:

• used this helm values configuration:

admissionController:
  replicas: 7
  container:
    resources:
      limits:
        cpu: 1
        memory: 4Gi
      requests:
        cpu: 500m
        memory: 2Gi

backgroundController:
  replicas: 4
  resources:
    limits:
      cpu: 1
      memory: 2Gi
    requests:
      cpu: 500m
      memory: 1Gi

cleanupController:
  replicas: 2
  resources:
    limits:
      cpu: 1
      memory: 2Gi
    requests:
      cpu: 500m
      memory: 1Gi

reportsController:
  enabled: false

features:
  admissionReports:
    enabled: false

• created about 1.2k policy resources in a namespace
• tried to create a deployment in the namespace affected by the policy resources

The cluster survives and I can operate via kubectl normally. However, a deployment resource in the affected namespace cannot be created. I saw this error in the k8s apiserver:

E0620 09:56:09.719777       1 writers.go:118] apiserver was unable to write a JSON response: http: Handler timeout
E0620 09:56:09.719840       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout
I0620 09:56:09.719969       1 trace.go:205] Trace[1344687648]: "Call mutating webhook" configuration:kyverno-resource-mutating-webhook-cfg,webhook:mutate.kyverno.svc-fail,resource:apps/v1, Resource=deployments,subresource:,operation:CREATE,UID:a4b09e52-ae1c-4fcb-a38f-11ded5033ad4 (20-Jun-2024 09:55:59.719) (total time: 10000ms):
Trace[1344687648]: [10.000241574s] [10.000241574s] END
W0620 09:56:09.720478       1 dispatcher.go:195] Failed calling webhook, failing closed mutate.kyverno.svc-fail: failed calling webhook "mutate.kyverno.svc-fail": failed to call webhook: Post "https://kyverno-svc.kyverno.svc:443/mutate/fail?timeout=10s": context canceled
E0620 09:56:09.720529       1 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 720.705µs, panicked: false, err: Internal error occurred: failed calling webhook "mutate.kyverno.svc-fail": failed to call webhook: Post "https://kyverno-svc.kyverno.svc:443/mutate/fail?timeout=10s": context canceled, panic-reason: <nil>
E0620 09:56:09.721148       1 writers.go:131] apiserver was unable to write a fallback JSON response: http: Handler timeout
I0620 09:56:09.722683       1 trace.go:205] Trace[319844195]: "Create" url:/apis/apps/v1/namespaces/tool-tf-test/deployments,user-agent:jobs-api toolforge_weld/1.5.0 python-requests/2.32.3,audit-id:19c7d335-77c2-4435-af96-450228524d99,client:172.18.0.2,accept:*/*,protocol:HTTP/1.1 (20-Jun-2024 09:55:59.719) (total time: 10003ms):
Trace[319844195]: [10.003439407s] [10.003439407s] END
E0620 09:56:09.723480       1 timeout.go:141] post-timeout activity - time-elapsed: 3.61054ms, POST "/apis/apps/v1/namespaces/tool-tf-test/deployments" result: <nil>

I will run a few more tests, including distributing the policy resources across namespaces, to see if that makes any difference.

[aborrero]

I used this script to try to load 4000 policy resources in 4000 different namespaces https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/blob/d35542a438699b6f18b293559b04f5bf77dc2515/helpers/toolforge_kyverno_load_many_resources.sh

The script could only create about half the policy resources before the cluster died. I think this time the problem was memory shortage on the API server. I saw it eating huge amount of memory:

$ kubectl top pods -n kube-system
NAME                                              CPU(cores)   MEMORY(bytes)   
coredns-57575c5f89-fsvsp                          2m           22Mi            
coredns-57575c5f89-p2wk5                          1m           20Mi            
etcd-toolforge-control-plane                      46m          395Mi           
kindnet-gn92f                                     1m           12Mi            
kindnet-p2dg6                                     1m           12Mi            
kindnet-wp4vb                                     1m           15Mi            
kindnet-xs54t                                     1m           12Mi            
kube-apiserver-toolforge-control-plane            415m         3445Mi          
kube-controller-manager-toolforge-control-plane   15m          169Mi           
kube-proxy-hjl7w                                  1m           19Mi            
kube-proxy-q76mh                                  1m           19Mi            
kube-proxy-w42k7                                  1m           19Mi            
kube-proxy-x6mhn                                  1m           19Mi            
kube-scheduler-toolforge-control-plane            2m           27Mi

I'll try next to scale the k8s control plane components.

[aborrero]

I have good news, scaling the k8s control plane helped to stabilize the setup and I can no longer reproduce the problem.

Overrall, I did the following changes compared to my initial report:

• scaled memory allocation for the k8s api-server (bigger base memory in the virtual machine hosting the control plane). This prevents the api-server running out of memory, and therefore crash the whole system.
• used a more realistic version of my helper script to load the 4000 policy resources. Now doing with 1 policy resource in each namespace. This helps the api-server don't timeout on the webhook as reported in an earlier comment
• increased number of replicas of the admission controller to 7
• increased number of replicas of the background controller to 4
• disabled the reports controller
• disabled admissionReports

[realshuting]

Thanks for confirming!

used a more realistic version of my helper script to load the 4000 policy resources. Now doing with 1 policy resource in each namespace. This helps the api-server don't timeout on the webhook as reported in an #10458 (comment)

I wonder what makes the API server eat up the memory? What if you have a single cluster policy matching all namespaces?

[aborrero]

I wonder what makes the API server eat up the memory? What if you have a single cluster policy matching all namespaces?

I don't know if we can easily implement this on our side.

Part of the reason we have a policy per namespace is that pods in each namespace should have a different runAsUser and runAsGroup. This means we have 4000 kyverno policies each with a different number for those fields.
We have a piece of software called maintain-kubeusers that fetches the information from LDAP and creates the namespaced kyverno policy resources with the right runAsUser and runAsGroup.

If you know how we could implement a similar ClusterPolicy I'm all ears :-)

Before closing this issue, I'd like to propose a change in the documentation, to add hints of the impact kyverno may have in the k8s api-server memory footprint.