Automate Performance Testing

[realshuting]

Is your feature request related to a problem? Please describe.

With #2248, we managed to build the test env and spin up workloads manually to measure Kyverno performance.

This issue can be used to track the full automation of the scaling testing and generate the final report for Kyverno performance.

[dhavalgshah]

Is there any progress around this issue?

Are there any relevant discussions apart from that of #2248 going on for this?

Background:

In Kyverno Office Hours at KubeConEU 22 a fellow attendee shared his experience on performance issues with Kyverno.

That led me to do some quick checks and I found this issue and #2248.

[husnialhamdani]

Hi @dhavalgshah,
We are about to finish our automate performance testing tools, based on our observation so far, the Kyverno required us to adjust the default memory limit config in large workloads (around 4k-5k Kubernetes total resources), so we would recommend to update the memory limit to 768Mi (2x default config) to avoid OOM problem.

This screenshot we test in 4-5k Kubernetes total resources and Kyverno memory usage was hitted the default max limit,

Could you also please share your environment details?, like how many Kubernetes worker and what Kyverno policies that installed in your cluster.