feat(audit): use a worker pool for Audit policies

[KhaledEmaraDev]

Explanation

This PR limits the number of goroutines used by Audit policies to 8 to limit contention and scheduling overhead. This should net a nice performance improvement.

Related issue

Milestone of this PR

Documentation (required for features)

My PR contains new or altered behavior to Kyverno.

• I have sent the draft PR to add or update the documentation and the link is:

What type of PR is this

Proposed Changes

Proof Manifests

The following results were tested with settings:

1. event generation off
2. CPU limits set to 2 for the k8s node docker update --cpus=2 <node>
3. ./start.sh tests/kyverno-pods-dry-run.js 100 1000

• audit PSS policies with this change

http_req_duration: avg=409.45ms min=99.54ms med=398.78ms max=1.29s    p(90)=698.04ms p(95)=798.72ms

• audit PSS policies with v1.12.0-rc.4

http_req_duration: avg=846.89ms min=2.98ms  med=601.92ms max=4.29s    p(90)=1.8s     p(95)=2.1s

• enforce PSS policies with this change

http_req_duration: avg=1.21s   min=102.67ms med=1.19s   max=2.89s    p(90)=1.79s    p(95)=1.99s

• enforce PSS policies with v1.12.0-rc.4

http_req_duration: avg=1.25s    min=94.74ms  med=1.19s   max=3.19s    p(90)=1.8s     p(95)=2.19s

Checklist

• I have read the contributing guidelines.
• I have read the PR documentation guide and followed the process including adding proof manifests to this PR.
• This is a bug fix and I have added unit tests that prove my fix is effective.
• This is a feature and I have added CLI tests that are applicable.
• My PR needs to be cherry picked to a specific release branch which is .
• My PR contains new or altered behavior to Kyverno and
  • CLI support should be added and my PR doesn't contain that functionality.

Further Comments

[realshuting]

I added stress test results comparing this change to rc.4.

[realshuting]

Fixed unit tests.

[codecov]

Codecov Report

Attention: Patch coverage is 41.86047% with 50 lines in your changes are missing coverage. Please review.

Project coverage is 10.10%. Comparing base (90d1440) to head (af94a53).

❗ Current head af94a53 differs from pull request most recent head b630257. Consider uploading reports for the commit b630257 to get more accurate results

Files	Patch %	Lines
pkg/webhooks/resource/validation/validation.go	22.91%	32 Missing and 5 partials ⚠️
cmd/kyverno/main.go	0.00%	6 Missing ⚠️
pkg/webhooks/resource/validation/utils.go	45.45%	3 Missing and 3 partials ⚠️
pkg/webhooks/resource/handlers.go	90.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff            @@
##            main   #10048      +/-   ##
=========================================
+ Coverage   9.96%   10.10%   +0.14%     
=========================================
  Files       1029     1030       +1     
  Lines      91686    91722      +36     
=========================================
+ Hits        9134     9266     +132     
+ Misses     81554    81437     -117     
- Partials     998     1019      +21     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[realshuting]

3/4 of PSS load tests failed, did they exceed the threshold?

https://github.com/kyverno/kyverno/actions/runs/8660137654/job/23747565640?pr=10048

[vishal-chdhry]

@realshuting we need to add documentation for the new flags

[realshuting]

@realshuting we need to add documentation for the new flags

Yes, opened kyverno/website#1210.

[realshuting]

/cherry-pick release-1.12

[gcp-cherry-pick-bot]

Cherry-pick failed with Merge error fb40aa5f38500ebf739b7efd7f6f3d4f74d9c8a1 into temp-cherry-pick-4a89e6-release-1.12