Make webhooks configurable

[JimBugwadia]

Is your feature request related to a problem? Please describe.

Currently Kyverno auto-creates and updates the validating and mutating webhooks, and user changes will be overwritten.

Describe the solution you'd like

Users should be allowed to tune the webhook configurations for their deployments.

Resource filters (currently set via args and configmap) should be applied to the webhook settings, to optimize which requests are handled by Kyverno.

Users should be able to migrate to failurePolicy=fail from failurePolicy=ignore.

Additional context

See slack discussions:

https://kubernetes.slack.com/archives/CLGR9BJU9/p1623155044256500

https://kubernetes.slack.com/archives/CLGR9BJU9/p1622078250126000

Also see: #893

[yanniszark]

Pasting an idea here from Kubernetes slack: https://kubernetes.slack.com/archives/CLGR9BJU9/p1623155044256500

Instead of letting users figure out the correct filter, why not continuously autogenerate it in the kyverno controller based on the currently existing policies? This ensure the filter is continuously optimal. More specifically:

• For resource selector, this is very straightforward, just compile a list of (apiVersion, kind) used by policies in the cluster.
• For namespace selector, this is not straightforward since Kyverno supports wildcard matching, which K8s does not.
  • We could compile a list of namespaces by running every policy's namespace selector regex against all existing namespaces. We would have to reason about the time it takes.
• For object selector, this is not straightforward since Kyverno supports wildcard matching, which K8s does not.
  • We could follow the same tactic I mentioned for the namespace, but I think it gets prohibitively expensive at this point.

TLDR, implementing just the first bullet would be a huge improvement over the current state.
It would also make it easier to migrate to failurePolicy=fail, since only the availability of APIs appearing in Kyverno policies would be affected.

[chipzoller]

I also support @yanniszark idea in the above comment. The webhook should be dynamically composed based upon the subjects of the policies under it. If a user only has policies created which apply to Service resources, the webhook should only be configured to send Service AdmissionReviews to Kyverno. Etc.

[diranged]

Seconding this bit:

Resource filters (currently set via args and configmap) should be applied to the webhook settings, to optimize which requests are handled by Kyverno.

We have run into a dozen situations or more where Kyverno was not in a good state and normal cluster operations could not succeed because of the failing webhooks.

[realshuting]

Linking to the design doc https://docs.google.com/document/d/1Y7_7ow4DgCLyCFQcFVz1vHclghazAKZyolIfprtNURc/edit?usp=sharing.