Enhance K8s Events for resources "touched" by Kyverno policies

[chipzoller]

Is your feature request related to a problem? Please describe.
Kubernetes Events are a very important source of information on a given resource and serve as one of the starting points for any troubleshooting efforts. It's also usually the first stop on the road for any useful information an administrator/user might have to answer questions about a resource's disposition.

Today in Kyverno, Events on resources that are impacted or "touched" by Kyverno aren't given much consideration and are either basic and slightly misleading or nonexistent entirely. (By "touched" I mean resources that were subject to a policy in some way, ex, a Pod which violated a policy in audit mode; a Service which was mutated as the result of a policy; or a NetworkPolicy which was generated as a result of a policy.)

For example, in the case of a validate rule in audit mode, a Pod in violation may generate the following Event:

Type     Reason             Age   From                   Message
----     ------             ----  ----                   -------
Warning  PolicyViolation    12s   admission-controller   Rule(s) 'check-label-app' of policy 'check-label-app' failed to apply on the resource

A couple problems with this:

1. The From column isn't very descriptive for users who might not know what produced this. It's an admission controller but which one? Kubernetes clusters these days may have several, not including homegrown ones.
2. The statement "failed to apply on the resource" is somewhat misleading. This makes it sound like something should have happened but didn't for unknown reasons while in reality it's operating exactly as intended when a policy is placed in audit mode. The resource didn't "fail" to apply anything, it's just in violation of one or more rules.

But on resources that were mutated or generated by Kyverno, there are no events created on those target resources at all. This is especially significant because, unlike validate rules placed in audit mode which are stored in a PolicyReport CR, aside from the Kyverno logs there are no pieces of information created outwards in the cluster which tell of those mutations or generations. Without such events, some confusion tends to arise:

1. Was this supposed to happen or is there a bug somewhere?
2. Why is the desired state of my resource different from what I have recorded in my authoritative source (git, local file, etc.)?
3. Where did this new resource come from in this Namespace I manage? Who or what is responsible?
4. How do I reason about and/or treat the added or changed resources? Should they be deleted? Can I change them? What would happen?

Describe the solution you'd like
There exists some substantial room for improvement here across the board.

1. Enhance the existing events created on resources subject to validate rules to a) state the source as being Kyverno rather than "admission-controller"; b) change the message body itself to say something like "Resource X is currently in violation of rule(s) Y defined in policy Z. This is recorded in the policy report for this Namespace or Cluster."
2. Create an event for a resource which was the target of a mutate rule stating something like, "Resource X has been successfully mutated according to rule Y defined in policy Z."
3. Create an event for a resource which was created by Kyverno due to a generate rule and state the behavior of that resource according to the type of rule. For example, a resource generated from a data directive in the policy itself might create an event that reads, "Resource X has been generated according to rule Y defined in policy Z. It may be updated if the source policy changes." And a similar type event may be created that slightly alters the message if the source of the new resource is a clone directive.

Describe alternatives you've considered

• Alter or create a new form of report at the Namespace and/or Cluster level which allows Namespace/Cluster administrators to see what actions were taken on existing resources (i.e., not validate results).
• Provide more descriptive information in the form of an annotation written to the mutated/generated resource. Hope someone finds and looks at it to answer their questions.
• Do not change any of the current behavior and rely upon humans to appropriately communicate what the expected results may be given various conditions.

Additional context
None

[chipzoller]

cc @valentinEmpy (because this is related to your past PR here)

[chipzoller]

Any comments on this, @JimBugwadia or @realshuting ?

[JimBugwadia]

Agreed, these are important enhancements we should implement.

[JimBugwadia]

This is partially address in #3741.

[chipzoller]

Some updates (as of Kyverno 1.10) here since the issue is rather old:

1. The source of all Kyverno-created Events begins with the string kyverno- and typically has the controller responsible.
2. Events are created for standard mutations (as well as mutation on existing resources) but the message could be improved.

Most important enhancements:

1. An event upon successful mutation should be created for the resource that was actually mutated and not just the policy responsible
2. An event upon successful generation should be created for both policy and resource.

[chipzoller]

Both items under "Most important enhancements" are completed as of Kyverno 1.11.