mutating webhook: audit log mutation existence and patch

[roycaihw]

Issue:

Mutating webhooks can silently mutate objects and cause API requests to fail validation (#62666). Currently when a request goes through admission chain, kube-apiserver doesn’t log / store trace of 1. which mutating webhooks mutated the request; 2. what was changed in the request.

Moreover, when a mutated request fails validation, the validation error message doesn’t necessarily reveal the actual cause (mutating webhooks) for the request to be invalid (#65569 (comment)). This makes it hard to debug a Kubernetes cluster, when (misconfigured) mutating webhooks silently invalid valid requests and put the cluster in undesired states.

What this PR does:

This PR records name of mutating webhooks in audit log to help answer the question "which mutating webhooks mutated the request".

Does this PR introduce a user-facing change?:

Audit events now log the existence and patch of mutating webhooks. 
* At Metadata audit level or higher, an annotation with key "mutation.webhook.admission.k8s.io/round_{round idx}_index_{order idx}" gets logged with JSON payload indicating a webhook gets invoked for given request and whether it mutated the object or not.
* At Request audit level or higher, an annotation with key "patch.webhook.admission.k8s.io/round_{round idx}_index_{order idx}" get logged with the JSON payload logging the patch sent by a webhook for given request.

/sig api-machinery

[fedebongio]

/cc @tallclair
could you take a look to this one please?

[roycaihw]

/cc @caesarxuchao

I'm adding an e2e test use integration test instead

[roycaihw]

/retest

[sttts]

Is this blocked on anything?

[sttts]

staging/src/k8s.io/apiserver/pkg/audit/request.go:230:28: "presist" is a misspelling of "persist"

[liggitt]

/hold

sorry for the hold, json construction needs a slight rework so we don't build potentially unsafe json

[liggitt]

lgtm, go ahead and squash fixup commits

[roycaihw]

squashed. Thanks for the review

[roycaihw]

/retest

[liggitt]

/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: liggitt, roycaihw, sttts

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• staging/src/k8s.io/apiserver/OWNERS [liggitt,sttts]
• test/OWNERS [liggitt,sttts]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[roycaihw]

/hold cancel

[roycaihw]

/retest

[roycaihw]

/retest

[fejta-bot]

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

[k8s-ci-robot]

@roycaihw: The following test failed, say /retest to rerun them all:

Test name	Commit	Details	Rerun command
pull-kubernetes-conformance-kind-ipv6	98ad20c	link	/test pull-kubernetes-conformance-kind-ipv6

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[tedyu]

@roycaihw
If you think the above two comments make sense, I can send out a PR.

Thanks

[roycaihw]

/area admission-control