Enable audit logging for all of our test environments

[ankeesler]

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

• @enj and I were debugging a mysteriously deleted Secret, and we had a really hard time figuring out why it was getting deleted.
• We enabled audit logging, and immediately discovered what entity was deleting the Secret and we were able to figure out our bug.
• More generally: it would be helpful when debugging test environments to have an audit log to help us understand what is going on.

Describe the solution you'd like
A clear and concise description of what you want to happen.

• Enable kube-apiserver audit logs in our test environments (i.e., our test kind clusters).
• We can write this audit log to a file inside of the kind docker container.

Describe alternatives you've considered

• None.

Are you considering submitting a PR for this feature?

• How will this project improvement be tested?
• Manually checking that audit logs are being populated after this fix goes in.
• How does this change the current architecture?
• It doesn't change our source code architecture, as it is a test change.
• It will fill up our kind cluster disks more quickly, but these disks are ephemeral as they are inside of the kind container.
• How will this change be backwards compatible?
• Yes - this is a purely additive test change.
• How will this feature be documented?
• Perhaps we should have some sort of "how to debug test PR test failures" section in our CONTRIBUTING.md?

Additional context
Here is what @enj and I did to enable audit logs in one of our kind clusters.

1. SSH into the VM on which our test kind cluster was running.
2. Exec into the kind container.
3. cd /etc/kubernetes
4. Create an audit-policy.yaml file, something like the below.

apiVersion: audit.k8s.io/v1beta1
kind: Policy
metadata:
  name: Default
# Don't generate audit events for all requests in RequestReceived stage.
omitStages:
- "RequestReceived"
rules:
# Don't log requests for events
- level: None
  resources:
  - group: ""
    resources: ["events"]
# Don't log authenticated requests to certain non-resource URL paths.
- level: None
  userGroups: ["system:authenticated", "system:unauthenticated"]
  nonResourceURLs:
  - "/api*" # Wildcard matching.
  - "/version"
  - "/healthz"
  - "/readyz"
# A catch-all rule to log all other requests at the Metadata level.
- level: Metadata
  # Long-running requests like watches that fall under this rule will not
  # generate an audit event in RequestReceived.
  omitStages:
  - "RequestReceived"

5. Add the --audit-policy-file=/etc/kubernetes/audit-policy.yaml flag to the manifests/kube-apiserver.yaml command array (surely there is a way in kind to do this).
6. Add the --audit-log-path=/var/log/kube-audit.log flag to the manifests/kube-apiserver.yaml command array (surely there is a way in kind to do this).
7. Add volumeMounts and volumes for those files (surely there is a way in kind to do this).

   volumeMounts:
    - mountPath: /var/log
      name: log
    - mountPath: /etc/kubernetes/audit-policy.yaml
      name: audit
      readOnly: true
...

  volumes:
  - hostPath:
      path: /var/log
      type: DirectoryOrCreate
    name: log
  - hostPath:
      path: /etc/kubernetes/audit-policy.yaml
      type: File
    name: audit

[enj]

This is implemented now in all Kind clusters used in CI (the KAS pod logs collected via export-cluster-diagnostics contain the audit logs as well). Leaving this issue open to track future enhancements to Kind clusters running on developer machines as well as non-Kind based CI environments such as EKS.

[mattmoyer]

I think we can consider this issue closed now that we have coverage on Kind clusters. I don't think we'll ever get to 100% coverage with this capability, and this meets most of our needs.