Synchronous & unbatched audit log writes

[tallclair]

What this PR does / why we need it:
When enabling buffered audit log file writes to reduce latency under high load, we shouldn't be batching the writes, as the large data write can have an inverse (though unpredictable) impact. Additionally, batched audit log writes should not be done asynchronously, as this just creates lock contention on the log writer.

This is a clean-ed up version of #61217

Which issue(s) this PR fixes
Fixes #61932

Release note:

Defaults for file audit logging backend in batch mode changed:
- Logs are written 1 at a time (no batching)
- Only a single writer process (lock contention)

/sig auth
/priority important-soon
/kind bug
/milestone v1.12

[tallclair]

Fixed broken tests.

[tallclair]

This could be considered a breaking change as it changes some default flag values. However, I think we should consider it a bug fix since those values didn't make any sense. I regret that we even exposed those options as flags, but we're stuck with them now.

[liggitt]

staging/src/k8s.io/apiserver/pkg/server/options/audit.go:48:16: "paramaters" is a misspelling of "parameters"

thanks, bots

[CaoShuFeng]

closedTimeCh is not used any more, I think.

[tallclair]

Done. Also inlined closedStopCh.

[x13n]

Copying my question from #61932:

I may be missing some context, but what's the point of having a synchronous buffer? Why is it better than having no buffer at all?

[tallclair]

Copying my response from #61932 :)

The buffer is still async, but calling the delagate backend can be synchronous for the buffer worker. More specifically, this is the old behavior, with each line being a separate go-routine

1. request handler: queue audit event in buffer
2. buffer worker: unqueue audit events, batch them, pass batch to delegate go-routine
3. delegate go-routine: process batched events

This makes sense for the webhook backend with a different routine to handle each audit request, but it doesn't make sense for the log backend where the different writing routines just create lock contention. With the new PR & AsyncDelegate = false, you get:

1. request handler: queue audit event in buffer
2. buffer worker: unqueue audit events, batch them, call delegate to process the events (in the same routine)

Make sense?

[tallclair]

/retest

[x13n]

Thanks! Looks good, please just squash the commits.

[tallclair]

Squashed.

/assign @sttts
for approval

[x13n]

/lgtm

[sttts]

/retest
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: sttts, tallclair, x13n

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• cmd/kube-apiserver/OWNERS [sttts]
• staging/src/k8s.io/apiserver/OWNERS [sttts]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[fejta-bot]

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel comment for consistent failures.

[k8s-github-robot]

/test all [submit-queue is verifying that this PR is safe to merge]

[k8s-github-robot]

Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions here.