Fix admission metrics bucket sizes

[jpbetz]

When #72343 fixed admission metrics to use seconds instead of microseconds as the measured unit (which was totally my fault), the bucket sizes didn't get updated and so are off by 6 orders of magnitude. This fixes them.

/kind bug

Fix admission metrics histogram bucket sizes to cover 25ms to ~2.5 seconds.

/sig api-machinery
/cc @logicalhan @danielqsj @brancz @sttts @liggitt @cheftako

[fedebongio]

/assign @logicalhan

[liggitt]

/priority important-soon
/milestone v1.15

we would cherry-pick this into the 1.15 release even if it merged post 1.15

[logicalhan]

/lgtm

[logicalhan]

/retest

[logicalhan]

/cc @brancz

What do you think about these bucket sizes?

[logicalhan]

The current buckets generated by the expression underneath are: [0.025 0.0625 0.15625 0.390625 0.9765625] so the comment is actually still inaccurate.

[logicalhan]

If we want to stay within the constraints of five buckets then my suggestion is {0.025, 0.05, 0.25, 1.0, 5.0}. We should also update the comment to match the buckets.

[cheftako]

I realize trace threshold is configurable.
However we do currently have it configured at 100 and 500 milliseconds (and 10 seconds for watch).
If we set the start to .02 rather than .025 it would better align with the trace thresholds, which might be useful.

Not sure exponential bucket are great as they seem to be causing some confusion. Maybe something like {0.025, 0.1, 0.25, 0.5, 1, 5}

[jpbetz]

I realize trace threshold is configurable.
However we do currently have it configured at 100 and 500 milliseconds (and 10 seconds for watch).
If we set the start to .02 rather than .025 it would better align with the trace thresholds, which might be useful.

Not sure exponential bucket are great as they seem to be causing some confusion. Maybe something like {0.025, 0.1, 0.25, 0.5, 1, 5}

I like this idea. Since the admission webhook default (and max allowed) timeout is 30s, maybe:

{0.025, 0.1, 0.5, 1, 5, 30}

?

[cheftako]

/lgtm

[jpbetz]

Going with buckets: {.005, .025, 0.1, 0.5, 2.5}. This keeps out bucket count the same, but gives us a better fast bucket and aligns with the 100 and 500ms default trace thresholds.

[logicalhan]

/lgtm

Thanks Joe!

[cheftako]

/lgtm

[danielqsj]

/lgtm

[brancz]

/lgtm

[jpbetz]

@liggitt or @deads2k Could we get approval?

[logicalhan]

/retest

[jpbetz]

rebased

[liggitt]

/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jpbetz, liggitt

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• staging/src/k8s.io/apiserver/OWNERS [liggitt]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[fejta-bot]

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.