Add request body size metric for the write path.

[logicalhan]

We can use this metric in conjunction with apiserver_storage_objects to
determine if a LIST is likely to be near a timeout threshold.

What type of PR is this?

/kind cleanup

What this PR does / why we need it:

This metric exposes the request body sizes that have been accepted as being under the apiserver max request body threshold. This metric has two primary usecases:

1. alerting for when writes begin to approach the threshold in the apiserver
2. calculating the approximate size of a LIST request by the following function:
   • avg(apiserver_request_body_size{resource="a"}) * apiserver_storage_objects{resource="a"}

Which issue(s) this PR fixes:

Fixes #111893

Does this PR introduce a user-facing change?

NONE

[k8s-ci-robot]

Please note that we're already in Test Freeze for the release-1.25 branch. This means every merged PR will be automatically fast-forwarded via the periodic ci-fast-forward job to the release branch of the upcoming v1.25.0 release.

Fast forwards are scheduled to happen every 6 hours, whereas the most recent run was: Thu Aug 18 13:40:40 UTC 2022.

[logicalhan]

/retest

[apelisse]

There's the size of the request (the size of the body in the request), and the size of the object as we store in etcd. Differences comes from format used for the request can be different from the storage format (YAML vs Protobuf), but also yaml has some alias and anchor mechanisms that can significantly reduce the size of the object compared to its expanded form. Is this what we want to measure?

[logicalhan]

There's the size of the request (the size of the body in the request), and the size of the object as we store in etcd. Differences comes from format used for the request can be different from the storage format (YAML vs Protobuf), but also yaml has some alias and anchor mechanisms that can significantly reduce the size of the object compared to its expanded form. Is this what we want to measure?

One thing we want to measure is the size of the incoming request. And that's because we have a 3 meg limit and we want to know if our incoming payloads are getting close to that threshold.

The second thing we want is basically size of the outgoing object, which I actually do realize this does not accurately represent. The reason for approximating the size of the outgoing object is basically to have something as a proxy for outgoing object size so that we can do math and determine the estimate size of a full list request so that we can know if the total volume of objects that we have is potentially close to the timeout of a full list.

But actually, I think I need to make the buckets more granular for this.

[leilajal]

/triage accepted

[apelisse]

/lgtm
/approve
Thanks Han!

[logicalhan]

/assign @lavalamp

[logicalhan]

/retest

[lavalamp]

calculating the approximate size of a LIST request by the following function

You can't do that with this metric, request body and stored object sizes have very little to do with each other (imagine encoding differences & create / update sequences)

[lavalamp]

this looks like a change detector, is there a better way to verify what happened?

[logicalhan]

I mean, I want to test the exact metric output, so a change detector is actually kinda what I want..

[lavalamp]

The second thing we want is basically size of the outgoing object, which I actually do realize this does not accurately represent. The reason for approximating the size of the outgoing object is basically to have something as a proxy for outgoing object size so that we can do math and determine the estimate size of a full list request so that we can know if the total volume of objects that we have is potentially close to the timeout of a full list.

I think this metric doesn't get anywhere near close enough for this purpose. The way to do this is to update a metric on GET / LIST requests (divide by number of items returned)

[logicalhan]

I think this metric doesn't get anywhere near close enough for this purpose. The way to do this is to update a metric on GET / LIST requests (divide by number of items returned)

Discussed offline, I'll create another metric for this.

[lavalamp]

/approve

avg(apiserver_request_body_size{resource="a"}) * apiserver_storage_objects{resource="a"}

I think I'm not worried about people thinking this would work because it won't be obvious at all how to account for create/update/patch request differences.

The other use seems valid though.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: apelisse, lavalamp, logicalhan

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• staging/src/k8s.io/apiserver/OWNERS [lavalamp]
• vendor/OWNERS [lavalamp]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[logicalhan]

/retest