Implement per-request watch cache opt-out

[serathius]

What would you like to be added?

Part of kubernetes/enhancements#2340

From https://github.com/kubernetes/enhancements/blob/master/keps/sig-api-machinery/2340-Consistent-reads-from-cache/README.md#what-if-the-watch-cache-is-stale

Per request override should allow user to compare request results without impacting other requests or requiring to redeploy whole cluster. The exact details of override API will be clarified during API review. In healthy situation, using this override should not cause any impact on the response, however it might increase resource usage. In our tests cpu load could increase tenfold. To prevent abuse access to it should be limited to users with cluster-admin role, rejecting the request otherwise.

This issue will require designing an API and going through API review. Definitely not a beginner issue

Up to now we have discussed using HTTP headers or adding another option for ListOptions struct. Some of my thoughts:

• Don't want to commit to stable API, as we don't want to design a full request debugging API.
• It changes how requests are internally handled, but not API semantics. It doesn't make sense to be part of ListOptions that is passed to both watch cache and storage.
• On the other hand we want to avoid a obscure flag that make API surface hard to track.
• Should be included in API request cost calculations

Why is this needed?

A way to debug watch cache

[serathius]

/sig api-machinery
/triage accepted

[serathius]

/cc @deads2k @liggitt
for suggestions about the API design.

[serathius]

/assign

[serathius]

The PR #123472 sparked a lengthy discussion about the proposed design. We need to reach consensus among all stakeholders before implementing it @deads2k @liggitt @wojtek-t

@p0lyn0mial maybe you could help?