KEP-2535: (Ensure Secret Pulled Images): add metrics, bump to beta in 1.34

[stlaz]

• One-line PR description: This PR looks into the metrics of the KEP and additionally bumps it to beta for 1.34.

• Issue link: Ensure secret pulled images #2535

• Other comments: The original KEP suggested histograms but I am not sure how I would wire those up. I think counts and gauges might do the job.

/cc enj liggitt mikebrow haircommander

[BenTheElder]

Did we consider splitting the cache directory into subdirectories based on a partial key?

This approach is common for on-disk content addressed caches to avoid having too many items in a single directory (see for example, git, ls -la .git/objects).

That might be good to change before locking in these details and having users depending on the layout to debug.

[aramase]

Also need to update https://github.com/kubernetes/enhancements/blob/master/keps/prod-readiness/sig-node/2535.yaml

[stlaz]

Did we consider splitting the cache directory into subdirectories based on a partial key?

We were briefly discussing this during design sessions but chose the flat design in the end. We shouldn't be hitting any directory limits.

[stlaz]

@jpbetz 👋
I chose you because you were doing PRR for alpha, please let me know in case you would like me to pick somebody else 🙏

[jpbetz]

/approve
For PRR

(Thanks @BenTheElder for the shadow review!)

[aramase]

/assign @haircommander @mikebrow

for SIG Node review

[haircommander]

/lgtm

[mikebrow]

see suggestions for clarification and additional useful metrics based on kubelet policy vs image pull policy

[mikebrow]

need some way to show the difference between a successful anonymous access request .. and having to try N different sets of credentials.... one image verification might take a large number of tries when a number of private sets of credentials are used on a large number of containers in a pod.

[stlaz]

I think that perhaps that may not matter as much for observability of this feature but rather for observability of image pulls, which today is not great - we don't seem to track anything besides how long an image pull took, and that is also only tracked in case of a successful pull..

If we're to track the number of pulls per image until success, I think we can do that even outside this KEP. Would you agree?

[mikebrow]

I agree this is a needed metric that exists without the ensure image kep. More specifically the number of attempted image pulls that fail due to incorrect credentials anonymous(pull 1) key ring entry 1(pull 2) private key 1(pull 3)

Failed credential looping on pull requests could cost more than actually pulling the images.

The other issue that could use a metric is cost of using :latest to connect and recheck manifests again and again...

[mikebrow]

/lgtm

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jpbetz, mikebrow, mrunalp, stlaz

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• keps/prod-readiness/OWNERS [jpbetz]
• keps/sig-node/OWNERS [mrunalp]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment