Support image pull per runtime class

[kiashok]

Enhancement Description

• One-line enhancement description (can be used as a release note): Add support to containerd/kubelet/CRI to support image pull per runtime class
• Kubernetes Enhancement Proposal: https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/4216-image-pull-per-runtime-class/
• Discussion Link: https://kubernetes.slack.com/archives/C0BP8PW9G/p1693000200223669. Also discussed in SIG-Windows, WG for Image GC and with @mikebrow
• Primary contact (assignee): @kiashok
• Responsible SIGs: SIG-Nodes, SIG-Windows
• Enhancement target (which target equals to which milestone):
  • Alpha release target (x.y): 1.29
  • Beta release target (x.y):
  • Stable release target (x.y):
• Alpha
  • KEP (k/enhancements) update PR(s): 4216: Image pull per runtime class KEP #4217
  • Code (k/k) update PR(s):
    • Add runtime handler field to ImageSpec struct kubernetes#121121
    • KEP 4216: Add changes for alpha version under RuntimeClassInImageCriApi feature gate kubernetes#121456
  • Docs (k/website) update PR(s): Add docs for KEP 4216: Image pull per runtime class website#43541
• Beta
  • KEP (k/enhancements) update PR(s):
  • Code (k/k) update PR(s):
  • Docs (k/website) update(s):

Please keep this description up to date. This will help the Enhancement Team to track the evolution of the enhancement efficiently.

[kiashok]

cc @jsturtevant @mikebrow @marosset @haircommander @SergeyKanzhelev @aravindhp @dmcgowan @kevpar @dcantah

[aravindhp]

/sig windows
/sig node

[jsturtevant]

/milestone v1.29

[npolshakova]

Hello @kiashok, 1.29 Enhancements team here! Is this enhancement targeting 1.29? If it is, can you follow the instructions here to opt in the enhancements and make sure the lead-opted-in label is set so it can get added to the tracking board? Thanks!

[aravindhp]

/label lead-opted-in

[sanchita-07]

Hello @kiashok 👋, Enhancements team here.

Just checking in as we approach enhancements freeze on Friday, 6th October 2023.

This enhancement is targeting for stage alpha for 1.29 (correct me, if otherwise)

Here's where this enhancement currently stands:

• KEP readme using the latest template has been merged into the k/enhancements repo.
• KEP status is marked as implementable for latest-milestone: 1.29.
• KEP readme has up-to-date graduation criteria
• KEP has a production readiness review that has been completed and merged into k/enhancements. (For more information on the PRR process, check here).

For this KEP, we would just need to add/update the following:

• The latest readme template has more items in production readiness review questionnaire that need to be addressed.
• The status should be marked as implementable in the kep.yaml file and the latest-milestone should be set to v1.29.
• Ensure that the PR including the production readiness review has been reviewed and merged into k/enhancements.

The status of this enhancement is marked as at risk for enhancement freeze. Please keep the issue description up-to-date with appropriate stages as well. Thank you!

[kiashok]

Hello @kiashok 👋, Enhancements team here.

Just checking in as we approach enhancements freeze on Friday, 6th October 2023.

This enhancement is targeting for stage alpha for 1.29 (correct me, if otherwise)

Here's where this enhancement currently stands:

• KEP readme using the latest template has been merged into the k/enhancements repo.
• KEP status is marked as implemented for latest-milestone: 1.29.
• KEP readme has up-to-date graduation criteria
• KEP has a production readiness review that has been completed and merged into k/enhancements. (For more information on the PRR process, check here).

For this KEP, we would just need to add/update the following:

• The latest readme template has more items in production readiness review questionnaire that need to be addressed.
• The status should be marked as implementable in the kep.yaml file and the latest-milestone should be set to v1.29.
• Ensure that the PR including the production readiness review has been reviewed and merged into k/enhancements.

The status of this enhancement is marked as at risk for enhancement freeze. Please keep the issue description up-to-date with appropriate stages as well. Thank you!

Hi @sanchita-07 , thanks a lot for the note. I have updated the KEP PR with your suggestions and I also requested for PRR review from the slack channel. Please let me know if there is anything more I need to do. Thanks!

[kiashok]

lead-opted-in

Thanks @npolshakova . @marosset reviewed the KEP today and set this label.

[sanchita-07]

Hi @sanchita-07 , thanks a lot for the note. I have updated the KEP PR with your suggestions and I also requested for PRR review from the slack channel. Please let me know if there is anything more I need to do. Thanks!

We just need the PR #4217 to be merged, then we are good to go to mark the status as tracked for enhancement freeze. 😃

[kiashok]

Hi @sanchita-07 , thanks a lot for the note. I have updated the KEP PR with your suggestions and I also requested for PRR review from the slack channel. Please let me know if there is anything more I need to do. Thanks!

We just need the PR #4217 to be merged, then we are good to go to mark the status as tracked for enhancement freeze. 😃

@sanchita-07 could you please help assign a PRR reviewer? Thanks.

[sanchita-07]

It looks like @johnbelamaric has been assigned and looking over it.

[npolshakova]

Hello @kiashok 👋, Enhancement Lead here. This enhancement is tracked for enhancement freeze, but there is some additional feedback on the KEP PR that was already merged: #4217 (review). Please address the feedback and open a follow-up PR as soon as you can. Thanks!

[kiashok]

Hello @kiashok 👋, Enhancement Lead here. This enhancement is tracked for enhancement freeze, but there is some additional feedback on the KEP PR that was already merged: #4217 (review). Please address the feedback and open a follow-up PR as soon as you can. Thanks!

Hi @npolshakova , I am addressing the comments now. The comments are only around restructuring some sentences and have no impact on the design that the KEP is talking about. It should not affect the KEP for 1.29

cc @jsturtevant @aravindhp @mrunalp @marosset @johnbelamaric @mikebrow

[kiashok]

Hello @kiashok 👋, Enhancement Lead here. This enhancement is tracked for enhancement freeze, but there is some additional feedback on the KEP PR that was already merged: #4217 (review). Please address the feedback and open a follow-up PR as soon as you can. Thanks!

Hi @npolshakova , I am addressing the comments now. The comments are only around restructuring some sentences and have no impact on the design that the KEP is talking about. It should not affect the KEP for 1.29

cc @jsturtevant @aravindhp @mrunalp @marosset @johnbelamaric @mikebrow

FYI: opened PR #4283 to address the changes

[katcosgrove]

Hey there @kiashok! 👋, v1.29 Docs Lead here.
Does this enhancement work planned for v1.29 require any new docs or modification to existing docs?
If so, please follows the steps here to open a PR against dev-1.29 branch in the k/website repo. This PR can be just a placeholder at this time and must be created before Thursday, 19 October 2023.
Also, take a look at Documenting for a release to get yourself familiarize with the docs requirement for the release.
Thank you!

[kiashok]

Hey there @kiashok! 👋, v1.29 Docs Lead here. Does this enhancement work planned for v1.29 require any new docs or modification to existing docs? If so, please follows the steps here to open a PR against dev-1.29 branch in the k/website repo. This PR can be just a placeholder at this time and must be created before Thursday, 19 October 2023. Also, take a look at Documenting for a release to get yourself familiarize with the docs requirement for the release. Thank you!

I'll take a look at this tomorrow and share an update. Thanks! Also what is the code freeze date for alpha changes of the KEP?

[kiashok]

Hi @kiashok, 👋 from the v1.29 Release Team-Communications! We would like to check if you have any plans to publish a blog for this KEP regarding new features, removals, and deprecations for this release.

If so, you need to open a PR placeholder in the website repository. The deadline will be on Tuesday 14th November 2023 (after the Docs deadline PR ready for review)

Here is the 1.29 calendar

@a-mccarthy I already opened one two weeks ago: #4216 (comment)

[kiashok]

Follow up discussion to be had in sig-node for beta version: kubernetes/kubernetes#121658

[a-mccarthy]

@a-mccarthy I already opened one two weeks ago: #4216 (comment)

@kiashok, that looks like a PR for docs changes. If you would like to also write a blog for this change, please create a separate placeholder PR by November 14th. Thanks!

[kiashok]

@a-mccarthy I already opened one two weeks ago: #4216 (comment)

@kiashok, that looks like a PR for docs changes. If you would like to also write a blog for this change, please create a separate placeholder PR by November 14th. Thanks!

kubernetes/website#43923 @a-mccarthy . I will have the actual changes go out early tomorrow

[salehsedghpour]

/remove-label lead-opted-in

[salehsedghpour]

Hello 👋 1.30 Enhancements Lead here,

I'm closing milestone 1.29 now,
If you wish to progress this enhancement in v1.30, please follow the instructions here to opt in the enhancement and make sure the lead-opted-in label is set so it can get added to the tracking board and finally add /milestone v1.30. Thanks!

/milestone clear

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[ruiwen-zhao]

containerd is adding the capability of pulling images with different snapshotters, based on the pod's runtime handler. So if kubelet requests containerd to pull image per runtime class, then containerd might pull them with different snapshotters.

And since each snapshotter keeps its own imageFs usage, this will cause trouble for kubelet's tracking of imageFs usage. More spefically, kubelet currently only reads the imagefs usage of containerd's global snapshotter ([1] [2]), so if we pull a lot of images thru a non-global snapshotter, the disk usage will be ignored.

I cannot think of any easy way to address this, mostly because kubelet's cri_stats_provider doesn't know which pods are using which snapshotter. Should we add this in to the scope of this KEP, to make sure kubelet either track imageFs usage per runtime class, or kubelet combine the imageFs of all snapshotters.

[1] https://github.com/containerd/containerd/blob/bbac058cf31359d7ca7b2a98f48bce086b817dcf/internal/cri/server/images/imagefs_info.go#L54

[2] https://github.com/kubernetes/kubernetes/blob/26923b91e8cbd2d409e5d177ddd509429b76cb35/pkg/kubelet/stats/cri_stats_provider.go#L401

[kiashok]

containerd is adding the capability of pulling images with different snapshotters, based on the pod's runtime handler. So if kubelet requests containerd to pull image per runtime class, then containerd might pull them with different snapshotters.

And since each snapshotter keeps its own imageFs usage, this will cause trouble for kubelet's tracking of imageFs usage. More spefically, kubelet currently only reads the imagefs usage of containerd's global snapshotter ([1] [2]), so if we pull a lot of images thru a non-global snapshotter, the disk usage will be ignored.

I cannot think of any easy way to address this, mostly because kubelet's cri_stats_provider doesn't know which pods are using which snapshotter. Should we add this in to the scope of this KEP, to make sure kubelet either track imageFs usage per runtime class, or kubelet combine the imageFs of all snapshotters.

[1] https://github.com/containerd/containerd/blob/bbac058cf31359d7ca7b2a98f48bce086b817dcf/internal/cri/server/images/imagefs_info.go#L54

[2] https://github.com/kubernetes/kubernetes/blob/26923b91e8cbd2d409e5d177ddd509429b76cb35/pkg/kubelet/stats/cri_stats_provider.go#L401

@ruiwen-zhao I think this is a problem that exists even today, so I don't think we need to block this KEP on that. We could try and address it for GA of this KEP/feature.

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[pacoxu]

/remove-lifecycle rotten
/reopen

It seems to be doable to promote this to Beta.

[k8s-ci-robot]

@pacoxu: Reopened this issue.

In response to this:

/remove-lifecycle rotten
/reopen

It seems to be doable to promote this to Beta.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[pacoxu]

@mikebrow @kiashok Are there any blockers to promote this to Beta?