VolumeSource: OCI Artifact and/or Image

[sallyom]

Enhancement Description

• One-line enhancement description (can be used as a release note): VolumeSource: OCI Artifact or OCI Image
• Kubernetes Enhancement Proposal: https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/4639-oci-volume-source
• Discussion Link:
• Primary contact (assignee): sgrunert@redhat.com, somalley@redhat.com
• Responsible SIGs: sig-node, sig-storage
• Enhancement target (which target equals to which milestone):
  • Alpha release target (x.y): 1.31
  • Beta release target (x.y): 1.33
  • Stable release target (x.y):
• Alpha
  • KEP (k/enhancements) update PR(s):
    • KEP-4639: adding OCI VolumeSource #4642
  • Code (k/k) update PR(s):
    • [KEP-4639] Add OCI VolumeSource CRI API kubernetes#125659
      • [KEP-4639] Update CRI API and workflow #4751
    • [KEP-4639] Add ImageVolumeSource API kubernetes#125660
    • [KEP-4639] Add ImageVolumeSource implementation kubernetes#125663
    • Nice to have:
      • [KEP-4639] Mention that fsGroupChangePolicy has no effect kubernetes#126281
      • Fix runtime panic in imagevolume CanSupport method kubernetes#126323
      • [KEP-4639] Add ImageVolumeSource node e2e tests kubernetes#126220
  • Out of tree update PR(s):
    • Add OCI Volume Source support to crictl [create|run] --with-pull kubernetes-sigs/cri-tools#1464
    • Add OCI/Image Volume Source support cri-o/cri-o#8317
    • Make ImageVolume garbage collection work cri-o/cri-o#8408
    • CRI-O: add pull-kubernetes-node-crio-cgrpv2-imagevolume-e2e test job test-infra#33071
    • Fix imagevolume test focus test-infra#33088
  • Docs (k/website) update PR(s):
    • Document OCI volume sources / KEP-4639 website#46946
    • Blog post about OCI volume sources / KEP-4639 website#46960
• Beta
  • KEP (k/enhancements) update PR(s):
    • KEP-4639: update to most recent implementation state and graduate to beta #4897
  • Code (k/k) update PR(s):
    • [KEP-4639] Graduate image volume sources to beta kubernetes#130135
    • Enable ImageVolume tests for pull-kubernetes-node-e2e-containerd-alpha-features test-infra#34398
    • Optional: [KEP-4639] Image volume subpath e2e kubernetes#130681
  • Docs (k/website) update(s):
    • [KEP-4639] Graduate image volume source to beta website#49936
    • [KEP-4639] Feature blog 1.33: Image volumes graduate to beta website#49984
  • Out of tree update PR(s):
    • Add image volume subpath support cri-o/cri-o#9050
    • Update cri-api to latest commit kubernetes-sigs/cri-tools#1793
    • Bump CRI-O cgroups v2 jobs to bdcf72 test-infra#34547
    • Fix CNI setup for CRI-O test-infra#34549

Please keep this description up to date. This will help the Enhancement Team to track the evolution of the enhancement efficiently.

[sallyom]

I've had informal discussions about this - there's enough interest IMO to open a KEP & I will present this issue at upcoming sig-node, sig-storage mtgs with a KEP draft

/sig node
/sig storage

[xing-yang]

Can you use the Volume Populator? It allows you to create a PVC from an extenal data source. https://github.com/kubernetes/enhancements/tree/master/keps/sig-storage/1495-volume-populators

[saschagrunert]

Happy to support here from a SIG node perspective.

cc @kubernetes/sig-node-proposals

[mikebrow]

+1 happy to help from the SIG-Node/CRI and OCI image/distribution spec perspectives..

[SergeyKanzhelev]

/label lead-opted-in
/milestone v1.31

as discussed at SIG Node meeting this week, we will try and see if this can make it to 1.31

[SergeyKanzhelev]

/stage alpha

[rhuss]

For reference, in KServe a workaround for directly accessing files within an OCI image is currently implemented and available via a sidecar approach ("modelcar") by leveraging root FS system access via the /proc filesystem when shareProcessNamespace: true is set on the Pod. You can find details in the KServe documentation and in the Design Document. It actually implements the desired behavior with current means, but of course is more or less just a workaround for the OCI volume type (as discussed here and raised already a long time ago in kubernetes/kubernetes#831).

So KServe would be more than happy to leverage such a volume type, and we are happy to support any efforts in this direction.

[sreeram-venkitesh]

/stage alpha

[sreeram-venkitesh]

Hello @sallyom 👋, v1.31 Enhancements team here.

Just checking in as we approach enhancements freeze on 02:00 UTC Friday 14th June 2024 / 19:00 PDT Thursday 13th June 2024.

This enhancement is targeting for stage alpha for v1.31 (correct me, if otherwise)

Here's where this enhancement currently stands:

• KEP readme using the latest template has been merged into the k/enhancements repo.
• KEP status is marked as implementable for latest-milestone: v1.31. KEPs targeting stable will need to be marked as implemented after code PRs are merged and the feature gates are removed.
• KEP readme has up-to-date graduation criteria
• KEP has a production readiness review that has been completed and merged into k/enhancements. (For more information on the PRR process, check here). If your production readiness review is not completed yet, please make sure to fill the production readiness questionnaire in your KEP by the PRR Freeze deadline (6th June) so that the PRR team has enough time to review your KEP before the enhancements freeze.

For this KEP, most of the above items are taken care of in #4642. We'd need to do the following:

• Update the status from provisional to implementable in the kep.yaml file here
• Create a prod-readiness yaml file as shown here.
• Update the graduation criteria in the KEP readme file.
• Make sure that the PRR questionnaire is filled.

The status of this enhancement is marked as At risk for enhancements freeze. Once the above tasks are done, I can mark it as tracked.

If you anticipate missing enhancements freeze, you can file an exception request in advance. Let me know if you have any questions! Thank you!

[sreeram-venkitesh]

@sallyom Pinging once again as a slight reminder that we're approaching the enhancements freeze deadline on 14th June, this Friday!

[dipesh-rawat]

Hi @sallyom @SergeyKanzhelev 👋, 1.31 Enhancements team here,

Just a quick friendly reminder as we approach the enhancements freeze in few hours, at 02:00 UTC Friday 14th June 2024 / 19:00 PDT Thursday 13th June 2024.

The current status of this enhancement is marked as at risk for enhancement freeze. There are a few requirements mentioned in the comment #4639 (comment) that are addressed as part of PR #4642 which still needs to be merged.

If you anticipate missing enhancements freeze, you can file an exception request in advance. Thank you!

[dipesh-rawat]

Hello @sallyom @SergeyKanzhelev 👋, 1.31 Enhancements team here.

Unfortunately, this enhancement did not meet requirements for enhancements freeze.

If you still wish to progress this enhancement in v1.31, please file an exception request as soon as possible, within three days. If you have any questions, you can reach out in the #release-enhancements channel on Slack and we'll be happy to help. Thanks!

[sreeram-venkitesh]

/milestone clear

[bitoku]

/assign

[mikebrow]

pushing container images to the container registry with compression off (example: --disable-compression) is a client tooling feature that may or may not be available containers/buildah#3904

+1 to sasha's comment -- sync pod context means before the pod is run, kubelet requests the images to be pulled locally normally this is done with unpack,... containerd's 1.7/2.0 config.toml entry for the default timeout is image_pull_progress_timeout = '5m0s'

[chenditc]

Thanks for sharing valuable insight!

We were using kubernetes version 1.30.4 + containerd 1.6.26 + pulling image from Azure acr. The image is build using docker with LLM model weight from hugging face: https://huggingface.co/microsoft/Phi-3.5-vision-instruct/tree/main.

When a pod pulling the container image (20G+), if it timeout or failed due to various reason, the danling image layer won't get garbage collect right away, leaving some orphant tar file. The retry will worse the scenario, as there were less ephemeral disk to use. The size of the ephemeral disk is just enough to do a "happy day" download and untar, so it's very fragile to any failure.

@saschagrunert I don't have a stable reproducer yet, as this issue seems happen more frequently in environment with weak network or lower ephemeral disk.

@sudo-bmitch @mikebrow Thanks for sharing the option to disable compression, I will try disable the image compression and see if that helps.

One additional concern is the quota management for ephemeral disk. As the volume image is pull using similar mechanism as container image, it will use some ephemeral disk, but it doesn't have a clear isolation boundary like pvc, if volume image pull encounter error, the ephemeral disk pressure might impact all pods on that node.

[fykaa]

Hello @sallyom @saschagrunert 👋,

The v1.33 Enhancements team here again!

With all the implementation (code-related) PRs merged as per the issue description:

• [KEP-4639] Graduate image volume sources to beta kubernetes#130135
• Enable ImageVolume tests for pull-kubernetes-node-e2e-containerd-alpha-features test-infra#34398

I noticed that kubernetes/kubernetes#130681 is currently marked as optional. Could you confirm whether it’s required for this release, or if we can consider the enhancement complete? If not, we’ll go ahead and mark it as Tracked for Code Freeze for v1.33.

Additionally, are there any other PRs in k/k that we should be tracking for this KEP to ensure an accurate status update?

Thanks! 🚀

[saschagrunert]

@fykaa code for this enhancement is complete for now, I don't think that the optional e2e test will land because we have no runtime support yet.

[Barakmor1]

Hi,

Currently in the containerStatuses field of the pod status, the imageID field includes the Digest suffix for each container, like this:

imageID: ...@sha256:f8110bad01b7e1a08b961613f38d104f4f8e05e66cea12fbda17e2ab206fc857

This is quite useful for detecting when an image has changed, especially for those using the latest tag. Is there any plan to include similar information for the imageVolume in the pod's status?

[dipesh-rawat]

@saschagrunert Thanks, for the update and confirming that all required changes are merged (here). We can mark this as tracked for code freeze. Also, please let us know if anything changes before the freeze or if there are any other PRs in k/k we should track for this KEP to keep the status accurate.

This enhancement is now marked as tracked for code freeze for the v1.33 Code Freeze!

[jenshu]

Hi @saschagrunert 👋, 1.34 Enhancements Lead here.

I am closing the v1.33 milestone now.

If you'd like to work on this enhancement in v1.34, please have the SIG lead opt-in by adding the lead-opted-in label, which ensures it gets added to the tracking board. Also, please set the milestone to v1.34 using /milestone v1.34.

Thanks!

/remove-label lead-opted-in
/remove-label tracked/yes
/milestone clear

[saschagrunert]

@jenshu thanks for reaching out! There is no graduation planned for v1.34. 👍

[iholder101]

Hey @saschagrunert!

@jenshu thanks for reaching out! There is no graduation planned for v1.34. 👍

Thanks for the heads-up.
Do you mind sharing why graduation is not planned, or what are the significant blockers / remaining issues? Thank you!

In addition, regarding the comment above,

Currently in the containerStatuses field of the pod status, the imageID field includes the Digest suffix for each container, like this:

imageID: ...@sha256:f8110bad01b7e1a08b961613f38d104f4f8e05e66cea12fbda17e2ab206fc857

This is quite useful for detecting when an image has changed, especially for those using the latest tag.

Do you think this is something we can plan for GA?
Would it help id I'd open an issue, or create a KEP update PR to add this to GA requirements? WDYT?

Thanks!

[theory]

Where would be the appropriate venue to request that this feature support adding image volumes without restarting a Pod? The CloudNativePG project is looking at using it to deploy Postgres extensions via OCI images, and most don't require a cluster restart, just the insertion of the files. Would be nice not to have to restart the database.