СRI image pulling with progress notification

[byako]

Enhancement Description

• One-line enhancement description (can be used as a release note): СRI image pulling with progress notification
• Kubernetes Enhancement Proposal: KEP-3542: СRI image pulling with progress notification #3547
• Discussion Link: Sep 27th, 2022 sig-node weekly meeting
• Primary contact (assignee): @byako
• Responsible SIGs: sig-node
• Enhancement target (which target equals to which milestone):
  • Alpha release target (x.y): 1.30
  • Beta release target (x.y): 1.31
  • Stable release target (x.y): 1.32
• Alpha
  • KEP (k/enhancements) update PR(s): KEP-3542: СRI image pulling with progress notification
  • Code (k/k) update PR(s): RFC: CRI PullImageWithProgress implementation PoC kubernetes#118326
  • Docs (k/website) update PR(s):

Please keep this description up to date. This will help the Enhancement Team to track the evolution of the enhancement efficiently.

[byako]

/sig node

[k8s-ci-robot]

@byako: The label(s) sig/sig-node cannot be applied, because the repository doesn't have them.

In response to this:

/sig sig-node

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[byako]

/sig node

[marosset]

/milestone v1.26
/label lead-opted-in
(I'm doing this on behalf of @ruiwen-zhao / SIG-node)

[rhockenbury]

/stage alpha
/label tracked/yes

[derekwaynecarr]

/remove-label lead-opted-in

This design needs more time in SIG Node and would be reviewed during 1.26 for evaluation in 1.27.

[rhockenbury]

/label tracked/no
/remove-label tracked/yes
/milestone clear

[byako]

The design is proposed now in KEP PR.

[rhockenbury]

Just want to clarify - in #3547, it looks like the milestones are targeting v1.26. Based on a prior comment from @derekwaynecarr, we have this enhancement as Removed from Milestone. Let me know if you do in fact want to opt in for the v1.26 cycle.

[byako]

Yes, @rhockenbury, that is the intention, if the proposal is approved or agreed for its details to be ironed out during v1.26 cycle - we'd like to opt this into v1.26 cycle.

[rhockenbury]

I can opt it in but it's going to be pretty tight to meet the requirements for enhancements freeze since freeze will happen in a few hours.

/label lead-opted-in
/label tracked/yes
/remove-label tracked/no

Just checking in as we approach enhancements freeze on 18:00 PDT on Thursday 6th October 2022.

This enhancement is targeting for stage alpha for 1.26 (correct me, if otherwise)

Here's where this enhancement currently stands:

• KEP readme using the latest template has been merged into the k/enhancements repo.
• KEP status is marked as implementable for latest-milestone: 1.26
• KEP readme has a updated detailed test plan section filled out
• KEP readme has up to date graduation criteria
• KEP has a production readiness review that has been completed and merged into k/enhancements.

For this KEP, we would need to update the following:

• Add the PRR review file and get a PRR review
• Merge up the open PR KEP-3542: СRI image pulling with progress notification #3547

The status of this enhancement is marked as at risk.

[rhockenbury]

Hello 👋, 1.26 Enhancements Lead here.

Unfortunately, this enhancement did not meet requirements for enhancements freeze.

If you still wish to progress this enhancement in v1.26, please file an exception request. Thanks!

/milestone clear
/label tracked/no
/remove-label tracked/yes
/remove-label lead-opted-in

[saschagrunert]

@byako exactly. Let's also check with @kubernetes/sig-node-feature-requests that it's tracked for 1.29. 👍

[byako]

At least it was present in tracking project https://github.com/orgs/kubernetes/projects/161

[saschagrunert]

Yes, but SIG Node leads have to opt-in as well.

[SergeyKanzhelev]

/label lead-opted-in
/milestone v1.29

@byako @saschagrunert who should be marked as a primary contact for this KEP?

[byako]

@SergeyKanzhelev I'm fine with being a primary contact, I've some bandwidth for this this quarter.

[salehsedghpour]

Hello @byako 👋, Enhancements team here.

Just checking in as we approach enhancements freeze on Friday, 6th October 2023.

This enhancement is targeting for stage alpha for 1.29 (correct me, if otherwise)

Here's where this enhancement currently stands:

• KEP readme using the latest template has been merged into the k/enhancements repo.
• KEP status is marked as implementable for latest-milestone: 1.29.
• KEP has a production readiness review that has been completed and merged into k/enhancements. (For more information on the PRR process, check here).

For this KEP, we would just need to update the following:

• The latest read me template has more items in production readiness review questionnaire that need to be addressed.
• The status should be marked as implementable in the kep.yaml file.
• Ensure that the PR including the production readiness review has been reviewed and merged into k/enhancements.

The status of this enhancement is marked as at risk for enhancement freeze. Please keep the issue description up-to-date with appropriate stages as well. Thank you!

[byako]

Changed status to implementable, waiting for sig-node approvers to give an approval for KEP PR.

[salehsedghpour]

Hi @byako , checking in once more as we approach the 1.29 enhancement freeze deadline on 01:00 UTC, Friday, 6th October, 2023. The status of this enhancement is marked as at risk. It looks like #3547 will address most of the requirements.

Let me know if I missed anything. Thanks!

[byako]

That's right, I've tried to address everything there was to address. Let's see if I can get anyone to approve anything.

[npolshakova]

Hello 👋, 1.29 Enhancements Lead here.
Unfortunately, this enhancement did not meet requirements for v1.29 enhancements freeze.
Feel free to file an exception to add this back to the release tracking process. Thanks!

/milestone clear

[drigz]

@byako I'm not sure whether I can help, but is there a particular person or group that you're waiting for review or approval from?

We're running Kubernetes in edge environments with slow downlinks, sometimes as low as 10 Mbps, so this would be really valuable feature and much more friendly than my current hack (watch ls -s /var/lib/containerd/io.containerd.content.v1.content/ingest/). Thank you for getting it to this point!

[byako]

@drigz, I'm happy to see that someone else is interested in this feature!
Long story short - this did not make is into 1.28, 1.29 either because of a bad luck or miscommunication, or both. See KEP PR #3547

I'll try to make this happen in 1.30, which starts about now.

The KEP itself consists of two parts: the CRI protocol part and Kubelet implementation part. What needs to be done still is Kubelet implementation part of this KEP, it has to be re-designed. SIG-scalability had concerns about impact of this feature at scale, and there were alternative implementation suggestions. The CRI protocol change appears to be fine, but I failed to secure the approval label, that seems to be just a formality.

In last cycle we agreed to remove the Kubelet implementation part from the KEP because there was too little time left to iron it out, but that didn't help either. Now that the new cycle is upon us, I think we can try to come up with new Kubelet implementation suggestion and hopefully both of them will be approved. If not - we can then push harder this time to get at least CRI change in 1.30.

I'm not sure how much time I will have in 1.30 cycle for this KEP, but there will be some. We'll see at next SIG-node meeting if I'm still driving this KEP or if someone else has more time for it than I do.

[drigz]

Thank you for the explanation! The scalability question does sound tricky, as a cluster administrator I can understand Wojciech's point that it's hard to stay abreast of new features and evaluate in advance whether they'll cause problems. I can also imagine a worst case where some highly-replicated pods are updated to new containers, but the registry is overloaded and, say, trickles 1 byte / second to every node, which could generate # nodes * # pods events every minute.

Did you look into whether there is any backpressure mechanism that could protect the apiserver in a case like this? Our clusters are small so I'm not familiar with these techniques, but API Priority & Fairness seems like it could enforce a cluster-level limit on the rate of progress events. Maybe that would address the question about a protection mechanism. However, I don't know where this traffic would belong - the default levels seem focused on requests that are important for normal cluster operation, rather than informative log events like this, which are primarily of interest to humans observing the cluster. Maybe we need a new node-low priority level?

Please ignore this question if it's an unhelpful tangent!

[byako]

I have not checked any protection measures available yet, but I'll have a look at Priority and Fairness doc, thank you.
If I'm not mistaken, current suggestion was to only publish events when something has subscribed to it. Details are to be defined.

[salehsedghpour]

/remove-label lead-opted-in

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[jon-nfc]

+1

[tonybart1337]

/remove-lifecycle rotten

[BloodyIron]

I for one really feel blind not knowing the progress of an image pull. It really would be useful to have some insights into the speed being pulled (so I can also troubleshoot network related problems), some form of percentage progress, and maybe an estimate of completion.