Kubelet limit of Parallel Image Pulls

[ruiwen-zhao]

Enhancement Description

• One-line enhancement description (can be used as a release note): Add a node-level limit to kubelet to limit the number of parallel image pulls
• Kubernetes Enhancement Proposal: https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/3673-kubelet-parallel-image-pull-limit/README.md
• Discussion Link:
• Primary contact (assignee): @pacoxu @ruiwen-zhao
• Responsible SIGs: SIG Node
• Enhancement target (which target equals to which milestone):
  • Alpha release target (x.y): 1.27
  • Beta release target (x.y): 1.32
  • Stable release target (x.y):
• Alpha
  • KEP (k/enhancements) update PR(s):
    • KEP-3673: Kubelet limit of Parallel Image Pulls #3713
  • Code (k/k) update PR(s):
    • Add MaxParallelImagePulls support kubernetes#115220
  • Docs (k/website) update PR(s):
    • Document update for MaxParallelImagePulls website#39732
• Beta
  • KEP (k/enhancements) update PR(s):
    • 1.28 KEP-3673: Promote Parallel Image Pull Limit to Beta #4036;
    • 1.29 update kubelet parallel image pull limit for v1.29 #4173
    • 1.30 postponed kep-3673 parallel image pull limit to v1.30 for beta #4452
    • 1.32 [cleanup] update KEP-3673 status: e2e added in v1.32 for beta #4938
  • Code (k/k) update PR(s):
    • 1.32 [node-e2e] add test cases for serialize and parallel image pulling kubernetes#121604
  • Docs (k/website) update(s): doc is merged a little earlier. E2E was added in v1.32.
    • Promote parallel image pulls to beta website#42061 merged in v1.28, and reverted).
    • update maxParallelImagePulls to beta website#48681

Related issues：

• Parallel image pull should pull unique images only once kubernetes#125164

Please keep this description up to date. This will help the Enhancement Team to track the evolution of the enhancement efficiently.

[ruiwen-zhao]

/sig node

[pacoxu]

Some intial ideas can be found in https://docs.google.com/document/d/1MMZlKSxkDBDz0kEn9JGm848_IJgydJqXWz5t9D3Ev8I/edit# and kubernetes/kubernetes#112242 kubernetes/kubernetes#112044

[wojtek-t]

/lead opted-in
/milestone v1.27

[wojtek-t]

/label lead-opted-in

[marosset]

Hello @pacoxu & @ruiwen-zhao 👋, Enhancements team here.

Just checking in as we approach enhancements freeze on 18:00 PDT Thursday 9th February 2023.

This enhancement is targeting for stage alpha for v1.27 (correct me, if otherwise)

Here's where this enhancement currently stands:

• KEP readme using the latest template has been merged into the k/enhancements repo.
• KEP status is marked as implementable for latest-milestone: v1.27
• KEP readme has a updated detailed test plan section filled out
• KEP readme has up to date graduation criteria
• KEP has a production readiness review that has been completed and merged into k/enhancements.

For this enhancement, it looks like #3713 will address most of these requirements.
Please be sure to also:

• Update the status to implemnetable in the kep.yaml (when ready)

The status of this enhancement is marked as at risk. Please keep the issue description up-to-date with appropriate stages as well.
Thank you!

[ruiwen-zhao]

Hello @pacoxu & @ruiwen-zhao 👋, Enhancements team here.

Just checking in as we approach enhancements freeze on 18:00 PDT Thursday 9th February 2023.

This enhancement is targeting for stage alpha for v1.27 (correct me, if otherwise)

Here's where this enhancement currently stands:

• KEP readme using the latest template has been merged into the k/enhancements repo.
• KEP status is marked as implementable for latest-milestone: v1.27
• KEP readme has a updated detailed test plan section filled out
• KEP readme has up to date graduation criteria
• KEP has a production readiness review that has been completed and merged into k/enhancements.

For this enhancement, it looks like #3713 will address most of these requirements. Please be sure to also:

• Update the status to implemnetable in the kep.yaml (when ready)

The status of this enhancement is marked as at risk. Please keep the issue description up-to-date with appropriate stages as well. Thank you!

Updated the KEP PR #3713 to change the status to implemnetable. Once it is merged then all of the requirements will be addressed.

[marosset]

Thanks @ruiwen-zhao - this enhancement is ready to be tracked for v1.27

[pacoxu]

/assign @pacoxu @ruiwen-zhao

[ruiwen-zhao]

Draft document update: kubernetes/website#39732

[mickeyboxell]

Hi @ruiwen-zhao @pacoxu 👋, I’m reaching out from the 1.27 Release Docs team. This enhancement is marked as ‘Needs Docs’ for the 1.27 release.

Please follow the steps detailed in the documentation to open a PR against dev-1.27 branch in the k/website repo. This PR can be just a placeholder at this time, and must be created by March 16. For more information, please take a look at Documenting for a release to familiarize yourself with the documentation requirements for the release.

Please feel free to reach out with any questions. Thanks!

[ruiwen-zhao]

Hi @ruiwen-zhao @pacoxu 👋, I’m reaching out from the 1.27 Release Docs team. This enhancement is marked as ‘Needs Docs’ for the 1.27 release.

Please follow the steps detailed in the documentation to open a PR against dev-1.27 branch in the k/website repo. This PR can be just a placeholder at this time, and must be created by March 16. For more information, please take a look at Documenting for a release to familiarize yourself with the documentation requirements for the release.

Please feel free to reach out with any questions. Thanks!

Is kubernetes/website#39732 sufficient for now?

[marosset]

Hi @ruiwen-zhao @pacoxu 👋,

Checking in as we approach 1.27 code freeze at 17:00 PDT on Tuesday 14th March 2023.

Please ensure the following items are completed:

• All PRs to the Kubernetes repo that are related to your enhancement are linked in the above issue description (for tracking purposes).
• All PRs are fully merged by the code freeze deadline.

Please let me know if there are any other PRs in k/k I should be tracking for this KEP.
I see the following PR was closed (kubernetes/kubernetes#112242) - Is it going to be replaced with another PR?

As always, we are here to help should questions come up. Thanks!

[sreeram-venkitesh]

/milestone clear

[sreeram-venkitesh]

/label tracked/no

[haircommander]

@ruiwen-zhao @pacoxu do you plan on working on this in 1.32?

[haircommander]

it seems this one is waiting on a test, and folks are working on it. I've moved it to Proposed for Consideration and we can see if we can move forward on it.

[tjons]

Hi, enhancements lead here - I inadvertently added this to the 1.32 tracking board 😀. Please readd it if you wish to progress this enhancement in 1.32.

/remove-label lead-opted-in

[pacoxu]

If kubernetes/kubernetes#127495 can be merged, we can restart the work of adding e2e test for this feature then.

I would update the test once there are some updates. This may be a little uncertain in the 1.32 release cycle, but I think it should be possible in 1.33.

[pacoxu]

After a quick review, the effort is not big and I think this can be done in v1.32.
@SergeyKanzhelev

I will update the KEP ASAP to make it in v1.32.

[pacoxu]

Hi, enhancements lead here - I inadvertently added this to the 1.32 tracking board 😀. Please readd it if you wish to progress this enhancement in 1.32.

The e2e test was merged in v1.32 which is the beta criteria for this feature.

There is no feature gate for this feature. #4938 should update the KEP to be beta since the E2E test was added. And the website PR is in kubernetes/website#48681.

Can we re-add the milestone v1.32 for this issue? @tjons

[tjons]

@pacoxu we are well past enhancements freeze at this point, and the KEP changes are not merged. Unfortunately this would require an exception, you can open one if you'd like but I doubt it would be accepted given that enhancements freeze was ~3+ weeks ago.

[pacoxu]

Thanks for the information.

If so, we can update the KEP in 1.33 release cycle.

[pacoxu]

Can we use serializeImagePulls: false and maxParallelImagePulls: 5 or 10 to accelerate current e2e?

I may work on this to give it a try to see if it can accelerate the e2e CIs.

[aojea]

Can we use serializeImagePulls: false and maxParallelImagePulls: 5 or 10 to accelerate current e2e?

I may work on this to give it a try to see if it can accelerate the e2e CIs.

@pacoxu I'm not a fan of diverging CI config from normal deployments, I suggest to use a separate canary job first to analyze it and if results are good documenting it for users

[pacoxu]

@pacoxu I'm not a fan of diverging CI config from normal deployments, I suggest to use a separate canary job first to analyze it and if results are good documenting it for users

I did some test in CI, and I found the time cost is similar whether we enable the parallel image pull or not. I think there are a few reasons.

1. pod in CI are always pause/busybox or some re-used images.
2. the image pulling time is quite fast.
3. pods that were created in CIs are not using big images in CI.