Description
This is an issue we can watch since quite some time now. It used to be so much of a problem, so either this is a regression or something else has changed.
So what happens, when we pull images with ctr, it will start pulling them and then stop doing so for some of the progressing layers. The bigger the layers, the longer it will take. Ultimately, the download will stop entirely. Only when the download finally is stopped completely and we retry the download, it will eventually continue downloading and finish the download.
/var/lib/rancher/rke2/bin/ctr --address /run/k3s/containerd/containerd.sock --namespace k8s.io i pull registry.gitlab.com/gitlab-org/build/cng/gitlab-workhorse-ee:v18.3.2
registry.gitlab.com/gitlab org/build/cng fetching image content
└──index (303019b0513e) already exists
├──manifest (92ebedb6448c) already exists
│ └──config (c78196c06a50) already exists
└──manifest (66e18b69a6c6) already exists
├──config (302e104830b8) already exists
├──layer (8d2459d3b16a) already exists
├──layer (e32ee7c428f5) downloading |++++++++++++++++++++++++++++++++++----| 281.4 Mi/312.7 MiB
├──layer (f6612896bb95) already exists
├──layer (05e725ddd689) already exists
└──layer (556185b9d0d5) already exists
application/vnd.docker.distribution.manifest.list.v2+json sha256:303019b0513e4806f73dbda2b77b8cffa531a881f3fc427df1bfb374ec062472
Pulling from OCI Registry (registry.gitlab.com/gitlab-org/build/cng/gitlab-workhorse-ee:v18.3.2) elapsed: 1.2 s total: 333.1 (283.6 MiB/s)
Here an example where it got stuck downloading and did not continue anymore.
Here some of the versions:
root@:/home/ubuntu# /var/lib/rancher/rke2/bin/ctr --version
ctr github.com/k3s-io/containerd v2.0.5-k3s2
root@:/home/ubuntu# /var/lib/rancher/rke2/bin/containerd --version
containerd github.com/k3s-io/containerd v2.0.5-k3s2 a753664d5b5138d2798d8597b2aaaa92afd0a4f4
And some other params: This comes from the rke2 distribution, it is running on ubuntu with a 6.8.0 linux kernel. The public internet NIC interface has a reduced MTU of 1450 (from the behavior one guess would be that containerd/ctr for some reason completely ignores the MTU/MSS, as this is a common behavior to be seen in those cases).
Let me know if we can provide any more information that might be useful.
Steps to reproduce the issue
Describe the results you received and expected
download should continue in one go
What version of containerd are you using?
v2.0.5-k3s2 a753664
Any other relevant information
No response
Show configuration if it is related to CRI plugin.
No response
Description
This is an issue we can watch since quite some time now. It used to be so much of a problem, so either this is a regression or something else has changed.
So what happens, when we pull images with ctr, it will start pulling them and then stop doing so for some of the progressing layers. The bigger the layers, the longer it will take. Ultimately, the download will stop entirely. Only when the download finally is stopped completely and we retry the download, it will eventually continue downloading and finish the download.
Here an example where it got stuck downloading and did not continue anymore.
Here some of the versions:
And some other params: This comes from the rke2 distribution, it is running on ubuntu with a 6.8.0 linux kernel. The public internet NIC interface has a reduced MTU of 1450 (from the behavior one guess would be that containerd/ctr for some reason completely ignores the MTU/MSS, as this is a common behavior to be seen in those cases).
Let me know if we can provide any more information that might be useful.
Steps to reproduce the issue
Describe the results you received and expected
download should continue in one go
What version of containerd are you using?
v2.0.5-k3s2 a753664
Any other relevant information
No response
Show configuration if it is related to CRI plugin.
No response