docker pull does not know how to resume previous failed attempt

[jchia]

• This is a bug report
• This is a feature request
• I searched existing issues before opening this one

Expected behavior

When a first attempt to docker pull an image fails after it has partial downloaded some data, a subsequent attempt should not start from scratch but resume the download from wherever the previous attempt failed. This way, when downloading an image that is large relative to network reliability, there is a high-enough probability of docker pull succeeding within a reasonable number of attempts.

Actual behavior

Subsequent docker pull attempts start from scratch even if a previous attempt has "pull complete" for some layers. This means that for a large image and an intermittently failing connection, the chance of docker pull completing within a reasonable number of tries approaches zero as the image size increases. Consequently, this behavior can be considered to be a usability bug.

$ docker pull xilinx/vitis-ai:latest
latest: Pulling from xilinx/vitis-ai
f22ccc0b8772: Pull complete 
3cf8fb62ba5f: Pull complete 
e80c964ece6a: Pull complete 
afd3e84ff233: Pull complete 
caba11bf89c1: Pull complete 
21ff7ffad7fb: Pull complete 
0a1fb73e7f8f: Pull complete 
b700632d3b49: Pull complete 
f32b167b0bc9: Pull complete 
d5dca61bcafa: Pull complete 
bb15414437e3: Pull complete 
e73de575fa84: Pull complete 
b1a14839c8e3: Pull complete 
75052f7c411f: Pull complete 
e6c777df29dc: Pull complete 
741b871f4d0b: Pull complete 
bd09e2cec15d: Pull complete 
49359a403f15: Pull complete 
3dc515bfbb5a: Pull complete 
705cf80df2f3: Pull complete 
53d1bd56d0d4: Pull complete 
d2572bf03a3f: Pull complete 
c970ce782924: Pull complete 
b18006f4f833: Pull complete 
7dc2ec620aa4: Pull complete 
31cda23c4bce: Pull complete 
c4a6256a9c1b: Downloading [==================================================>]  3.801GB/3.801GB
b0190525f152: Download complete 
b44b92ba8003: Download complete 
ab4ebe064c35: Download complete 
unexpected EOF
$ docker pull xilinx/vitis-ai:latest
latest: Pulling from xilinx/vitis-ai
f22ccc0b8772: Downloading [======================================>            ]  20.56MB/26.71MB
...

Steps to reproduce the behavior

1. docker pull xilinx/vitis-ai:latest
2. Ctrl-C after download finishes for some layer (simulating network failure).
3. docker pull xilinx/vitis-ai:latest and see that it starts from scratch.

Output of docker version:

$ docker version
Client:
 Version:           19.03.8
 API version:       1.40
 Go version:        go1.13.8
 Git commit:        afacb8b7f0
 Built:             Wed Oct 14 19:43:43 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          19.03.8
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.8
  Git commit:       afacb8b7f0
  Built:            Wed Oct 14 16:41:21 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.3.3-0ubuntu2
  GitCommit:        
 runc:
  Version:          spec: 1.0.1-dev
  GitCommit:        
 docker-init:
  Version:          0.18.0
  GitCommit:

Output of docker info:

Client:
 Debug Mode: false

Server:
 Containers: 3
  Running: 0
  Paused: 0
  Stopped: 3
 Images: 43
 Server Version: 19.03.8
 Storage Driver: overlay2
  Backing Filesystem: <unknown>
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 
 runc version: 
 init version: 
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 5.4.0-47-generic
 Operating System: Ubuntu 20.04.1 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 31.29GiB
 Name: frodo
 ID: KMLP:N2DN:SFDB:3AH4:ATYJ:6TUE:JKWM:SNY5:MHPT:LTG4:KJIW:CGWE
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No swap limit support

Additional environment details (AWS, VirtualBox, physical, etc.)
Ubuntu 20.04.1 LTS

[mika-cn]

I also met this problem. In my case there was no space left on device. But after the command failed, I checked my disk usage and it was not full. So the docker pull command will delete all downloaded bytes when it's execution fails?

I also noticed that the docker pull command can do incremental downloads. If the command keeps the last downloaded result. The next execution can continue from where the last execution failed rather than download everything again.

Belowing is my case:

$ docker pull someImage

d8d02d457314: Pull complete 
aedaaab71de6: Pull complete 
6dd0b11147d1: Downloading [==================================================>]  161.6MB/161.6MB
9ad552f1001e: Download complete 
c0436fc23e5b: Download complete 
dc5b4c232dbb: Download complete 
write /var/lib/docker/tmp/GetImageBlob965483379: no space left on device

[Cartman0]

hi, i also want this feature such as curl -C, wget -c

[Cartman0]

we may be able to "resume" by saving temp of each image-layer.

[shakibamoshiri]

I already thought docker has this feature, till I tested it manually and realized there is no such a feature.
Is there anything we should enable? An option or configuration?
Or any version which after that we can have this feature?
For people who have slow network this is super useful.

[jspilinek]

I ran into this issue recently. I was trying to pull Oracle 19c database which includes a 3GB file. It kept timing out after downloading 500MB to 1GB. I think something was going on with my internet connection that day. It eventually worked after attempting to pull about 8 times. Very frustrating that there's no option to resume the download.

[rajanant]

Extremely super useful feature

[chrisbward]

Insane

[Nefcanto]

So, is this implemented now?

[knusbaum]

This would be extremely helpful.

Pulling an image from docker hub that had many layers kept hanging for me today. Because it didn't resume and keep the successfully-downloaded layers, I had to retry several dozen times. Each time some random layer (or two) would hang. This is especially annoying and time consuming when the image contains lots of large layers, where each pull operation pulls a GB or more before failing.

[cpuguy83]

This will be in the next version of Docker.
Specifically you'll need to enable the containerd storage backend (which is/will-be opt-in) to support retries.
Also docker-desktop should have this today if you enable the experimental containerd storage backend.

I'm going to close this for house-keeping purposes since this work is already merged upstream.
Give it a try!

[Nefcanto]

@cpuguy83 Sorry to reopen this again. Please guide us more. Where are the docs regarding this? At least give us a link. How should we do it on Linux? What steps should I take in my Dockerfile or Docker Hub account, or in the docker command or anywhere else to enable it and use it? Please give us more information. Thanks.

[cpuguy83]

On Linux when using a build from HEAD, add this to dockerd's command line arguments: --features='{"containerd-snapshotter": true}'. Then whatever is passed to --storage-driver will be interpreted as a containerd snapshotter rather than a docker graphdriver. Regardless though, this enables pulling using containerd, which supports restarting from a partial pull.

In Docker Desktop, in settings>features in development, select "use containerd for pulling and storing images"

Blog post: https://www.docker.com/blog/extending-docker-integration-with-containerd/

[mpeschel10]

Hi. I am on Arch Linux with docker 24.0.2. I was not able to make the --features flag work on my system. I tried changing the ExecStart line of /usr/lib/systemd/system/docker.service, but systemctl start docker would fail. journalctl -u docker gave "Status: unknown flag: --features".

However, I was able to enable docker pull to resume interrupted downloads by adding "features": {"containerd-snapshotter": true} to /etc/docker/daemon.json.

sudo mkdir /etc/docker
sudo gedit /etc/docker/daemon.json

Then make it look like:

{
    "features": {"containerd-snapshotter": true}
}

And restart docker.

sudo systemctl restart docker

Link to config file documentation.

[cpuguy83]

@mpeschel10
I actually quickly responded half way into reading the email notification first thing in the morning then I saw the rest of it and I'm like "oh I'm an idiot, let me just delete my post". 🤣

Thanks for reporting back!