Copy to docker-daemon overlay2 storage

[half2me]

Is there a way to copy an image to the overlay2 storage that docker daemon uses without running the docker daemon? I'd like to just specify a path on the filesystem where the overlay2 storage is.
I'm running in an environment where I cannot run docker daemon but I need to load the images into the overlay2 storage so when docker daemon is started it has those images preloaded

[mtrmac]

Thanks for reaching out.

That feature doesn’t exist, and I don’t think it can be reasonably supported. Docker’s image storage is, presumably, an internal implementation detail of Docker; so, running docker pull or maybe docker load (incl. using a docker-daemon: destination in Skopeo), perhaps in a container/VM/some other isolation, would be the only supportable way to modify that storage.

[half2me]

thanks @mtrmac for clearing that up. I'm struggling to build a customised raspberry pi image with some preloaded docker images already loaded. I wanted to avoid exporting to tar files and running docker load on first boot as that slows things down quite a lot, especially with slow sd cards. I wonder if there is any other way to achieve this.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[ethanjli]

I'm struggling to build a customised raspberry pi image with some preloaded docker images already loaded.

I'm doing something similar on GitHub Actions for CI builds of custom Raspberry Pi SD card images, but I haven't found an ideal solution yet either. The approach I took was to:

1. Pre-download images using Skopeo into docker-archive: tarballs (in an unbooted systemd-nspawn container or pimod chroot with QEMU binary virtualization in GitHub Actions, which is quite fast; I also cache those tarballs with GitHub Actions caching to avoid getting rate-limited by Docker Hub in matrix builds of SD card image variants which all need the same set of container images); and then
2. Have my GitHub Actions workflow boot the Raspberry Pi SD card image in a QEMU VM to run docker image load on those pre-downloaded tarballs.

Using Docker to load container images from pre-downloaded tarballs and instantiate containers from them in a booted QEMU VM takes a painfully long amount of time (~30-40 minutes per CI workflow run, with ~20 min of that spent on loading tarballs into the daemon and the rest on instantiating images into containers) when I have ~10-15 container images to load and several of those images are large (~100 MB each for ~5 images); it's much slower than loading images on my real Raspberry Pi with a decent SD card, but it allows me to achieve a first-boot duration of ~45 sec, with ~25 sec of that spent on doing Docker-related stuff (~10 sec on starting containerd.service + docker.service, and ~15 sec doing docker compose up on a bunch of Compose apps). If you can tolerate long CI workflow runs and you really need to minimize first-boot duration, this approach might be an acceptable tradeoff.

I'm hoping that once GitHub Actions makes their ARM-hosted runners generally available beyond beta (and hopefully free to use), it might be feasible to run the Docker daemon in an unbooted systemd-nspawn container instead of a booted QEMU VM, and that doing so might be significantly faster.

[ethanjli]

Update: I've found a pretty nice workaround, which is to preload container images into containerd instead of Docker. I can do this using nerdctl --namespace moby load or ctr --namespace moby images import; either command runs successfully in an unbooted systemd-nspawn container attached to the Raspberry Pi SD card image's filesystem after I've started the containerd daemon in the systemd-nspawn container, without the Docker daemon. Making this change dramatically reduced the amount of time I had to spend in the booted QEMU VM afterwards (by ~20 minutes, with the remaining 10 minutes spent on instantiating containers from the loaded images using the Docker daemon).

The caveat to make this work is that you must enable an experimental Docker setting to use the containerd image store so that Docker will use the images loaded into containerd - but if you're willing to do that, you can preload your container images before first boot in a relatively tolerable amount of time.

[half2me]

@ethanjli this is a nice solution! Thx