docker image ls does not show images loaded from a tar file

[GatFox]

Description

After changing the storage driver to containerd and loading images from tar file from other storage drivers, images are not showing in "docker image ls" but it does show up in "ctr -n moby images ls".

Reproduce

From old storage driver.

docker save image1 -o image1.tar
docker save image2 -o image2.tar
docker-compose down
change the storage driver in sduo vi /etc/docker/daemon.json
"features" : { "containerd-snapshotter": true },
sudo systemctl docker
docker load --input image1.tar
docker load --input image2.tar
docker-compose up
docker ps
docker image ls # does not show any images.
sudo ctr -n moby images ls # shows both images are present.

Expected behavior

Received.
docker image ls
no images shows up which is loaded via docker load from a tar file (which was created from different storage driver)

Exepcted:
docker image ls
should show all the loaded images.

docker version

Client:
 Version:           24.0.5
 API version:       1.43
 Go version:        go1.20.3
 Git commit:        24.0.5-0ubuntu1~20.04.1
 Built:             Mon Aug 21 19:50:14 2023
 OS/Arch:           linux/amd64
 Context:           default

Server:
 Engine:
  Version:          24.0.5
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.3
  Git commit:       24.0.5-0ubuntu1~20.04.1
  Built:            Mon Aug 21 19:50:14 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.12-0ubuntu1~20.04.3
  GitCommit:
 runc:
  Version:          1.1.7-0ubuntu1~20.04.2
  GitCommit:
 docker-init:
  Version:          0.19.0
  GitCommit:

docker info

Client:
 Version:    24.0.5
 Context:    default
 Debug Mode: false

Server:
 Containers: 5
  Running: 5
  Paused: 0
  Stopped: 0
 Images: 9
 Server Version: 24.0.5
 Storage Driver: overlayfs
  driver-type: io.containerd.snapshotter.v1
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 nvidia runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version:
 runc version:
 init version:
 Security Options:
  apparmor
  seccomp
   Profile: builtin
 Kernel Version: 5.4.0-146-generic
 Operating System: Ubuntu 20.04.6 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 15.53GiB
 Name: RGDTEST
 ID: 505296e0-1621-47ce-8811-fe172e518265
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No swap limit support

Additional Info

No response

[GatFox]

Tried the workaround listed in this issue-44579,
Tagging the image does not seems to work after loading it from a tar file.
Image does not show up in docker image ls

[vvoland]

There were a lot of changes related to the containerd integration since v24. Could you please try reproducing with v26?

[GatFox]

@vvoland , upgrading docker version to 26 did not solve the issue,
I can see the image in ctr -n moby images ls but it does not show up in docker image ls command.

Here is the docker version I upgraded to.
:~# docker --version
Docker version 26.0.0, build 2ae903e

[vvoland]

Does it also happen with alpine image?

If it doesn't, is there anything different about your image?
Could you show the content of the index.json from the saved archive?

For example:

❱ tar xO index.json < golangci.tar | jq
{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.index.v1+json",
  "manifests": [
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:ed4bd7bb174cdb029a39224da6cc73aa860c0f623d9dbf931aa35e1f63b0ee18",
      "size": 1529,
      "annotations": {
        "io.containerd.image.name": "docker.io/golangci/golangci-lint:v1.50.1",
        "org.opencontainers.image.ref.name": "v1.50.1"
      },
      "platform": {
        "architecture": "arm64",
        "os": "linux"
      }
    }
  ]
}

[GatFox]

@vvoland ,
Yes it did happened to alpine image as well!

BUT, the second piece about index.json has solved the mystery for us!

I totally zoned out and forgot to mention one most important step. which is.
Upgrading the docker to a newer version in between saving the images, switching the storage driver and then loading the images.

We had this older docker version installed as a .deb package (by copying locally).
Saving images from this older version did not had index.json file even for alpine image

While trying to upgrade the docker (after saving the image), it was actually upgrading the kernel as well, which we did NOT want it to upgraded for this change.
So we were actually upgrading docker via sudo apt install --only-upgrade docker.io=24.0.5-0ubuntu1~20.04.1

After following this link on how to upgrade/install docker, we actually had to remove docker.io and install other packages listed in this link here.

I have tested the whole process again and it installed the latest version of docker Docker version 26.0.1, build d260a54

This seems to have solved this issue of images not being seen in docker image ls.

[vvoland]

Good to hear! I'll close the issue then.

[GatFox]

It appears that the image saved from old version of docker does not have index.json file and so we can not find the digest ID for it.
So above solution of performing following tasks have solved the issue of images are not being seen in docker image ls
Saving images with older version of docker
removing older version of docker
re-installing new version of docker
change it to containerd snapshotter
loading the saved images

However any new tags we try to pull for that loaded image in new version of docker again does not show up in docker image ls
however it can be seen in ctr -n moby images ls

Any help will be appreciated!

[vvoland]

I managed to reproduce this on v26 with:

$ docker load -i <alpine-non-containerd.tar>
$ docker pull alpine:latest
# docker images will not show alpine

Note: This only happens if the pulled image DiffID is identical (all layers have the same uncompressed digest) layer, so it wouldn't happen for example if the alpine:latest now points to a new image.

Quick notes on why it happens:

1. Old image is exported from graphdrivers which recreates the layer tar file and stores it in uncompressed form
2. Pulling the new image downloads the new image, but doesn't fetch the layer content (due to client.Pull(..., WithPullUnpack) doesn't fetch layer contents if snapshot already exists containerd/containerd#8973), because a snapshot for that DiffID already exists.
3. Old image is deleted, along with the layer content blob (because it's no longer referenced by the new manifest, as it references the digest of the compressed tar).
4. The new image still has a snapshot, but the content blob doesn't exist, so the image doesn't show in docker images because it's incomplete.

[GatFox]

I really appreciate your response and re opening this issue!

As you know we had docker.io installed from a local deb package (version Docker version 20.10.21, build 20.10.21-0ubuntu1~20.04.1)
Is it safe to upgrade the older package docker.io to Docker version 24.0.5, build 24.0.5-0ubuntu1~20.04.1 first and then save images? It appears once it is upgraded to 24.0.5 tar file seems to have index.json file along with all blobs.
Then upgrade it to 26.01 and change the storage driver to containerd snapshotter?

We are trying to avoid the docker pull on all the images after the update and switching the storage drivers.

Thank you.

[GatFox]

Okay so this does not seems to be the save issue from older version of docker.
I have saved all the images from the same version Docker version 26.0.1, build d260a54,
switched to containerd ,
loaded all the images.
All images shows up.
But when we pull the next tag or latest,
those images does NOT show up.

Is there an interim work around we can try and test? we would really like to migrate to use containerd snapshotter for all our needs.

Thank you.

[vvoland]

Starting from v24+, docker save outputs OCI-compatible archives (part of this is the index.json), but it doesn't actually matter in this case.

The issue happens when re-pulling an image that was loaded from a image tar saved from the graphdriver (non-containerd snapshotter enabled Docker).

You can do what you do now, just remember that if you do docker pull for the first time, and the pulled image will be exactly the same, you need to delete it first.

# on graphdriver store (for example on 20.10)
$ docker save alpine >alpine.tar

# on a 26.0 with `features: { "containerd-snapshotter": true }`
$ docker load -i /v/alp.tar
Loaded image: alpine:latest

# The image is there
$ docker images
REPOSITORY   TAG       IMAGE ID       CREATED        SIZE
alpine       latest    b3d53b9c3336   2 months ago   16.4MB

# If you want to pull it again, remove it first
$ docker rmi alpine
Untagged: alpine:latest
Deleted: sha256:b3d53b9c333611da7a2888893bf5e6d8884406035837e63facfb5006e24a0fa2

$ docker pull alpine
Using default tag: latest
latest: Pulling from library/alpine
bca4290a9639: Download complete
Digest: sha256:c5b1261d6d3e43071626931fc004f70149baeba2c8ec672bd4f27761f8e1ad6b
Status: Downloaded newer image for alpine:latest
docker.io/library/alpine:latest

# It will still be there if you delete the previous one before pull
$ docker images
REPOSITORY   TAG       IMAGE ID       CREATED        SIZE
alpine       latest    c5b1261d6d3e   2 months ago   11.8MB

[GatFox]

Thank you for your help and quick responses in this matter.
Our purpose of saving and loading the images was to save the time it takes to pull and network bandwidth utilization.

However we were able to solve that issue by using the local registry.

For anyone interested, Here is how we did it without having to save/load and without having to pull the images again.

1). Update to latest supported version of docker

2). Create a local registry with bind volume mount.

docker run -d -p 5000:5000 --restart always -v /var/tmp/docker_image_registry:/var/lib/registry --name local_registry registry:2

3). For each image tag and push to local registry.

	export IMAGES="image1:release-1 image1:release-2 image2:latest myrepo/image3:0.1.1"
	#A). Tag and push the image.
	for image in $IMAGES; do 
   	     echo $image; 
 	    docker tag $image localhost:5000/$image;
            docker push localhost:5000/$image;
	done

	# B) Verify local registry have all the repository
	curl -X GET http://localhost:5000/v2/_catalog

4). Stop the local registry and delete all the images using docker rmi -f

	docker stop local_registry
	for image in $(docker images |grep -v -E "local_registry|REPOSITORY" |awk '{print $1, $2}' |tr " " ":"); do
	    docker rmi $image
	done
	# system prune 
	docker system prune

5). Switch to containerd (features: { "containerd-snapshotter": true })

6). Restart docker for change to take affect
systemctl restart docker

7). Re-run local registry
docker run -d -p 5000:5000 --restart always -v /var/tmp/docker_image_registry:/var/lib/registry --name local_registry registry:2

8). Verify all repositories.
curl -X GET http://localhost:5000/v2/_catalog

9). For each image pull from local registry and re-tag

	export IMAGES="image1:release-1 image1:release-2 image2:latest myrepo/image3:0.1.1"
	# A).  Pull and re-tag all images.
	for image in $IMAGES; do 
	    echo $image; 
	    docker pull localhost:5000/$image;
            docker tag localhost:5000/$image $image;
	done

10). Verify all images
docker image ls

11. Remove local tags.

	export IMAGES="image1:release-1 image1:release-2 image2:latest myrepo/image3:0.1.1"
	for image in $IMAGES; do
	    echo $image
	    docker rmi -f localhost:5000/$image
	done

12). Clean up.

	# Remove content of the bind mount directory /var/tmp/docker_image_registry
	rm -rf /var/tmp/docker_image_registry`
	# Remove local registry
	docker stop local_registry
	docker container rm local_registry
	docker rmi -f registry:2

       # System prune:
	docker system prune