Different digest for a single-arch reproducible image

[KoviRobi]

Description

Hi,

I've got an image which we are loading via docker load -i image.tar, when loading on most computers it gives the expected hash which is in the manifest.json

❯ tar Oxf test.tar.gz manifest.json|jq '.[].Config'
"5e981582c25298a01492a4af9f5e9548e4855248106aecec35515f7a726dc396.json"
❯ docker --version
Docker version 29.4.2, build v29.4.2
❯ docker load -i ./test.tar.gz
Loaded image: test:7bwkndsdbb4ialj34vsz8jx3i3hs3kdh
❯ docker images --digests --no-trunc
REPOSITORY            TAG                                DIGEST    IMAGE ID                                                                  CREATED        SIZE
test                  7bwkndsdbb4ialj34vsz8jx3i3hs3kdh   <none>    sha256:5e981582c25298a01492a4af9f5e9548e4855248106aecec35515f7a726dc396   56 years ago   0B
❯ docker inspect test:7bwkndsdbb4ialj34vsz8jx3i3hs3kdh | jq '.[].Id'
"sha256:5e981582c25298a01492a4af9f5e9548e4855248106aecec35515f7a726dc396"

(Side note, this is a NixOS computer, but I also have a Debian GNU/Linux 12 (bookworm) computer with the expected results, it's just less accessible because it's actively used in CI)

Full docker inspect result on expected behaviour

❯ docker inspect test:7bwkndsdbb4ialj34vsz8jx3i3hs3kdh                                
[
    {
        "Id": "sha256:5e981582c25298a01492a4af9f5e9548e4855248106aecec35515f7a726dc396",
        "RepoTags": [
            "test:7bwkndsdbb4ialj34vsz8jx3i3hs3kdh"
        ],
        "RepoDigests": [],
        "Comment": "store paths: ['/nix/store/b2m2wwiv88pj7p0dxjqb94sad5nvcz2d-test-customisation-layer']",
        "Created": "1970-01-01T00:00:01Z",
        "Config": {},
        "Architecture": "amd64",
        "Os": "linux",
        "Size": 0,
        "GraphDriver": {
            "Data": {
                "MergedDir": "/var/lib/docker/overlay2/2dfeecbedded5c0c85f37fd4d0e445d7c6f54ca480598cf9951039944abc94eb/merged",
                "UpperDir": "/var/lib/docker/overlay2/2dfeecbedded5c0c85f37fd4d0e445d7c6f54ca480598cf9951039944abc94eb/diff",
                "WorkDir": "/var/lib/docker/overlay2/2dfeecbedded5c0c85f37fd4d0e445d7c6f54ca480598cf9951039944abc94eb/work"
            },
            "Name": "overlay2"
        },
        "RootFS": {
            "Type": "layers",
            "Layers": [
                "sha256:f003e077fcffe7c286bb778c1514ebe6c7388d057e705b8bc43d6be413837a43"
            ]
        },
        "Metadata": {
            "LastTagTime": "0001-01-01T00:00:00Z"
        }
    }
]

But on one particular node, it gets a different digest, the one in the index.json when exporting the image.

jenkins@docker-linux-2:~$ docker --version
Docker version 29.4.0, build 9d7ad9f
jenkins@docker-linux-2:~$ docker load -i ./test.tar.gz
Loaded image: test:7bwkndsdbb4ialj34vsz8jx3i3hs3kdh
jenkins@docker-linux-2:~$ docker images --digests --no-trunc
REPOSITORY            TAG                                DIGEST                                                                    IMAGE ID                                                                  CREATED        SIZE
test                  7bwkndsdbb4ialj34vsz8jx3i3hs3kdh   sha256:f8f91ad4f1a9c2106e9e9e011bd80ce9d2e34df11bf6adf3cee4809ac8ad2cfc   sha256:f8f91ad4f1a9c2106e9e9e011bd80ce9d2e34df11bf6adf3cee4809ac8ad2cfc   56 years ago   15.2kB
jenkins@docker-linux-2:~$ docker inspect test:7bwkndsdbb4ialj34vsz8jx3i3hs3kdh
[
    {
        "Id": "sha256:f8f91ad4f1a9c2106e9e9e011bd80ce9d2e34df11bf6adf3cee4809ac8ad2cfc",
        "RepoTags": [
            "test:7bwkndsdbb4ialj34vsz8jx3i3hs3kdh"
        ],
        "RepoDigests": [
            "test@sha256:f8f91ad4f1a9c2106e9e9e011bd80ce9d2e34df11bf6adf3cee4809ac8ad2cfc"
        ],
jenkins@docker-linux-2:~$ docker image save -o test.out.tar test:7bwkndsdbb4ialj34vsz8jx3i3hs3kdh
jenkins@docker-linux-2:~$ tar tf test.out.tar 
blobs/
blobs/sha256/
blobs/sha256/5e981582c25298a01492a4af9f5e9548e4855248106aecec35515f7a726dc396
blobs/sha256/f003e077fcffe7c286bb778c1514ebe6c7388d057e705b8bc43d6be413837a43
blobs/sha256/f8f91ad4f1a9c2106e9e9e011bd80ce9d2e34df11bf6adf3cee4809ac8ad2cfc
index.json
manifest.json
oci-layout
jenkins@docker-linux-2:~$ tar Oxf test.out.tar manifest.json
[{"Config":"blobs/sha256/5e981582c25298a01492a4af9f5e9548e4855248106aecec35515f7a726dc396","RepoTags":["test:7bwkndsdbb4ialj34vsz8jx3i3hs3kdh"],"Layers":["blobs/sha256/f003e077fcffe7c286bb778c1514ebe6c7388d057e705b8bc43d6be413837a43"]}]
jenkins@docker-linux-2:~$ tar Oxf test.out.tar index.json
{"schemaVersion":2,"mediaType":"application/vnd.oci.image.index.v1+json","manifests":[{"mediaType":"application/vnd.docker.distribution.manifest.v2+json","digest":"sha256:f8f91ad4f1a9c2106e9e9e011bd80ce9d2e34df11bf6adf3cee4809ac8ad2cfc","size":420,"annotations":{"io.containerd.image.name":"docker.io/library/test:7bwkndsdbb4ialj34vsz8jx3i3hs3kdh","org.opencontainers.image.ref.name":"7bwkndsdbb4ialj34vsz8jx3i3hs3kdh"}}]}

Full docker inspect result on different behaviour

jenkins@docker-linux-2:~$ docker inspect test:7bwkndsdbb4ialj34vsz8jx3i3hs3kdh
[
    {
        "Id": "sha256:f8f91ad4f1a9c2106e9e9e011bd80ce9d2e34df11bf6adf3cee4809ac8ad2cfc",
        "RepoTags": [
            "test:7bwkndsdbb4ialj34vsz8jx3i3hs3kdh"
        ],
        "RepoDigests": [
            "test@sha256:f8f91ad4f1a9c2106e9e9e011bd80ce9d2e34df11bf6adf3cee4809ac8ad2cfc"
        ],
        "Comment": "store paths: ['/nix/store/b2m2wwiv88pj7p0dxjqb94sad5nvcz2d-test-customisation-layer']",
        "Created": "1970-01-01T00:00:01Z",
        "Config": {},
        "Architecture": "amd64",
        "Os": "linux",
        "Size": 11146,
        "RootFS": {
            "Type": "layers",
            "Layers": [
                "sha256:f003e077fcffe7c286bb778c1514ebe6c7388d057e705b8bc43d6be413837a43"
            ]
        },
        "Metadata": {
            "LastTagTime": "2026-05-20T12:47:18.214713381Z"
        },
        "Descriptor": {
            "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
            "digest": "sha256:f8f91ad4f1a9c2106e9e9e011bd80ce9d2e34df11bf6adf3cee4809ac8ad2cfc",
            "size": 420,
            "annotations": {
                "io.containerd.image.name": "docker.io/library/test:7bwkndsdbb4ialj34vsz8jx3i3hs3kdh",
                "org.opencontainers.image.ref.name": "7bwkndsdbb4ialj34vsz8jx3i3hs3kdh"
            }
        }
    }
]

test.tar.gz

Reproduce

1. docker load -i ./test.tar.gz
2. docker run -it sha256:5e981582c25298a01492a4af9f5e9548e4855248106aecec35515f7a726dc396

Works on most computers (docker-linux-1 and my NixOS and others), fails on computer docker-linux-2.

Expected behavior

I would expect the digest to be consistent, to be able to pin images for CI reproducibility.

docker version

❯ docker version                    
Client:
 Version:           29.4.2
 API version:       1.54
 Go version:        go1.26.2
 Git commit:        v29.4.2
 Built:             Thu Jan  1 00:00:00 1970
 OS/Arch:           linux/amd64
 Context:           default

Server:
 Engine:
  Version:          29.4.2
  API version:      1.54 (minimum version 1.40)
  Go version:       go1.26.2
  Git commit:       v29.4.2
  Built:            Tue Jan  1 00:00:00 1980
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v2.2.3
  GitCommit:        refs/tags/v2.2.3
 runc:
  Version:          1.3.5
  GitCommit:        
 docker-init:
  Version:          0.19.0
  GitCommit:        

jenkins@docker-linux-2:~$ docker version
Client: Docker Engine - Community
 Version:           29.4.0
 API version:       1.54
 Go version:        go1.26.1
 Git commit:        9d7ad9f
 Built:             Tue Apr  7 08:36:03 2026
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          29.4.0
  API version:      1.54 (minimum version 1.40)
  Go version:       go1.26.1
  Git commit:       daa0cb7
  Built:            Tue Apr  7 08:36:03 2026
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v2.2.3
  GitCommit:        77c84241c7cbdd9b4eca2591793e3d4f4317c590
 runc:
  Version:          1.3.5
  GitCommit:        v1.3.5-0-g488fc13e
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

❯ docker info                                                       
Client:
 Version:    29.4.2
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.31.1
    Path:     /nix/store/b97nak5s3d0jisvsnsbmbwivph67dgc6-docker-buildx-0.31.1/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  5.1.3
    Path:     /nix/store/c9ws1cmxmlqbxa926m696f78dksy2g0c-docker-compose-5.1.3/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 2
 Server Version: 29.4.2
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: journald
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 CDI spec directories:
  /etc/cdi
  /var/run/cdi
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: refs/tags/v2.2.3
 runc version: 
 init version: 
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.18.28
 Operating System: NixOS 26.05 (Yarara)
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 31.09GiB
 Name: promethium-nix1
 ID: ade51fb8-fb13-4c0b-9ff2-4b2ac348571a
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  ::1/128
  127.0.0.0/8
 Live Restore Enabled: true
 Firewall Backend: iptables


jenkins@docker-linux-2:~$ docker info
Client: Docker Engine - Community
 Version:    29.4.0
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.33.0
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v5.1.3
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 2
 Server Version: 29.4.0
 Storage Driver: overlayfs
  driver-type: io.containerd.snapshotter.v1
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 CDI spec directories:
  /etc/cdi
  /var/run/cdi
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 77c84241c7cbdd9b4eca2591793e3d4f4317c590
 runc version: v1.3.5-0-g488fc13e
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.1.0-44-amd64
 Operating System: Debian GNU/Linux 12 (bookworm)
 OSType: linux
 Architecture: x86_64
 CPUs: 56
 Total Memory: 125.8GiB
 Name: docker-linux-2
 ID: 330090b0-a8cc-47c7-890a-45f553bc4892
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
  ::1/128
 Live Restore Enabled: false
 Firewall Backend: iptables

Additional Info

Version for docker-linux-1

jenkins@docker-linux-1:~$ docker version
Client: Docker Engine - Community
Version:           28.2.1
API version:       1.50
Go version:        go1.24.3
Git commit:        879ac3f
Built:             Wed May 28 19:25:17 2025
OS/Arch:           linux/amd64
Context:           default

Server: Docker Engine - Community
Engine:
  Version:          28.2.1
  API version:      1.50 (minimum version 1.24)
  Go version:       go1.24.3
  Git commit:       0e2cc22
  Built:            Wed May 28 19:25:17 2025
  OS/Arch:          linux/amd64
  Experimental:     false
containerd:
  Version:          1.7.27
  GitCommit:        05044ec0a9a75232cad458027ca83437aae3f4da
runc:
  Version:          1.2.5
  GitCommit:        v1.2.5-0-g59923ef
docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Info for docker-linux-1

jenkins@docker-linux-1:~$ docker info
Client: Docker Engine - Community
Version:    28.2.1
Context:    default
Debug Mode: false
Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.24.0
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.36.2
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
Containers: 21
  Running: 1
  Paused: 0
  Stopped: 20
Images: 881
Server Version: 28.2.1
Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
CDI spec directories:
  /etc/cdi
  /var/run/cdi
Swarm: inactive
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 05044ec0a9a75232cad458027ca83437aae3f4da
runc version: v1.2.5-0-g59923ef
init version: de40ad0
Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
Kernel Version: 6.1.0-37-amd64
Operating System: Debian GNU/Linux 12 (bookworm)
OSType: linux
Architecture: x86_64
CPUs: 56
Total Memory: 125.8GiB
Name: docker-linux-1
ID: 93d75795-2f94-4a5e-8916-918f4042654f
Docker Root Dir: /var/lib/docker
Debug Mode: false
Experimental: false
Insecure Registries:
  ::1/128
  127.0.0.0/8
Live Restore Enabled: false

[KoviRobi]

Something that pops up immediately from docker info (thanks for that, I wasn't aware) is the storage drivers

-Storage Driver: overlay2
-  Backing Filesystem: extfs
-  Supports d_type: true
-  Using metacopy: false
-  Native Overlay Diff: true
-  userxattr: false
+Storage Driver: overlayfs
+  driver-type: io.containerd.snapshotter.v1

I wonder if I configure this, it'll fix the issue for me (still, might be useful to document this image digest incompatibility somewhere -- entirely possible it is already there and I failed to find it)

This could be related to https://docs.docker.com/engine/storage/drivers/select-storage-driver/

Note

Docker Engine 29.0 and later uses the containerd image store by default for fresh installations. If you upgraded from an earlier version, your daemon continues using the classic storage drivers described on this page. You can migrate to the containerd image store following the instructions in the containerd image store documentation.

[KoviRobi]

Also I wonder if it is related to #3394

[vvoland]

This is related to attestations being generated by default. They contain a timestamp so they change with every build.
This doesn't affect the actual image layers which should still be reproducible if the build inputs didn't change.

Please see: moby/moby#48391

[KoviRobi]

Thanks for the quick response, after looking at the issue linked, I see that the workaround there is to add --provenance false, but this is missing for docker image load. Does this mean that docker image load needs a --provenance flag added? Since I don't (re)build the image, I just load an existing image. Though this doesn't mean it isn't due to provenance info, i.e. info could be added during load.

If the issue is due to timestamp during load, then wouldn't it have a different ID if I remove the image and load it again?

jenkins@docker-linux-2:~$ SOURCE_DATE_EPOCH=0 docker image load -i ./test.tar.gz
Loaded image: test:7bwkndsdbb4ialj34vsz8jx3i3hs3kdh
jenkins@docker-linux-2:~$ docker image inspect --format '{{.Id}}' test:7bwkndsdbb4ialj34vsz8jx3i3hs3kdh
sha256:f8f91ad4f1a9c2106e9e9e011bd80ce9d2e34df11bf6adf3cee4809ac8ad2cfc
jenkins@docker-linux-2:~$ docker rmi test:7bwkndsdbb4ialj34vsz8jx3i3hs3kdh 
Untagged: test:7bwkndsdbb4ialj34vsz8jx3i3hs3kdh
Deleted: sha256:f8f91ad4f1a9c2106e9e9e011bd80ce9d2e34df11bf6adf3cee4809ac8ad2cfc
jenkins@docker-linux-2:~$ docker image load -i ./test.tar.gz
Loaded image: test:7bwkndsdbb4ialj34vsz8jx3i3hs3kdh
jenkins@docker-linux-2:~$ docker image inspect --format '{{.Id}}' test:7bwkndsdbb4ialj34vsz8jx3i3hs3kdh
sha256:f8f91ad4f1a9c2106e9e9e011bd80ce9d2e34df11bf6adf3cee4809ac8ad2cfc
jenkins@docker-linux-2:~$ docker rmi test:7bwkndsdbb4ialj34vsz8jx3i3hs3kdh
Untagged: test:7bwkndsdbb4ialj34vsz8jx3i3hs3kdh
Deleted: sha256:f8f91ad4f1a9c2106e9e9e011bd80ce9d2e34df11bf6adf3cee4809ac8ad2cfc

Same even if I run docker system prune -a before loading.

[vvoland]

Load just loads whatever is in the archive.

The only difference is if you import across graphdrivers <> containerd.
In graphdrivers ID of the image is the digest of the image config.
For containerd the ID is the digest of the index/manifest.