Docker's layer content hashing scheme doesn't follow the canonicalization rules

[ixmatus]

Description

When saving an image to the filesystem, Docker computes a Content ID hash for the layer using a Chain ID injected into a JSON object with null or empty keys. In cases where the layer has a parent, the parent's Content ID is also injected. The top-level keys of this object follow docker's rules for canonicalized JSON in that they are lexically sorted, however this property is not applied recursively to the keys of the sub-object produced by serializing the empty struct datatype.

Describe the results you received:

{"container_config":{"Hostname":"","Domainname":"","User":"","AttachStdin":false,"AttachStdout":false,"AttachStderr":false,"Tty":false,"OpenStdin":false,"StdinOnce":false,"Env":null,"Cmd":null,"Image":"","Volumes":null,"WorkingDir":"","Entrypoint":null,"OnBuild":null,"Labels":null},"created":"0001-01-01T00:00:00Z","layer_id":"sha256:5e6f832cd2df18460af48ed117c5b63bc2189971c9346e6d952376b5a8ba74ff"}

Describe the results you expected:

{"container_config":{"AttachStderr":false,"AttachStdin":false,"AttachStdout":false,"Cmd":null,"DomainName":"","Entrypoint":"","Env":null,"Hostname":"","Image":"","Labels":null,"OnBuild":null,"OpenStdin":false,"StdinOnce":false,"Tty":false,"User":"","Volumes":null,"WorkingDir":""},"created":"0001-01-01T00:00:00Z","layer_id":"sha256:5e6f832cd2df18460af48ed117c5b63bc2189971c9346e6d952376b5a8ba74ff"}

Output of docker version:

» docker version
Client:
 Version:      1.12.3
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   6b644ec
 Built:        Wed Oct 26 22:01:48 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.3
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   6b644ec
 Built:        Wed Oct 26 22:01:48 2016
 OS/Arch:      linux/amd64

Output of docker info:

 » docker info
Containers: 163
 Running: 0
 Paused: 0
 Stopped: 163
Images: 253
Server Version: 1.12.3
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 1080
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: host overlay null bridge
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: apparmor seccomp
Kernel Version: 4.4.0-45-generic
Operating System: Ubuntu 16.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 15.59 GiB
Name: griffin
ID: QTOY:UBAR:ZHP2:K5B4:IWOA:32ZG:6OK4:Z5P4:TXAN:YU5E:YQUV:76M3
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Insecure Registries:
 127.0.0.0/8

[cpuguy83]

ping @tonistiigi

[justincormack]

ping @stevvooe

[stevvooe]

@ixmatus Thanks for the report!

If we look at the struct that generated this json, container.Config, we can see that the order in the serialized json is actually the struct order. This field ordering matches Go's json package output, which was documented here. The top-level struct does a two pass generation, eliding type safety in the process, but resulting sorted fields, since it passes through a map. I'll let @aaronlehmann or @tonistiigi comment further, but this was done to preserve compatibility.

The main problem here is the fundamental defect of CAS applications that require round trip serialization. Even when you think you've handled every possible source of instability, there is always something that is missed. Keeping this consistent from version to version is hard enough let across implementations. In this case, we did due diligence on specifying the use of canonical json, but didn't ensure that we used everywhere. The affect is that given any change in the output, the entire image itself is a different image.

Fortunately, this is okay. We address this problem by ensuring that this json is only ever generated once. We also have tests that ensure that from version to version, the generated json is as consistent as possible. In addition, content is verified at the byte-level, rather than the structure, to ensure that we can consume content generated with a different ordering or standard. However, if we do change this, we end up with incompatibility of image ids between versions.

Is there are particular problem you're trying to address?

[ixmatus]

@stevvooe This came up because I've written a Haskell CLI utility to download, from docker-distribution, a container's layers and config JSON and assemble those artifacts into a docker image V1 tar archive that can be loaded by a daemon.

This meant I needed to re-implement the Chain ID and Content ID hashing schemes (it would be nice if these were documented in the V1 image spec, too) implemented in Golang by Docker in order to assemble the fetched artifacts correctly because I couldn't find a way to fetch any of that information from the registry (as I think you could in the V1, Python-based registry).

It doesn't appear to break anything if load an image produced by my tool but it's an unfortunate inconsistency because it means the hashes that are produced are idiosyncratic to the Go language environment, if I'm understanding you correctly. So, what I fetch and construct as an image and can load into the daemon will not be the same thing that the daemon produces when saving the same loaded image. That is unless I preserve the non-canonical sub-object key ordering (which is unspecified and subject to change based on decisions made in Golang's json module from what it sounds like) that is specific to the language environment the daemon is implemented in.

[stevvooe]

@ixmatus Make sure you are declaring to the registry that you accept schema2 manifests by including application/vnd.docker.distribution.manifest.v2+json in the Accept header (or fetch by digest). You'll get a manifest with a config descriptor with a digest pointing at the image config. That digest is the image id, or what I think you're calling the content id.

In general, when you download the content, you don't need to ever process the content through a json parser to calculate the various identifiers, other than to extract hash material for ChainID. Expecting json to round trip is a practice in futility 🐹. Just save the bytes, deserialize the portion you need for the id, then write the bytes to their destination.

If you could show me the code, I may be able to point out the issue.

Note: it is not called Golang.

[aaronlehmann]

Make sure you are declaring to the registry that you accept schema2 manifests by including application/vnd.docker.distribution.manifest.v2+json in the Accept header (or fetch by digest). You'll get a manifest with a config descriptor with a digest pointing at the image config. That digest is the image id, or what I think you're calling the content id.

This is a good suggestion, but note that you will only get such a manifest if a sufficiently recent version of Docker (>= 1.10) pushed one in this format. The registry doesn't convert from schema1 to schema2, to avoid affecting manifest digests.

In general, the need to create your own config JSON only arises with old-style schema1 manifests. This is an unavoidable consequence of the transition from the old, non-content-addressable scheme. But the current manifest format avoids the need to create JSON and the canonicalization frustrations that come with that.

[ixmatus]

@stevvooe yes I specifically download the V2 manifest because that points at the image's config JSON. It's sufficiently safe for me to do so because we only push from Docker (>= 1.10) to our private registry.

The tool downloads the image config and layers referenced in that manifest. Both artifacts are not enough to assemble a docker image though because the layer directory's name (which is a hash) is not described by the manifest or the image config json. The json file within each of those directories contains an id and a parent key with values that are the same as the directory's and the directory of the parent layer. These are also not obtained from the manifest or the image config json (the diff_ids key is the only piece of data describing the layers but its values are just the result of a sha256sum on the decompressed layer's tar archive).

In order to produce something that resembles what docker save creates I had to re-implement the Chain ID and Content ID hashing algorithms.

Is there a method I do not know about to get those values without the need to derive them?

If you could show me the code, I may be able to point out the issue.

I would be happy to share it once I have permission from my company to release it with an OSS license.

Expecting json to round trip is a practice in futility 🐹. Just save the bytes, deserialize the portion you need for the id, then write the bytes to their destination.

I may be misunderstanding how Docker is attempting to tackle addressable content hashes. Also, the problem from what I can tell is not specifically with the image's config JSON. I in-fact do as you're saying but the image config JSON has nothing to do with any of the layers except for the top-most layer (at least from what I understand of what's going on).

The problem, for me at least, is in generating the empty JSON config files which are used to produce the Content IDs for each layer using the Chain IDs.

Note: it is not called Golang.

Sure. I wrote it that way because Go when I read it in my head feels confusing but I don't have a problem not writing it that way.

But the current manifest format avoids the need to create JSON and the canonicalization frustrations that come with that.

@aaronlehmann I may not understand fully what is going on - what docker save outputs is highly specific to the v1 image spec which I assumed is important to follow so docker load can import an assembled image. The individual json files within each layer's directory of the image format are an important part of producing the Content ID hash which is the directory's name (and id and parent within each json file).

I could not find any other way of getting the same hash, I even tried a few different Accept: types to see if I could get a json file from the blob path for the layer as enumerated in the V2 manifest. Though, again, I may be missing something.

[aaronlehmann]

@ixmatus: Have a look at https://github.com/docker/docker/blob/master/image/spec/v1.2.md#combined-image-json--filesystem-changeset-format

This builds on top of the old docker save format to add the image config JSON. I believe (but would need to check) that Docker >= 1.10 will use this manifest.json file instead of the individual json files if it is present. Thus you shouldn't have to worry about matching hashes with the way Docker computes them.

[ixmatus]

@aaronlehmann interesting...it appears I'm doing both! Thank you for clarifying this for me.

[thaJeztah]

Looks like this question is answered, so I'll close the issue for housekeeping, but feel free to continue the discussion

[ixmatus]

@aaronlehmann @stevvooe FYI I have this working now with the feedback you provided me. So thank you.

Something we've now run across, which isn't entirely unexpected, is the time it takes to docker load ... fairly large images. While not the largest problem in the world, I've been thinking about how we could workaround that problem and because we're using NixOS I then thought, perhaps, that NixOS' nix store could serve as the backing store for the images that Docker references.

I don't fully understand Docker's storage model though and how deeply intertwined the stateful storage (devicemapper, etc.) functionality is with storage of image layers... Could you perhaps provide me with some guidance on what to look at or read?

[ixmatus]

I'm reading this, and it's helping: https://docs.docker.com/engine/userguide/storagedriver/imagesandcontainers/

[stevvooe]

@ixmatus https://medium.com/microscaling-systems/spot-the-docker-difference-9f99adcc4aaf#.jrhc8mp2k may provide some more insight. To build the level of understanding you're looking for, I'd recommend reading the code.

When analyzing the docker code base, I generally start from the handlers. The pull handler can be understood from https://github.com/docker/docker/blob/master/daemon/image_pull.go#L76. From there, focus on the interaction with the image store. Many of the design constraints will come to light with an understanding of interactions present there.

Let me know if you need further pointers.