SO, what _is_ the way to compute the docker image id from a v2 registry?

[dcowden]

Hey, sorry: not really an 'issue', but 👍

I saw in this post:
http://stackoverflow.com/questions/34568746/what-is-the-relationship-between-a-docker-image-id-and-the-ids-in-the-manifests

That i'm trying to do what you have evidently already done: figure out how to take a locally generated Docker 1.10 'imageID' for a given imae name and tag 'myapp:1.7' and then find out whether the registry version of myapp:1.7 points to a different image?

I gathered from your post that the magic answer is to do HEAD /v2/myapp/manifests/1.7 and then look for the docker-content-digest header, and that should match what docker says is the image id right?

Somehow this isnt the case for me. when i do a HEAD request, i do not get the header back. When i do a GET request, i get the header, but the digest returned doesnt match my image id.

Have you figured this out?

[deitch]

Hey @dcowden , I really should just link the SO post and the repo, shouldn't I? :-)

I had banged my head on it for a while. Mainly, I wanted to be able to pull an image directly, which meant comparing it. I started to do that, but never got around to finishing it.

I gathered from your post that the magic answer is to do HEAD /v2/myapp/manifests/1.7 and then look for the docker-content-digest header, and that should match what docker says is the image id right?

I think your description is correct. See from https://gist.github.com/aaronlehmann/b42a2eaf633fc949f93b#new-image-config :

The image configuration JSON is what gets hashed to generate an image ID.

Tell me how you are testing it?

[deitch]

OK, so I just did another test. I just checked the hashes for deitch/mysql-backup:0.7.0. Here is what I got. FYI, running docker 1.10.0, and removed all containers and images, so clean slate.

• pull the image:

$ docker pull deitch/mysql-backup:0.7.0
0.7.0: Pulling from deitch/mysql-backup
e49bb2b1d8f0: Pull complete
9d64e2baf253: Pull complete
dd8ed7a8f3de: Pull complete
Digest: sha256:8a069edacec55b930b983a7c740b9802696cdc70acadfa249007950475f033ff
Status: Downloaded newer image for deitch/mysql-backup:0.7.0

So it shows the digest as sha256:8a069edacec55b930b983a7c740b9802696cdc70acadfa249007950475f033ff

• Check the local image via docker images:

$ docker images
REPOSITORY            TAG                 IMAGE ID            CREATED             SIZE
deitch/mysql-backup   0.7.0               c37d6f99d227        5 weeks ago         126.9 MB

Hmm, that definitely is not the same.

• Check the local image via inspect:

$ docker inspect deitch/mysql-backup:0.7.0 | head -6
[
    {
        "Id": "sha256:c37d6f99d22771d16b09ec67d29e0f7327b018d4793415acb79d954967249f71",
        "RepoTags": [
            "deitch/mysql-backup:0.7.0"
        ],

That matches docker images but not the sha256 reported by docker pull.

• Check the header on GET https://registry.hub.docker.com/v2/deitch/mysql-backup/manifests/0.7.0:

irb(main):013:0> ret.headers
=> {:content_length=>"4437", :content_type=>"application/vnd.docker.distribution.manifest.v1+prettyjws", :docker_content_digest=>"sha256:8a069edacec55b930b983a7c740b9802696cdc70acadfa249007950475f033ff", :docker_distribution_api_version=>"registry/2.0", :etag=>"\"sha256:8a069edacec55b930b983a7c740b9802696cdc70acadfa249007950475f033ff\"", :date=>"Thu, 21 Apr 2016 09:21:16 GMT", :strict_transport_security=>"max-age=31536000"}

So we have 2 hashes here:

1. The one reported by the header, which matches what docker pull gives
2. The one reported by docker images, which matches what docker inspect gives.

I hoped @thaJeztah would help there...

[dcowden]

Yes, that's exactly what i found as well. i was able to see that the digest
from docker pull matched what was in the header, but it didnt match what
docker reports as the 'imageID'.

So at this point, my conclusion is that docker has completely broken the
ability to determine if a local image is up-to date or not.

My objective is to determine if a local image ( pulled via tag) is up to
date. At first i thought maybe docker stores the digest somewhere when i
pull the image. But i found another discussion about the decision NOT to
store the digest locally if the image was pulled via tags, so the digest
doesnt appear anywhere in docker inspect either.

So at this point, I've concluded that there is literally no way to
determine if your local image is up to date except to store the digest when
you pull yourself, and then look at the remote header. Is that your
conclusion as well?

On Thu, Apr 21, 2016 at 5:32 AM, Avi Deitcher notifications@github.com
wrote:

OK, so I just did another test. I just checked the hashes for
deitch/mysql-backup:0.7.0. Here is what I got. FYI, running docker
1.10.0, and removed all containers and images, so clean slate.

1. pull the image:

$ docker pull deitch/mysql-backup:0.7.0
0.7.0: Pulling from deitch/mysql-backup
e49bb2b1d8f0: Pull complete
9d64e2baf253: Pull complete
dd8ed7a8f3de: Pull complete
Digest: sha256:8a069edacec55b930b983a7c740b9802696cdc70acadfa249007950475f033ff
Status: Downloaded newer image for deitch/mysql-backup:0.7.0````

So it shows the digest as sha256:8a069edacec55b930b983a7c740b9802696cdc70acadfa249007950475f033ff

1. Check the local image via docker images:

$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
deitch/mysql-backup 0.7.0 c37d6f99d227 5 weeks ago 126.9 MB

Hmm, that definitely is not the same.

1. Check the local image via inspect:

$ docker inspect deitch/mysql-backup:0.7.0 | head -6
[
{
"Id":
"sha256:c37d6f99d22771d16b09ec67d29e0f7327b018d4793415acb79d954967249f71",
"RepoTags": [
"deitch/mysql-backup:0.7.0"
],

That matches docker images but not the sha256 reported by docker pull.

1. Check the header on GET https://registry.hub.docker.com/v2/deitch/mysql-backup/manifests/0.7.0 https://registry.hub.docker.com/v2/deitch/mysql-backup/manifests/0.7.0:

irb(main):013:0> ret.headers
=> {:content_length=>"4437",
:content_type=>"application/vnd.docker.distribution.manifest.v1+prettyjws",
:docker_content_digest=>"sha256:8a069edacec55b930b983a7c740b9802696cdc70acadfa249007950475f033ff",
:docker_distribution_api_version=>"registry/2.0",
:etag=>""sha256:8a069edacec55b930b983a7c740b9802696cdc70acadfa249007950475f033ff"",
:date=>"Thu, 21 Apr 2016 09:21:16 GMT",
:strict_transport_security=>"max-age=31536000"}

That sha256 hash matches what docker pull reported, but not what docker images gives.

Huh.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#4 (comment)

[deitch]

No, it cannot be. Because if you do docker pull someImage, it checks to see if it is up to date with the registry using the same API that we see, and checking hashes. It does not care about the tag.

The entire comparison is between the contents and headers of the manifest retrieved via GET /v2/repo/manifests/tag and what is in /var/lib/docker

If you look in /var/lib/docker/image/aufs/imagedb/content/sha256/ and then by the image ID as reported by docker images (in the example above, c37....), you will see what looks like a manifest, but is distinctly different than what is returned from the GET.

Even more interesting, what is returned is giving a "schemaVersion":"1".

Really having a hard time making heads or tails of this.

[dcowden]

It is certainly possible that the docker client computes an ID which is
completely unrelated to the digest.

My guess is that what's happening is that its storing the digests
somehwere, and using that to compare with the remote manifest, but that
imageID is not used as a part of the comparison?

On Thu, Apr 21, 2016 at 9:03 AM, Avi Deitcher notifications@github.com
wrote:

No, it cannot be. Because if you do docker pull someImage, it checks to
see if it is up to date with the registry using the same API that we see,
and checking hashes. It does not care about the tag.

The entire comparison is between the contents and headers of the manifest
retrieved via GET /v2/repo/manifests/tag and what is in /var/lib/docker

If you look in /var/lib/docker/image/aufs/imagedb/content/sha256/ and
then by the image ID as reported by docker images (in the example above,
c37....), you will see what looks like a manifest, but is distinctly
different than what is returned from the GET.

Even more interesting, what is returned is giving a "schemaVersion":"1".

Really having a hard time making heads or tails of this.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#4 (comment)

[deitch]

It is possible. But I would expect that to be in the header... oh, wait, right. We are trying to find something in the content header. But then what in the world is the image id??

[deitch]

And why am I getting back "schemaVersion":1, and most importantly.... why is this all so opaque??

[dcowden]

well to some extent i think the opaqueness is by design, the v2 api was all
about security.

I've learned that the digest is actually a sha 256 hash of the manifiest,
excluding the signature portion.

But then what in the world is the image id??

​Exactly. I cannot figure that out either. Its the hash of something. but
it is not the digest, and it is not the hash of any of the individual
layers. I would post on SO about it, but you essentially already have.​

On Thu, Apr 21, 2016 at 9:17 AM, Avi Deitcher notifications@github.com
wrote:

And why am I getting back "schemaVersion":1, and most importantly.... why
is this all so opaque??

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#4 (comment)

[deitch]

I would post on SO about it, but you essentially already have.​

And we see how far that has gotten. :-)

But what version of the schema? V2 registry has v1 schema and v2 schema. And what parts do you hash?

I fail to understand why this cannot simply be written up.

[deitch]

I am going to open a new issue. Here:

distribution/distribution#1662

I guess we always could look at the docker source code....

[dcowden]

awesome thanks, i'll watch that one.

On Thu, Apr 21, 2016 at 9:30 AM, Avi Deitcher notifications@github.com
wrote:

I am going to open a new issue. Here:

distribution/distribution#1662
distribution/distribution#1662

I guess we always could look at the docker source code....

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#4 (comment)