Resource tries to pull image with incorrect digest

[martinsbalodis]

Hi!
I am running fly execute but it fails while pulling a docker image. From the output I can see that it tries to pull a version of a container that has never existed in the repository (private http repository).

Here is the task.yml part:

...
image_resource:
  type: docker-image
  source:
   repository: 1.2.3.4:5000/my-image
   insecure_registries: ["1.2.3.4:5000"]
   email: martins@example.com
   username: martins
   password: pass
...

Fly command:

fly -t lc execute -c task.yml

Output of the fly command:

...
initializing
WARNING: login credentials saved in /root/.docker/config.json
Login Succeeded
Pulling 1.2.3.4:5000/my-image@sha256:59795eb87abd170f1b97f82f5838db5637890962d7bd623f5ce04c7833ef29f0...
Error response from daemon: manifest unknown: manifest unknown

Pulling 1.2.3.4:5000/my-image@sha256:59795eb87abd170f1b97f82f5838db5637890962d7bd623f5ce04c7833ef29f0 (attempt 2 of 3)...
Error response from daemon: manifest unknown: manifest unknown

Pulling 1.2.3.4:5000/my-image@sha256:59795eb87abd170f1b97f82f5838db5637890962d7bd623f5ce04c7833ef29f0 (attempt 3 of 3)...
Error response from daemon: manifest unknown: manifest unknown


Failed to pull image 1.2.3.4:5000/my-image@sha256:59795eb87abd170f1b97f82f5838db5637890962d7bd623f5ce04c7833ef29f0.resource script '/opt/resource/in [/tmp/build/get]' failed: exit status 1
errored

The docker registry doesn't contain a version of this image that has sha256 like 59795eb87abd. I also checked the directory where the images are stored to make sure.

[concourse-bot]

Hi there!

We use Pivotal Tracker to provide visibility into what our team is working on. A story for this issue has been automatically created.

The current status is as follows:

• #117790211 Resource tries to pull image with incorrect digest

This comment, as well as the labels on the issue, will be automatically updated as the status in Tracker changes.

[vito]

Don't really know what to tell ya. We collect these versions from the registry API and then just go back and try to fetch them from the same registry. Sounds like your registry is broken or inconsistent? Are you running a v2 registry?

[databus23]

I've been hit by this too, today when we upgraded the registry from from 2.3.0 to 2.4.0.

@martinsbalodis Are you be any chance running a registry version >= 2.3.1?

I think this change is causing the problem: distribution/distribution@d7eb5d1

It seems the docker-image check method is fetching v1 digests but docker pull expects v2 digests.

Running the docker-image resource returns a digest that can't be pulled:

echo '{"source":{"repository":"docker.mo.sap.corp/concourse/monsoon-version","tag":"latest"}}'| docker run --rm -i docker-image-resource /opt/resource/check
[{"digest":"sha256:a8348e02225831b1b78b54654e78ae480430f4f652f145024859bfdb3ffb6dad"}]

docker pull docker.mo.sap.corp/concourse/monsoon-version@sha256:a8348e02225831b1b78b54654e78ae480430f4f652f145024859bfdb3ffb6dad
Error response from daemon: manifest unknown: manifest unknown

Fetching the manifest via curl gets me the same hash which is the v1 manifest digest

$> curl -I https://docker.mo.sap.corp/v2/concourse/monsoon-version/manifests/latest
HTTP/1.1 200 OK
Content-Length: 9314
Content-Type: application/vnd.docker.distribution.manifest.v1+prettyjws
Docker-Content-Digest: sha256:a8348e02225831b1b78b54654e78ae480430f4f652f145024859bfdb3ffb6dad
Docker-Distribution-Api-Version: registry/2.0
Etag: "sha256:a8348e02225831b1b78b54654e78ae480430f4f652f145024859bfdb3ffb6dad"
X-Content-Type-Options: nosniff
Date: Tue, 19 Apr 2016 13:10:13 GMT

When explicitly requesting the v2 manifest header I get different digest which can be pulled

$> curl -I -H 'Accept: application/vnd.docker.distribution.manifest.v2+json' https://docker.mo.sap.corp/v2/concourse/monsoon-version/manifests/latest
HTTP/1.1 200 OK
Content-Length: 2802
Content-Type: application/vnd.docker.distribution.manifest.v2+json
Docker-Content-Digest: sha256:9aeec19e23ad3e57fd872df6493bfc5f392d10b0e2fab25e46c7a18f2c93f517
Docker-Distribution-Api-Version: registry/2.0
Etag: "sha256:9aeec19e23ad3e57fd872df6493bfc5f392d10b0e2fab25e46c7a18f2c93f517"
X-Content-Type-Options: nosniff
Date: Tue, 19 Apr 2016 13:10:47 GMT

docker pull docker.mo.sap.corp/concourse/monsoon-version@sha256:9aeec19e23ad3e57fd872df6493bfc5f392d10b0e2fab25e46c7a18f2c93f517
sha256:9aeec19e23ad3e57fd872df6493bfc5f392d10b0e2fab25e46c7a18f2c93f517: Pulling from concourse/monsoon-version
Digest: sha256:9aeec19e23ad3e57fd872df6493bfc5f392d10b0e2fab25e46c7a18f2c93f517
Status: Downloaded newer image for docker.mo.sap.corp/concourse/monsoon-version@sha256:9aeec19e23ad3e57fd872df6493bfc5f392d10b0e2fab25e46c7a18f2c93f517

[martinsbalodis]

I think I found the problem. The docker-image-resource might be looking at the wrong digest. I dumped the communication between docker-registry and docker-image-resource with tcpdump. Below is a request that is made by the docker-image-resource client. It seems that it is requesting the manifest of the image. The docker-registry responds with a Docker-Content-Digest header. This digest is then used by the docker-image-resource as the required digest for image. I think this digest is the digest of the response message and not the image.

GET /v2/docker-image-name/manifests/latest HTTP/1.1
Host: 1.2.3.4:5000
User-Agent: Go-http-client/1.1
Authorization: Basic asd
Accept-Encoding: gzip
Connection: close

HTTP/1.1 200 OK
Content-Length: 20775
Content-Type: application/vnd.docker.distribution.manifest.v1+prettyjws
Docker-Content-Digest: sha256:59795eb87abd170f1b97f82f5838db5637890962d7bd623f5ce04c7833ef29f0
Docker-Distribution-Api-Version: registry/2.0
Etag: "sha256:59795eb87abd170f1b97f82f5838db5637890962d7bd623f5ce04c7833ef29f0"
X-Content-Type-Options: nosniff
Date: Wed, 20 Apr 2016 07:48:46 GMT
Connection: close

{
   "schemaVersion": 1,
   "name": "docker-image-name",
   "tag": "latest",
   "architecture": "amd64",
   "fsLayers": [
      {
         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
      },
...

The next request made to the docker-registry looks like this:

GET /v2/docker-image-name/manifests/sha256:59795eb87abd170f1b97f82f5838db5637890962d7bd623f5ce04c7833ef29f0 HTTP/1.1

[databus23]

@martinsbalodis The Docker-Content-Digest header of a manifest request is the digest that is needed to pull an image by digest. This is not the problem. The problem is that there are different schema versions of an image manifest yielding different digests. The docker-image-resource is not requesting the correct version for newer registries (>=2.3.1). See my previous post for more details.

[vito]

Closing this out as @databus23's PR was merged (thanks!). This will be in the next release.