Pushing an image can fail with schema 2 disabled and pull through enabled

[ncdc]

From test/extended/cmd.sh:

docker login -u imagensbuilder -p ${token} -e fake@example.org ${docker_registry}"
oc import-image busybox:latest --confirm"
docker pull busybox"
docker tag docker.io/busybox:latest ${docker_registry}/image-ns/busybox:latest"
docker push ${docker_registry}/image-ns/busybox:latest"

If schema 2 is disabled and pull through is enabled (which is the default), I'm seeing the attempt to docker push failing.

The schema 2 busybox manifest has just 1 layer blob, while the schema 1 busybox manifest has 2 layer blobs.

What happens is this:

1. docker daemon uploads the 1 layer blob referenced by the schema 2 manifest for busybox
2. docker daemon uploads the 1 config manifest blob referenced by the schema 2 manifest for busybox
3. docker daemon attempts to upload the schema 2 manifest for busybox
4. registry rejects schema 2 manifest because schema 2 is disabled
5. docker daemon falls back to uploading a schema 1 version of the manifest
6. docker daemon issues a HEAD request for one of the layer blobs in the schema 1 manifest
7. registry reports 200 and starts to download the blob in the background
8. docker daemon tries to upload the schema 1 manifest
9. registry tries to stat the blob that's being pulled through and fails because it can't find it
10. registry rejects push with 'blob unknown'

I did see in the registry logs attempts to stat layer links and/or layers. @smarterclayton mentioned we shouldn't need to do that any more?

cc @legionus @miminar @soltysh @mfojtik

[smarterclayton]

I would expect that the registry would stat the blob through the cache and fall back to lookup on the image stream (where it would find the layer). On Jan 4, 2017, at 4:26 PM, Andy Goldstein <notifications@github.com> wrote: From test/extended/cmd.sh: docker login -u imagensbuilder -p ${token} -e fake@example.org ${docker_registry}" oc import-image busybox:latest --confirm" docker pull busybox" docker tag docker.io/busybox:latest ${docker_registry}/image-ns/busybox:latest" docker push ${docker_registry}/image-ns/busybox:latest" If schema 2 is disabled and pull through is enabled (which is the default), I'm seeing the attempt to docker push failing. The schema 2 busybox manifest has just 1 layer blob, while the schema 1 busybox manifest has 2 layer blobs. What happens is this: 1. docker daemon uploads the 1 layer blob referenced by the schema 2 manifest for busybox 2. docker daemon uploads the 1 config manifest blob referenced by the schema 2 manifest for busybox 3. docker daemon attempts to upload the schema 2 manifest for busybox 4. registry rejects schema 2 manifest because schema 2 is disabled 5. docker daemon falls back to uploading a schema 1 version of the manifest 6. docker daemon issues a HEAD request for one of the layer blobs in the schema 1 manifest 7. registry reports 200 and starts to download the blob in the background 8. docker daemon tries to upload the schema 1 manifest 9. registry tries to stat the blob that's being pulled through and fails because it can't find it 10. registry rejects push with 'blob unknown' I did see in the registry logs attempts to stat layer links and/or layers. @smarterclayton <https://github.com/smarterclayton> mentioned we shouldn't need to do that any more? cc @legionus <https://github.com/legionus> @miminar <https://github.com/miminar> @soltysh <https://github.com/soltysh> @mfojtik <https://github.com/mfojtik> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#12389>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABG_py5BftXQNpE-G5NX4M1LZZ58ntGcks5rPA6cgaJpZM4LbFae> .

[miminar]

A duplicate of #12329. @legionus is already working on it. Thanks for the thorough description!

As discussed with @legionus, instead of using upstream repository to stat the blob locally, we should use our middleware repository, which will go through the cache, attempt local stat and then fallback to pullthrough stat.

@smarterclayton by the way, I presume mirroring the blobs on HEAD (an existence check) request was not your original intention, right? You've mentioned range request for that purpose.