Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Is 2.0.0 broken for Quay.io? #156

Closed
tboerger opened this issue Mar 7, 2022 · 10 comments · Fixed by #159
Closed

Is 2.0.0 broken for Quay.io? #156

tboerger opened this issue Mar 7, 2022 · 10 comments · Fixed by #159

Comments

@tboerger
Copy link

tboerger commented Mar 7, 2022

I'm using the manifest-tool already quite long and never had any issues with it until I updated to 2.0.0. With this version I'm receiving an error while trying to push the manifest to Quay:

time="2022-03-07T11:34:32Z" level=fatal msg="Error pushing manifest list/index to registry: sha256:8d63af9ef7ceccfc644d2507fae12251cfbeb05070e1af80525cc6cf6721311a: failed commit on ref \"manifest-sha256:a31e11ed2de7e55e5c9eb5697b88913a4d8f1209283694e04b8399baaa9a4009\": cannot reuse body, request must be retried"

Is there anything I can fix on my side? Or is Quay really currently broken with manifest-tool?
@estesp
Copy link
Owner

estesp commented Mar 7, 2022

I've tried to reproduce this without success; can you provide some more data; maybe with --debug? My local attempts with both main's HEAD commit (a few vendored updates since 2.0 release) as well as the exact v2.0.0 binary are below. I'm only including the push of the final manifest list entry with --debug to validate it pushed to quay:

ubuntu@devvm:~/go/src/github.com/estesp/manifest-tool/integration$ ./test-registry.sh quay.io/estesp
Warning: some commands will fail if you are not authenticated to quay.io/estesp
[ skipping lots of debug output ]
DEBU[0002] checking and pushing to                       digest="sha256:ab6a6c4d9ae12b0040b5d19c115f1666590c18e90124aa69ed95a802604b37d7" mediatype=application/vnd.docker.distribution.manifest.list.v2+json size=1272 url="https://quay.io/v2/estesp/alpine/manifests/latest"
DEBU[0002] do request                                    digest="sha256:ab6a6c4d9ae12b0040b5d19c115f1666590c18e90124aa69ed95a802604b37d7" mediatype=application/vnd.docker.distribution.manifest.list.v2+json request.header.accept="application/vnd.docker.distribution.manifest.list.v2+json, */*" request.header.user-agent=containerd/1.5.9+unknown request.method=HEAD size=1272 url="https://quay.io/v2/estesp/alpine/manifests/latest"
DEBU[0003] fetch response received                       digest="sha256:ab6a6c4d9ae12b0040b5d19c115f1666590c18e90124aa69ed95a802604b37d7" mediatype=application/vnd.docker.distribution.manifest.list.v2+json response.header.content-length=132 response.header.content-type=application/json response.header.date="Mon, 07 Mar 2022 18:48:04 GMT" response.header.server=nginx/1.14.1 response.status="404 Not Found" size=1272 url="https://quay.io/v2/estesp/alpine/manifests/latest"
DEBU[0003] do request                                    digest="sha256:ab6a6c4d9ae12b0040b5d19c115f1666590c18e90124aa69ed95a802604b37d7" mediatype=application/vnd.docker.distribution.manifest.list.v2+json request.header.content-type=application/vnd.docker.distribution.manifest.list.v2+json request.header.user-agent=containerd/1.5.9+unknown request.method=PUT size=1272 url="https://quay.io/v2/estesp/alpine/manifests/latest"
DEBU[0003] fetch response received                       digest="sha256:ab6a6c4d9ae12b0040b5d19c115f1666590c18e90124aa69ed95a802604b37d7" mediatype=application/vnd.docker.distribution.manifest.list.v2+json response.header.content-length=2 response.header.content-type="text/html; charset=utf-8" response.header.date="Mon, 07 Mar 2022 18:48:04 GMT" response.header.docker-content-digest="sha256:ab6a6c4d9ae12b0040b5d19c115f1666590c18e90124aa69ed95a802604b37d7" response.header.location="https://quay.io/v2/estesp/alpine/manifests/sha256:ab6a6c4d9ae12b0040b5d19c115f1666590c18e90124aa69ed95a802604b37d7" response.header.server=nginx/1.14.1 response.header.strict-transport-security="max-age=63072000; preload" response.header.x-frame-options=DENY response.status="201 Created" size=1272 url="https://quay.io/v2/estesp/alpine/manifests/latest"
Digest: sha256:ab6a6c4d9ae12b0040b5d19c115f1666590c18e90124aa69ed95a802604b37d7 1272

I then inspected this new image (just FYI: I deleted all prior tags/shas from the "estesp" repo before the above step, and also tested with existing tags in place) to verify it pushed properly:

$ manifest-tool inspect quay.io/estesp/alpine:latest
Name:   quay.io/estesp/alpine:latest (Type: application/vnd.docker.distribution.manifest.list.v2+json)
Digest: sha256:ab6a6c4d9ae12b0040b5d19c115f1666590c18e90124aa69ed95a802604b37d7
 * Contains 4 manifest references:
[1]     Type: application/vnd.docker.distribution.manifest.v2+json
[1]   Digest: sha256:2042a492bcdd847a01cd7f119cd48caa180da696ed2aedd085001a78664407d6
[1]   Length: 528
[1] Platform:
[1]    -      OS: linux
[1]    -    Arch: ppc64le
[1] # Layers: 1
     layer 01: digest = sha256:159b5dcb1717c815c76ff5ea1db730e18e8609c9090238e43282856db9e71f47

[2]     Type: application/vnd.docker.distribution.manifest.v2+json
[2]   Digest: sha256:e7d88de73db3d3fd9b2d63aa7f447a10fd0220b7cbf39803c803f2af9ba256b3
[2]   Length: 528
[2] Platform:
[2]    -      OS: linux
[2]    -    Arch: amd64
[2] # Layers: 1
     layer 01: digest = sha256:59bf1c3509f33515622619af21ed55bbe26d24913cedbca106468a5fb37a50c3

[3]     Type: application/vnd.docker.distribution.manifest.v2+json
[3]   Digest: sha256:49e322ab6690e73a4909f787bcbdb873631264ff4a108cddfd9f9c249ba1d58e
[3]   Length: 528
[3] Platform:
[3]    -      OS: linux
[3]    -    Arch: s390x
[3] # Layers: 1
     layer 01: digest = sha256:d6baca485f3d0f7c77221be60fbef5db014a5ef9d8f53db4a310c947c690d189

[4]     Type: application/vnd.docker.distribution.manifest.v2+json
[4]   Digest: sha256:6ace9a4351a3f02894c273cee7adc519318d6186f0d364fa37e418394576c5ef
[4]   Length: 528
[4] Platform:
[4]    -      OS: linux
[4]    -    Arch: arm64
[4]    - Variant: v8
[4] # Layers: 1
     layer 01: digest = sha256:bc1d962af60541fd028aaa6f21d62a662d718b3571577acf3fb65c04a61b1150

@mthalman
Copy link

mthalman commented Mar 7, 2022

I have the exact opposite problem with Quay. It works with 2.0.0 but fails with 1.0.3:

$ manifest-tool inspect quay.io/centos/centos:stream9-development
time="2022-03-07T19:30:15Z" level=error msg="Error trying v2 registry: unsupported manifest format"
time="2022-03-07T19:30:15Z" level=fatal msg="unsupported manifest format"

@estesp
Copy link
Owner

estesp commented Mar 7, 2022

I have the exact opposite problem with Quay. It works with 2.0.0 but fails with 1.0.3:

Yeah, I didn't have OCIv1 image spec support in the 1.x branch at all :( The CentOS images are true OCI images:

  • Name: quay.io/centos/centos:stream9-development (Type: application/vnd.oci.image.index.v1+json)
  • Digest: sha256:1c8b23de38720af98c65663f231e4ee79f38bbc49a4558dee57ec8b85ac284f9

@mthalman mthalman mentioned this issue Mar 7, 2022
Merged
@tboerger
Copy link
Author

tboerger commented Mar 9, 2022 •
edited

I have executed the manifest-tool now manually with an older version of my images. I tried it with the existing manifest and I also tried it after deleting the manifest, same results:

$ docker run --rm mplatform/manifest-tool --debug --username $QUAY_USERNAME --password $QUAY_PASSWORD push from-args --platforms linux/amd64,linux/arm64,linux/arm/v6 --template quay.io/webhippie/alpine:3.6-ARCH --target quay.io/webhippie/alpine:3.6
time="2022-03-09T08:00:18Z" level=info msg="Retrieving digests of member images"
time="2022-03-09T08:00:18Z" level=debug msg=resolving host=quay.io
time="2022-03-09T08:00:18Z" level=debug msg="do request" host=quay.io request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/1.5.9+unknown request.method=HEAD url="https://quay.io/v2/webhippie/alpine/manifests/3.6-amd64"
time="2022-03-09T08:00:19Z" level=debug msg="fetch response received" host=quay.io response.header.content-length=1367 response.header.content-type=application/vnd.docker.distribution.manifest.v2+json response.header.date="Wed, 09 Mar 2022 08:00:19 GMT" response.header.docker-content-digest="sha256:3fbd95657a45dfed369740b70575f19dd5363f96474dbc2f57ed4367c637bcf2" response.header.server=nginx/1.14.1 response.header.strict-transport-security="max-age=63072000; preload" response.header.x-frame-options=DENY response.status="200 OK" url="https://quay.io/v2/webhippie/alpine/manifests/3.6-amd64"
time="2022-03-09T08:00:19Z" level=debug msg=resolved desc.digest="sha256:3fbd95657a45dfed369740b70575f19dd5363f96474dbc2f57ed4367c637bcf2" host=quay.io
time="2022-03-09T08:00:19Z" level=debug msg=fetch digest="sha256:3fbd95657a45dfed369740b70575f19dd5363f96474dbc2f57ed4367c637bcf2" mediatype=application/vnd.docker.distribution.manifest.v2+json size=1367
time="2022-03-09T08:00:19Z" level=debug msg="do request" digest="sha256:3fbd95657a45dfed369740b70575f19dd5363f96474dbc2f57ed4367c637bcf2" mediatype=application/vnd.docker.distribution.manifest.v2+json request.header.accept="application/vnd.docker.distribution.manifest.v2+json, */*" request.header.user-agent=containerd/1.5.9+unknown request.method=GET size=1367 url="https://quay.io/v2/webhippie/alpine/manifests/sha256:3fbd95657a45dfed369740b70575f19dd5363f96474dbc2f57ed4367c637bcf2"
time="2022-03-09T08:00:19Z" level=debug msg="fetch response received" digest="sha256:3fbd95657a45dfed369740b70575f19dd5363f96474dbc2f57ed4367c637bcf2" mediatype=application/vnd.docker.distribution.manifest.v2+json response.header.content-length=1367 response.header.content-type=application/vnd.docker.distribution.manifest.v2+json response.header.date="Wed, 09 Mar 2022 08:00:19 GMT" response.header.docker-content-digest="sha256:3fbd95657a45dfed369740b70575f19dd5363f96474dbc2f57ed4367c637bcf2" response.header.server=nginx/1.14.1 response.header.strict-transport-security="max-age=63072000; preload" response.header.x-frame-options=DENY response.status="200 OK" size=1367 url="https://quay.io/v2/webhippie/alpine/manifests/sha256:3fbd95657a45dfed369740b70575f19dd5363f96474dbc2f57ed4367c637bcf2"
time="2022-03-09T08:00:19Z" level=debug msg=fetch digest="sha256:9e8c361252e68d2dd89150eff2242a8f67960d4f42b3c87bc75691aa0e03878d" mediatype=application/vnd.docker.container.image.v1+json size=2925
time="2022-03-09T08:00:19Z" level=debug msg="do request" digest="sha256:9e8c361252e68d2dd89150eff2242a8f67960d4f42b3c87bc75691aa0e03878d" mediatype=application/vnd.docker.container.image.v1+json request.header.accept="application/vnd.docker.container.image.v1+json, */*" request.header.user-agent=containerd/1.5.9+unknown request.method=GET size=2925 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:9e8c361252e68d2dd89150eff2242a8f67960d4f42b3c87bc75691aa0e03878d"
time="2022-03-09T08:00:19Z" level=debug msg="fetch response received" digest="sha256:9e8c361252e68d2dd89150eff2242a8f67960d4f42b3c87bc75691aa0e03878d" mediatype=application/vnd.docker.container.image.v1+json response.header.accept-ranges=bytes response.header.age=632 response.header.content-length=2925 response.header.content-type=binary/octet-stream response.header.date="Wed, 09 Mar 2022 07:49:48 GMT" response.header.etag="\"506d261ba5c9f1806e97cc85ae78e377-1\"" response.header.last-modified="Wed, 09 Mar 2022 03:35:41 GMT" response.header.server=AmazonS3 response.header.via="1.1 1a45d1e1304c39dfa9b034c2308f4976.cloudfront.net (CloudFront)" response.header.x-amz-cf-id="YjMRJIx7uHMhW72Hubr1VGwZP2ZdhndcTTjYaC7bMqcHqDA8lEIGcg==" response.header.x-amz-cf-pop=MUC50-P1 response.header.x-amz-replication-status=COMPLETED response.header.x-amz-server-side-encryption=AES256 response.header.x-amz-version-id=f59qh4eKw_GAZwMOE5jAuiFP9hQD2N24 response.header.x-cache="Hit from cloudfront" response.status="200 OK" size=2925 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:9e8c361252e68d2dd89150eff2242a8f67960d4f42b3c87bc75691aa0e03878d"
time="2022-03-09T08:00:19Z" level=debug msg=resolving host=quay.io
time="2022-03-09T08:00:19Z" level=debug msg="do request" host=quay.io request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/1.5.9+unknown request.method=HEAD url="https://quay.io/v2/webhippie/alpine/manifests/3.6-arm64"
time="2022-03-09T08:00:19Z" level=debug msg="fetch response received" host=quay.io response.header.content-length=1367 response.header.content-type=application/vnd.docker.distribution.manifest.v2+json response.header.date="Wed, 09 Mar 2022 08:00:19 GMT" response.header.docker-content-digest="sha256:316470549e4eb403bf2c67c454cdc644b99dba23d625c299e75ebd00a02fe5e3" response.header.server=nginx/1.14.1 response.header.strict-transport-security="max-age=63072000; preload" response.header.x-frame-options=DENY response.status="200 OK" url="https://quay.io/v2/webhippie/alpine/manifests/3.6-arm64"
time="2022-03-09T08:00:19Z" level=debug msg=resolved desc.digest="sha256:316470549e4eb403bf2c67c454cdc644b99dba23d625c299e75ebd00a02fe5e3" host=quay.io
time="2022-03-09T08:00:19Z" level=debug msg=fetch digest="sha256:316470549e4eb403bf2c67c454cdc644b99dba23d625c299e75ebd00a02fe5e3" mediatype=application/vnd.docker.distribution.manifest.v2+json size=1367
time="2022-03-09T08:00:19Z" level=debug msg="do request" digest="sha256:316470549e4eb403bf2c67c454cdc644b99dba23d625c299e75ebd00a02fe5e3" mediatype=application/vnd.docker.distribution.manifest.v2+json request.header.accept="application/vnd.docker.distribution.manifest.v2+json, */*" request.header.user-agent=containerd/1.5.9+unknown request.method=GET size=1367 url="https://quay.io/v2/webhippie/alpine/manifests/sha256:316470549e4eb403bf2c67c454cdc644b99dba23d625c299e75ebd00a02fe5e3"
time="2022-03-09T08:00:19Z" level=debug msg="fetch response received" digest="sha256:316470549e4eb403bf2c67c454cdc644b99dba23d625c299e75ebd00a02fe5e3" mediatype=application/vnd.docker.distribution.manifest.v2+json response.header.content-length=1367 response.header.content-type=application/vnd.docker.distribution.manifest.v2+json response.header.date="Wed, 09 Mar 2022 08:00:19 GMT" response.header.docker-content-digest="sha256:316470549e4eb403bf2c67c454cdc644b99dba23d625c299e75ebd00a02fe5e3" response.header.server=nginx/1.14.1 response.header.strict-transport-security="max-age=63072000; preload" response.header.x-frame-options=DENY response.status="200 OK" size=1367 url="https://quay.io/v2/webhippie/alpine/manifests/sha256:316470549e4eb403bf2c67c454cdc644b99dba23d625c299e75ebd00a02fe5e3"
time="2022-03-09T08:00:19Z" level=debug msg=fetch digest="sha256:022c77f5ae044caad4786fa51b28cdb27e5406120664c62f05b41c336138ee38" mediatype=application/vnd.docker.container.image.v1+json size=2925
time="2022-03-09T08:00:19Z" level=debug msg="do request" digest="sha256:022c77f5ae044caad4786fa51b28cdb27e5406120664c62f05b41c336138ee38" mediatype=application/vnd.docker.container.image.v1+json request.header.accept="application/vnd.docker.container.image.v1+json, */*" request.header.user-agent=containerd/1.5.9+unknown request.method=GET size=2925 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:022c77f5ae044caad4786fa51b28cdb27e5406120664c62f05b41c336138ee38"
time="2022-03-09T08:00:19Z" level=debug msg="fetch response received" digest="sha256:022c77f5ae044caad4786fa51b28cdb27e5406120664c62f05b41c336138ee38" mediatype=application/vnd.docker.container.image.v1+json response.header.accept-ranges=bytes response.header.age=632 response.header.content-length=2925 response.header.content-type=binary/octet-stream response.header.date="Wed, 09 Mar 2022 07:49:49 GMT" response.header.etag="\"2c704921fb41bcada721b0b741ae5141-1\"" response.header.last-modified="Wed, 09 Mar 2022 03:35:56 GMT" response.header.server=AmazonS3 response.header.via="1.1 1a45d1e1304c39dfa9b034c2308f4976.cloudfront.net (CloudFront)" response.header.x-amz-cf-id="2ljActRxc-O7Q08MTeUKDnfoUf8Eu-KvTZeOqvfvZNoHwpxKM_IQNQ==" response.header.x-amz-cf-pop=MUC50-P1 response.header.x-amz-replication-status=COMPLETED response.header.x-amz-server-side-encryption=AES256 response.header.x-amz-version-id=RJGYCnMYXfQvq9cOiFcPZ.Tg3C1emQK0 response.header.x-cache="Hit from cloudfront" response.status="200 OK" size=2925 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:022c77f5ae044caad4786fa51b28cdb27e5406120664c62f05b41c336138ee38"
time="2022-03-09T08:00:19Z" level=debug msg=resolving host=quay.io
time="2022-03-09T08:00:19Z" level=debug msg="do request" host=quay.io request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/1.5.9+unknown request.method=HEAD url="https://quay.io/v2/webhippie/alpine/manifests/3.6-arm"
time="2022-03-09T08:00:20Z" level=debug msg="fetch response received" host=quay.io response.header.content-length=1367 response.header.content-type=application/vnd.docker.distribution.manifest.v2+json response.header.date="Wed, 09 Mar 2022 08:00:20 GMT" response.header.docker-content-digest="sha256:637e6480a460ada228d356e94fb4945068c10c71ec91dc5342b24dc79f9b0153" response.header.server=nginx/1.14.1 response.header.strict-transport-security="max-age=63072000; preload" response.header.x-frame-options=DENY response.status="200 OK" url="https://quay.io/v2/webhippie/alpine/manifests/3.6-arm"
time="2022-03-09T08:00:20Z" level=debug msg=resolved desc.digest="sha256:637e6480a460ada228d356e94fb4945068c10c71ec91dc5342b24dc79f9b0153" host=quay.io
time="2022-03-09T08:00:20Z" level=debug msg=fetch digest="sha256:637e6480a460ada228d356e94fb4945068c10c71ec91dc5342b24dc79f9b0153" mediatype=application/vnd.docker.distribution.manifest.v2+json size=1367
time="2022-03-09T08:00:20Z" level=debug msg="do request" digest="sha256:637e6480a460ada228d356e94fb4945068c10c71ec91dc5342b24dc79f9b0153" mediatype=application/vnd.docker.distribution.manifest.v2+json request.header.accept="application/vnd.docker.distribution.manifest.v2+json, */*" request.header.user-agent=containerd/1.5.9+unknown request.method=GET size=1367 url="https://quay.io/v2/webhippie/alpine/manifests/sha256:637e6480a460ada228d356e94fb4945068c10c71ec91dc5342b24dc79f9b0153"
time="2022-03-09T08:00:20Z" level=debug msg="fetch response received" digest="sha256:637e6480a460ada228d356e94fb4945068c10c71ec91dc5342b24dc79f9b0153" mediatype=application/vnd.docker.distribution.manifest.v2+json response.header.content-length=1367 response.header.content-type=application/vnd.docker.distribution.manifest.v2+json response.header.date="Wed, 09 Mar 2022 08:00:20 GMT" response.header.docker-content-digest="sha256:637e6480a460ada228d356e94fb4945068c10c71ec91dc5342b24dc79f9b0153" response.header.server=nginx/1.14.1 response.header.strict-transport-security="max-age=63072000; preload" response.header.x-frame-options=DENY response.status="200 OK" size=1367 url="https://quay.io/v2/webhippie/alpine/manifests/sha256:637e6480a460ada228d356e94fb4945068c10c71ec91dc5342b24dc79f9b0153"
time="2022-03-09T08:00:20Z" level=debug msg=fetch digest="sha256:ea18fe12e67f3ad677e83d3ee74bc2c72aded8e77c42811e11352fbaa9742677" mediatype=application/vnd.docker.container.image.v1+json size=2938
time="2022-03-09T08:00:20Z" level=debug msg="do request" digest="sha256:ea18fe12e67f3ad677e83d3ee74bc2c72aded8e77c42811e11352fbaa9742677" mediatype=application/vnd.docker.container.image.v1+json request.header.accept="application/vnd.docker.container.image.v1+json, */*" request.header.user-agent=containerd/1.5.9+unknown request.method=GET size=2938 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:ea18fe12e67f3ad677e83d3ee74bc2c72aded8e77c42811e11352fbaa9742677"
time="2022-03-09T08:00:20Z" level=debug msg="fetch response received" digest="sha256:ea18fe12e67f3ad677e83d3ee74bc2c72aded8e77c42811e11352fbaa9742677" mediatype=application/vnd.docker.container.image.v1+json response.header.accept-ranges=bytes response.header.age=631 response.header.content-length=2938 response.header.content-type=binary/octet-stream response.header.date="Wed, 09 Mar 2022 07:49:50 GMT" response.header.etag="\"a31a9b409a4ebf882ce8f47875efe380-1\"" response.header.last-modified="Wed, 09 Mar 2022 03:36:11 GMT" response.header.server=AmazonS3 response.header.via="1.1 1a45d1e1304c39dfa9b034c2308f4976.cloudfront.net (CloudFront)" response.header.x-amz-cf-id="282hsOOJzaPoowT1jITQuGnUSXZ-EhLsqjr6sgSvtaWCrpVjGBLQZQ==" response.header.x-amz-cf-pop=MUC50-P1 response.header.x-amz-replication-status=COMPLETED response.header.x-amz-server-side-encryption=AES256 response.header.x-amz-version-id=1pyb_YPsRMxiuWPD2Fq0_KwZP3JZboWv response.header.x-cache="Hit from cloudfront" response.status="200 OK" size=2938 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:ea18fe12e67f3ad677e83d3ee74bc2c72aded8e77c42811e11352fbaa9742677"
time="2022-03-09T08:00:20Z" level=debug msg=push digest="sha256:e6c5a393d8d1e3faa732bc62622ad2e8dc8206d96f297892828f45dc248a7d96" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=11763211
time="2022-03-09T08:00:20Z" level=debug msg=push digest="sha256:f1e7e712b2794eb40ad5be9261c72e2b98f490ea6dbd9801136f1c36ef592fae" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=955
time="2022-03-09T08:00:20Z" level=debug msg="checking and pushing to" digest="sha256:e6c5a393d8d1e3faa732bc62622ad2e8dc8206d96f297892828f45dc248a7d96" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=11763211 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:e6c5a393d8d1e3faa732bc62622ad2e8dc8206d96f297892828f45dc248a7d96"
time="2022-03-09T08:00:20Z" level=debug msg="do request" digest="sha256:e6c5a393d8d1e3faa732bc62622ad2e8dc8206d96f297892828f45dc248a7d96" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=containerd/1.5.9+unknown request.method=HEAD size=11763211 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:e6c5a393d8d1e3faa732bc62622ad2e8dc8206d96f297892828f45dc248a7d96"
time="2022-03-09T08:00:20Z" level=debug msg=push digest="sha256:9e8c361252e68d2dd89150eff2242a8f67960d4f42b3c87bc75691aa0e03878d" mediatype=application/vnd.docker.container.image.v1+json size=2925
time="2022-03-09T08:00:20Z" level=debug msg=push digest="sha256:5a3ea8efae5d0abb93d2a04be0a4870087042b8ecab8001f613cdc2a9440616a" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=2017774
time="2022-03-09T08:00:20Z" level=debug msg="checking and pushing to" digest="sha256:f1e7e712b2794eb40ad5be9261c72e2b98f490ea6dbd9801136f1c36ef592fae" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=955 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:f1e7e712b2794eb40ad5be9261c72e2b98f490ea6dbd9801136f1c36ef592fae"
time="2022-03-09T08:00:20Z" level=debug msg="checking and pushing to" digest="sha256:9e8c361252e68d2dd89150eff2242a8f67960d4f42b3c87bc75691aa0e03878d" mediatype=application/vnd.docker.container.image.v1+json size=2925 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:9e8c361252e68d2dd89150eff2242a8f67960d4f42b3c87bc75691aa0e03878d"
time="2022-03-09T08:00:20Z" level=debug msg="do request" digest="sha256:9e8c361252e68d2dd89150eff2242a8f67960d4f42b3c87bc75691aa0e03878d" mediatype=application/vnd.docker.container.image.v1+json request.header.accept="application/vnd.docker.container.image.v1+json, */*" request.header.user-agent=containerd/1.5.9+unknown request.method=HEAD size=2925 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:9e8c361252e68d2dd89150eff2242a8f67960d4f42b3c87bc75691aa0e03878d"
time="2022-03-09T08:00:20Z" level=debug msg="checking and pushing to" digest="sha256:5a3ea8efae5d0abb93d2a04be0a4870087042b8ecab8001f613cdc2a9440616a" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=2017774 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:5a3ea8efae5d0abb93d2a04be0a4870087042b8ecab8001f613cdc2a9440616a"
time="2022-03-09T08:00:20Z" level=debug msg="do request" digest="sha256:5a3ea8efae5d0abb93d2a04be0a4870087042b8ecab8001f613cdc2a9440616a" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=containerd/1.5.9+unknown request.method=HEAD size=2017774 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:5a3ea8efae5d0abb93d2a04be0a4870087042b8ecab8001f613cdc2a9440616a"
time="2022-03-09T08:00:20Z" level=debug msg=push digest="sha256:cc0316aec136ecacf8d375c9bff734812dde163b2edcab97c1c87ef50809dfbf" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=1986249
time="2022-03-09T08:00:20Z" level=debug msg="checking and pushing to" digest="sha256:cc0316aec136ecacf8d375c9bff734812dde163b2edcab97c1c87ef50809dfbf" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=1986249 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:cc0316aec136ecacf8d375c9bff734812dde163b2edcab97c1c87ef50809dfbf"
time="2022-03-09T08:00:20Z" level=debug msg="do request" digest="sha256:cc0316aec136ecacf8d375c9bff734812dde163b2edcab97c1c87ef50809dfbf" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=containerd/1.5.9+unknown request.method=HEAD size=1986249 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:cc0316aec136ecacf8d375c9bff734812dde163b2edcab97c1c87ef50809dfbf"
time="2022-03-09T08:00:20Z" level=debug msg=push digest="sha256:a831600a8b89a2c7966e74ba39dbe8445a3f48485b515a98eb4064a65a14e74b" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=1436
time="2022-03-09T08:00:20Z" level=debug msg="checking and pushing to" digest="sha256:a831600a8b89a2c7966e74ba39dbe8445a3f48485b515a98eb4064a65a14e74b" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=1436 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:a831600a8b89a2c7966e74ba39dbe8445a3f48485b515a98eb4064a65a14e74b"
time="2022-03-09T08:00:20Z" level=debug msg="do request" digest="sha256:a831600a8b89a2c7966e74ba39dbe8445a3f48485b515a98eb4064a65a14e74b" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=containerd/1.5.9+unknown request.method=HEAD size=1436 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:a831600a8b89a2c7966e74ba39dbe8445a3f48485b515a98eb4064a65a14e74b"
time="2022-03-09T08:00:20Z" level=debug msg=push digest="sha256:98951c2dfabd790a6bda492ef5782234eff0bedaa02b366d562549436b592800" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=11651310
time="2022-03-09T08:00:20Z" level=debug msg="checking and pushing to" digest="sha256:98951c2dfabd790a6bda492ef5782234eff0bedaa02b366d562549436b592800" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=11651310 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:98951c2dfabd790a6bda492ef5782234eff0bedaa02b366d562549436b592800"
time="2022-03-09T08:00:20Z" level=debug msg=push digest="sha256:1a286ff9b6ce51fa6bf6c94a2c68fa2d2dd1bebe5c814b9482bf3417051e26b3" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=954
time="2022-03-09T08:00:20Z" level=debug msg="do request" digest="sha256:98951c2dfabd790a6bda492ef5782234eff0bedaa02b366d562549436b592800" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=containerd/1.5.9+unknown request.method=HEAD size=11651310 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:98951c2dfabd790a6bda492ef5782234eff0bedaa02b366d562549436b592800"
time="2022-03-09T08:00:20Z" level=debug msg="do request" digest="sha256:f1e7e712b2794eb40ad5be9261c72e2b98f490ea6dbd9801136f1c36ef592fae" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=containerd/1.5.9+unknown request.method=HEAD size=955 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:f1e7e712b2794eb40ad5be9261c72e2b98f490ea6dbd9801136f1c36ef592fae"
time="2022-03-09T08:00:20Z" level=debug msg="checking and pushing to" digest="sha256:1a286ff9b6ce51fa6bf6c94a2c68fa2d2dd1bebe5c814b9482bf3417051e26b3" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=954 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:1a286ff9b6ce51fa6bf6c94a2c68fa2d2dd1bebe5c814b9482bf3417051e26b3"
time="2022-03-09T08:00:20Z" level=debug msg=push digest="sha256:e8f81692e76c11ec1936df19ecdb11a3c7708adcb60b481f942689ccd9474c01" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=1936225
time="2022-03-09T08:00:20Z" level=debug msg=push digest="sha256:a831600a8b89a2c7966e74ba39dbe8445a3f48485b515a98eb4064a65a14e74b" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=1436
time="2022-03-09T08:00:20Z" level=debug msg="checking and pushing to" digest="sha256:e8f81692e76c11ec1936df19ecdb11a3c7708adcb60b481f942689ccd9474c01" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=1936225 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:e8f81692e76c11ec1936df19ecdb11a3c7708adcb60b481f942689ccd9474c01"
time="2022-03-09T08:00:20Z" level=debug msg=push digest="sha256:2fa29e73267cf241395be505134404b1d337d3ba639bf1d623a33d2fcd3243f3" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=11870737
time="2022-03-09T08:00:20Z" level=debug msg=push digest="sha256:022c77f5ae044caad4786fa51b28cdb27e5406120664c62f05b41c336138ee38" mediatype=application/vnd.docker.container.image.v1+json size=2925
time="2022-03-09T08:00:20Z" level=debug msg="checking and pushing to" digest="sha256:022c77f5ae044caad4786fa51b28cdb27e5406120664c62f05b41c336138ee38" mediatype=application/vnd.docker.container.image.v1+json size=2925 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:022c77f5ae044caad4786fa51b28cdb27e5406120664c62f05b41c336138ee38"
time="2022-03-09T08:00:20Z" level=debug msg="do request" digest="sha256:022c77f5ae044caad4786fa51b28cdb27e5406120664c62f05b41c336138ee38" mediatype=application/vnd.docker.container.image.v1+json request.header.accept="application/vnd.docker.container.image.v1+json, */*" request.header.user-agent=containerd/1.5.9+unknown request.method=HEAD size=2925 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:022c77f5ae044caad4786fa51b28cdb27e5406120664c62f05b41c336138ee38"
time="2022-03-09T08:00:20Z" level=debug msg="do request" digest="sha256:e8f81692e76c11ec1936df19ecdb11a3c7708adcb60b481f942689ccd9474c01" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=containerd/1.5.9+unknown request.method=HEAD size=1936225 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:e8f81692e76c11ec1936df19ecdb11a3c7708adcb60b481f942689ccd9474c01"
time="2022-03-09T08:00:20Z" level=debug msg=push digest="sha256:542ea154e03fb8b35b9ca62445bc090a7fe9e093c30538e0b653743b80f81cc4" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=10964820
time="2022-03-09T08:00:20Z" level=debug msg=push digest="sha256:110d1638fb7c3b0f1966cf0d608617b8f8cfb795288096ddc3f70209d1556b49" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=10179823
time="2022-03-09T08:00:20Z" level=debug msg="checking and pushing to" digest="sha256:542ea154e03fb8b35b9ca62445bc090a7fe9e093c30538e0b653743b80f81cc4" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=10964820 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:542ea154e03fb8b35b9ca62445bc090a7fe9e093c30538e0b653743b80f81cc4"
time="2022-03-09T08:00:20Z" level=debug msg="do request" digest="sha256:542ea154e03fb8b35b9ca62445bc090a7fe9e093c30538e0b653743b80f81cc4" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=containerd/1.5.9+unknown request.method=HEAD size=10964820 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:542ea154e03fb8b35b9ca62445bc090a7fe9e093c30538e0b653743b80f81cc4"
time="2022-03-09T08:00:20Z" level=debug msg=push digest="sha256:a831600a8b89a2c7966e74ba39dbe8445a3f48485b515a98eb4064a65a14e74b" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=1436
time="2022-03-09T08:00:20Z" level=debug msg=push digest="sha256:73c6d8f222a884789be203ea7cae6386b4f54a92e9c540dd991365950cc0b2be" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=10301040
time="2022-03-09T08:00:20Z" level=debug msg=push digest="sha256:ea18fe12e67f3ad677e83d3ee74bc2c72aded8e77c42811e11352fbaa9742677" mediatype=application/vnd.docker.container.image.v1+json size=2938
time="2022-03-09T08:00:20Z" level=debug msg="checking and pushing to" digest="sha256:110d1638fb7c3b0f1966cf0d608617b8f8cfb795288096ddc3f70209d1556b49" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=10179823 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:110d1638fb7c3b0f1966cf0d608617b8f8cfb795288096ddc3f70209d1556b49"
time="2022-03-09T08:00:20Z" level=debug msg="do request" digest="sha256:110d1638fb7c3b0f1966cf0d608617b8f8cfb795288096ddc3f70209d1556b49" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=containerd/1.5.9+unknown request.method=HEAD size=10179823 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:110d1638fb7c3b0f1966cf0d608617b8f8cfb795288096ddc3f70209d1556b49"
time="2022-03-09T08:00:20Z" level=debug msg=push digest="sha256:6cce0c6f0475dff0fe222025f10d3a6f36a9102938f1ec483b63e5f28b4fa5e3" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=955
time="2022-03-09T08:00:20Z" level=debug msg="do request" digest="sha256:1a286ff9b6ce51fa6bf6c94a2c68fa2d2dd1bebe5c814b9482bf3417051e26b3" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=containerd/1.5.9+unknown request.method=HEAD size=954 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:1a286ff9b6ce51fa6bf6c94a2c68fa2d2dd1bebe5c814b9482bf3417051e26b3"
time="2022-03-09T08:00:20Z" level=debug msg="checking and pushing to" digest="sha256:73c6d8f222a884789be203ea7cae6386b4f54a92e9c540dd991365950cc0b2be" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=10301040 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:73c6d8f222a884789be203ea7cae6386b4f54a92e9c540dd991365950cc0b2be"
time="2022-03-09T08:00:20Z" level=debug msg="do request" digest="sha256:73c6d8f222a884789be203ea7cae6386b4f54a92e9c540dd991365950cc0b2be" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=containerd/1.5.9+unknown request.method=HEAD size=10301040 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:73c6d8f222a884789be203ea7cae6386b4f54a92e9c540dd991365950cc0b2be"
time="2022-03-09T08:00:20Z" level=debug msg="checking and pushing to" digest="sha256:6cce0c6f0475dff0fe222025f10d3a6f36a9102938f1ec483b63e5f28b4fa5e3" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=955 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:6cce0c6f0475dff0fe222025f10d3a6f36a9102938f1ec483b63e5f28b4fa5e3"
time="2022-03-09T08:00:20Z" level=debug msg="checking and pushing to" digest="sha256:2fa29e73267cf241395be505134404b1d337d3ba639bf1d623a33d2fcd3243f3" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=11870737 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:2fa29e73267cf241395be505134404b1d337d3ba639bf1d623a33d2fcd3243f3"
time="2022-03-09T08:00:20Z" level=debug msg="do request" digest="sha256:2fa29e73267cf241395be505134404b1d337d3ba639bf1d623a33d2fcd3243f3" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=containerd/1.5.9+unknown request.method=HEAD size=11870737 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:2fa29e73267cf241395be505134404b1d337d3ba639bf1d623a33d2fcd3243f3"
time="2022-03-09T08:00:20Z" level=debug msg="checking and pushing to" digest="sha256:ea18fe12e67f3ad677e83d3ee74bc2c72aded8e77c42811e11352fbaa9742677" mediatype=application/vnd.docker.container.image.v1+json size=2938 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:ea18fe12e67f3ad677e83d3ee74bc2c72aded8e77c42811e11352fbaa9742677"
time="2022-03-09T08:00:20Z" level=debug msg="do request" digest="sha256:ea18fe12e67f3ad677e83d3ee74bc2c72aded8e77c42811e11352fbaa9742677" mediatype=application/vnd.docker.container.image.v1+json request.header.accept="application/vnd.docker.container.image.v1+json, */*" request.header.user-agent=containerd/1.5.9+unknown request.method=HEAD size=2938 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:ea18fe12e67f3ad677e83d3ee74bc2c72aded8e77c42811e11352fbaa9742677"
time="2022-03-09T08:00:20Z" level=debug msg="do request" digest="sha256:6cce0c6f0475dff0fe222025f10d3a6f36a9102938f1ec483b63e5f28b4fa5e3" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=containerd/1.5.9+unknown request.method=HEAD size=955 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:6cce0c6f0475dff0fe222025f10d3a6f36a9102938f1ec483b63e5f28b4fa5e3"
time="2022-03-09T08:00:20Z" level=debug msg="fetch response received" digest="sha256:9e8c361252e68d2dd89150eff2242a8f67960d4f42b3c87bc75691aa0e03878d" mediatype=application/vnd.docker.container.image.v1+json response.header.accept-ranges=bytes response.header.cache-control="max-age=31436000" response.header.content-length=2925 response.header.content-type=application/octet-stream response.header.date="Wed, 09 Mar 2022 08:00:20 GMT" response.header.docker-content-digest="sha256:9e8c361252e68d2dd89150eff2242a8f67960d4f42b3c87bc75691aa0e03878d" response.header.server=nginx/1.14.1 response.header.strict-transport-security="max-age=63072000; preload" response.header.x-frame-options=DENY response.status="200 OK" size=2925 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:9e8c361252e68d2dd89150eff2242a8f67960d4f42b3c87bc75691aa0e03878d"
time="2022-03-09T08:00:20Z" level=debug msg="fetch response received" digest="sha256:e6c5a393d8d1e3faa732bc62622ad2e8dc8206d96f297892828f45dc248a7d96" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.cache-control="max-age=31436000" response.header.content-length=11763211 response.header.content-type=application/octet-stream response.header.date="Wed, 09 Mar 2022 08:00:20 GMT" response.header.docker-content-digest="sha256:e6c5a393d8d1e3faa732bc62622ad2e8dc8206d96f297892828f45dc248a7d96" response.header.server=nginx/1.14.1 response.header.strict-transport-security="max-age=63072000; preload" response.header.x-frame-options=DENY response.status="200 OK" size=11763211 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:e6c5a393d8d1e3faa732bc62622ad2e8dc8206d96f297892828f45dc248a7d96"
time="2022-03-09T08:00:20Z" level=debug msg="fetch response received" digest="sha256:a831600a8b89a2c7966e74ba39dbe8445a3f48485b515a98eb4064a65a14e74b" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.cache-control="max-age=31436000" response.header.content-length=1436 response.header.content-type=application/octet-stream response.header.date="Wed, 09 Mar 2022 08:00:20 GMT" response.header.docker-content-digest="sha256:a831600a8b89a2c7966e74ba39dbe8445a3f48485b515a98eb4064a65a14e74b" response.header.server=nginx/1.14.1 response.header.strict-transport-security="max-age=63072000; preload" response.header.x-frame-options=DENY response.status="200 OK" size=1436 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:a831600a8b89a2c7966e74ba39dbe8445a3f48485b515a98eb4064a65a14e74b"
time="2022-03-09T08:00:20Z" level=debug msg="fetch response received" digest="sha256:98951c2dfabd790a6bda492ef5782234eff0bedaa02b366d562549436b592800" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.cache-control="max-age=31436000" response.header.content-length=11651310 response.header.content-type=application/octet-stream response.header.date="Wed, 09 Mar 2022 08:00:20 GMT" response.header.docker-content-digest="sha256:98951c2dfabd790a6bda492ef5782234eff0bedaa02b366d562549436b592800" response.header.server=nginx/1.14.1 response.header.strict-transport-security="max-age=63072000; preload" response.header.x-frame-options=DENY response.status="200 OK" size=11651310 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:98951c2dfabd790a6bda492ef5782234eff0bedaa02b366d562549436b592800"
time="2022-03-09T08:00:20Z" level=debug msg="fetch response received" digest="sha256:5a3ea8efae5d0abb93d2a04be0a4870087042b8ecab8001f613cdc2a9440616a" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.cache-control="max-age=31436000" response.header.content-length=2017774 response.header.content-type=application/octet-stream response.header.date="Wed, 09 Mar 2022 08:00:20 GMT" response.header.docker-content-digest="sha256:5a3ea8efae5d0abb93d2a04be0a4870087042b8ecab8001f613cdc2a9440616a" response.header.server=nginx/1.14.1 response.header.strict-transport-security="max-age=63072000; preload" response.header.x-frame-options=DENY response.status="200 OK" size=2017774 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:5a3ea8efae5d0abb93d2a04be0a4870087042b8ecab8001f613cdc2a9440616a"
time="2022-03-09T08:00:20Z" level=debug msg="fetch response received" digest="sha256:022c77f5ae044caad4786fa51b28cdb27e5406120664c62f05b41c336138ee38" mediatype=application/vnd.docker.container.image.v1+json response.header.accept-ranges=bytes response.header.cache-control="max-age=31436000" response.header.content-length=2925 response.header.content-type=application/octet-stream response.header.date="Wed, 09 Mar 2022 08:00:20 GMT" response.header.docker-content-digest="sha256:022c77f5ae044caad4786fa51b28cdb27e5406120664c62f05b41c336138ee38" response.header.server=nginx/1.14.1 response.header.strict-transport-security="max-age=63072000; preload" response.header.x-frame-options=DENY response.status="200 OK" size=2925 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:022c77f5ae044caad4786fa51b28cdb27e5406120664c62f05b41c336138ee38"
time="2022-03-09T08:00:20Z" level=debug msg="fetch response received" digest="sha256:cc0316aec136ecacf8d375c9bff734812dde163b2edcab97c1c87ef50809dfbf" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.cache-control="max-age=31436000" response.header.content-length=1986249 response.header.content-type=application/octet-stream response.header.date="Wed, 09 Mar 2022 08:00:20 GMT" response.header.docker-content-digest="sha256:cc0316aec136ecacf8d375c9bff734812dde163b2edcab97c1c87ef50809dfbf" response.header.server=nginx/1.14.1 response.header.strict-transport-security="max-age=63072000; preload" response.header.x-frame-options=DENY response.status="200 OK" size=1986249 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:cc0316aec136ecacf8d375c9bff734812dde163b2edcab97c1c87ef50809dfbf"
time="2022-03-09T08:00:20Z" level=debug msg="fetch response received" digest="sha256:ea18fe12e67f3ad677e83d3ee74bc2c72aded8e77c42811e11352fbaa9742677" mediatype=application/vnd.docker.container.image.v1+json response.header.accept-ranges=bytes response.header.cache-control="max-age=31436000" response.header.content-length=2938 response.header.content-type=application/octet-stream response.header.date="Wed, 09 Mar 2022 08:00:20 GMT" response.header.docker-content-digest="sha256:ea18fe12e67f3ad677e83d3ee74bc2c72aded8e77c42811e11352fbaa9742677" response.header.server=nginx/1.14.1 response.header.strict-transport-security="max-age=63072000; preload" response.header.x-frame-options=DENY response.status="200 OK" size=2938 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:ea18fe12e67f3ad677e83d3ee74bc2c72aded8e77c42811e11352fbaa9742677"
time="2022-03-09T08:00:20Z" level=debug msg="fetch response received" digest="sha256:e8f81692e76c11ec1936df19ecdb11a3c7708adcb60b481f942689ccd9474c01" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.cache-control="max-age=31436000" response.header.content-length=1936225 response.header.content-type=application/octet-stream response.header.date="Wed, 09 Mar 2022 08:00:20 GMT" response.header.docker-content-digest="sha256:e8f81692e76c11ec1936df19ecdb11a3c7708adcb60b481f942689ccd9474c01" response.header.server=nginx/1.14.1 response.header.strict-transport-security="max-age=63072000; preload" response.header.x-frame-options=DENY response.status="200 OK" size=1936225 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:e8f81692e76c11ec1936df19ecdb11a3c7708adcb60b481f942689ccd9474c01"
time="2022-03-09T08:00:20Z" level=debug msg="fetch response received" digest="sha256:73c6d8f222a884789be203ea7cae6386b4f54a92e9c540dd991365950cc0b2be" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.cache-control="max-age=31436000" response.header.content-length=10301040 response.header.content-type=application/octet-stream response.header.date="Wed, 09 Mar 2022 08:00:20 GMT" response.header.docker-content-digest="sha256:73c6d8f222a884789be203ea7cae6386b4f54a92e9c540dd991365950cc0b2be" response.header.server=nginx/1.14.1 response.header.strict-transport-security="max-age=63072000; preload" response.header.x-frame-options=DENY response.status="200 OK" size=10301040 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:73c6d8f222a884789be203ea7cae6386b4f54a92e9c540dd991365950cc0b2be"
time="2022-03-09T08:00:20Z" level=debug msg="fetch response received" digest="sha256:110d1638fb7c3b0f1966cf0d608617b8f8cfb795288096ddc3f70209d1556b49" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.cache-control="max-age=31436000" response.header.content-length=10179823 response.header.content-type=application/octet-stream response.header.date="Wed, 09 Mar 2022 08:00:20 GMT" response.header.docker-content-digest="sha256:110d1638fb7c3b0f1966cf0d608617b8f8cfb795288096ddc3f70209d1556b49" response.header.server=nginx/1.14.1 response.header.strict-transport-security="max-age=63072000; preload" response.header.x-frame-options=DENY response.status="200 OK" size=10179823 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:110d1638fb7c3b0f1966cf0d608617b8f8cfb795288096ddc3f70209d1556b49"
time="2022-03-09T08:00:20Z" level=debug msg="fetch response received" digest="sha256:2fa29e73267cf241395be505134404b1d337d3ba639bf1d623a33d2fcd3243f3" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.cache-control="max-age=31436000" response.header.content-length=11870737 response.header.content-type=application/octet-stream response.header.date="Wed, 09 Mar 2022 08:00:20 GMT" response.header.docker-content-digest="sha256:2fa29e73267cf241395be505134404b1d337d3ba639bf1d623a33d2fcd3243f3" response.header.server=nginx/1.14.1 response.header.strict-transport-security="max-age=63072000; preload" response.header.x-frame-options=DENY response.status="200 OK" size=11870737 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:2fa29e73267cf241395be505134404b1d337d3ba639bf1d623a33d2fcd3243f3"
time="2022-03-09T08:00:20Z" level=debug msg="fetch response received" digest="sha256:542ea154e03fb8b35b9ca62445bc090a7fe9e093c30538e0b653743b80f81cc4" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.cache-control="max-age=31436000" response.header.content-length=10964820 response.header.content-type=application/octet-stream response.header.date="Wed, 09 Mar 2022 08:00:20 GMT" response.header.docker-content-digest="sha256:542ea154e03fb8b35b9ca62445bc090a7fe9e093c30538e0b653743b80f81cc4" response.header.server=nginx/1.14.1 response.header.strict-transport-security="max-age=63072000; preload" response.header.x-frame-options=DENY response.status="200 OK" size=10964820 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:542ea154e03fb8b35b9ca62445bc090a7fe9e093c30538e0b653743b80f81cc4"
time="2022-03-09T08:00:20Z" level=debug msg="fetch response received" digest="sha256:1a286ff9b6ce51fa6bf6c94a2c68fa2d2dd1bebe5c814b9482bf3417051e26b3" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.cache-control="max-age=31436000" response.header.content-length=954 response.header.content-type=application/octet-stream response.header.date="Wed, 09 Mar 2022 08:00:20 GMT" response.header.docker-content-digest="sha256:1a286ff9b6ce51fa6bf6c94a2c68fa2d2dd1bebe5c814b9482bf3417051e26b3" response.header.server=nginx/1.14.1 response.header.strict-transport-security="max-age=63072000; preload" response.header.x-frame-options=DENY response.status="200 OK" size=954 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:1a286ff9b6ce51fa6bf6c94a2c68fa2d2dd1bebe5c814b9482bf3417051e26b3"
time="2022-03-09T08:00:20Z" level=debug msg="fetch response received" digest="sha256:f1e7e712b2794eb40ad5be9261c72e2b98f490ea6dbd9801136f1c36ef592fae" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.cache-control="max-age=31436000" response.header.content-length=955 response.header.content-type=application/octet-stream response.header.date="Wed, 09 Mar 2022 08:00:20 GMT" response.header.docker-content-digest="sha256:f1e7e712b2794eb40ad5be9261c72e2b98f490ea6dbd9801136f1c36ef592fae" response.header.server=nginx/1.14.1 response.header.strict-transport-security="max-age=63072000; preload" response.header.x-frame-options=DENY response.status="200 OK" size=955 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:f1e7e712b2794eb40ad5be9261c72e2b98f490ea6dbd9801136f1c36ef592fae"
time="2022-03-09T08:00:20Z" level=debug msg="fetch response received" digest="sha256:6cce0c6f0475dff0fe222025f10d3a6f36a9102938f1ec483b63e5f28b4fa5e3" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.cache-control="max-age=31436000" response.header.content-length=955 response.header.content-type=application/octet-stream response.header.date="Wed, 09 Mar 2022 08:00:20 GMT" response.header.docker-content-digest="sha256:6cce0c6f0475dff0fe222025f10d3a6f36a9102938f1ec483b63e5f28b4fa5e3" response.header.server=nginx/1.14.1 response.header.strict-transport-security="max-age=63072000; preload" response.header.x-frame-options=DENY response.status="200 OK" size=955 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:6cce0c6f0475dff0fe222025f10d3a6f36a9102938f1ec483b63e5f28b4fa5e3"
time="2022-03-09T08:00:20Z" level=debug msg=push digest="sha256:316470549e4eb403bf2c67c454cdc644b99dba23d625c299e75ebd00a02fe5e3" mediatype=application/vnd.docker.distribution.manifest.v2+json size=1367
time="2022-03-09T08:00:20Z" level=debug msg="checking and pushing to" digest="sha256:316470549e4eb403bf2c67c454cdc644b99dba23d625c299e75ebd00a02fe5e3" mediatype=application/vnd.docker.distribution.manifest.v2+json size=1367 url="https://quay.io/v2/webhippie/alpine/manifests/3.6"
time="2022-03-09T08:00:20Z" level=debug msg="do request" digest="sha256:316470549e4eb403bf2c67c454cdc644b99dba23d625c299e75ebd00a02fe5e3" mediatype=application/vnd.docker.distribution.manifest.v2+json request.header.accept="application/vnd.docker.distribution.manifest.v2+json, */*" request.header.user-agent=containerd/1.5.9+unknown request.method=HEAD size=1367 url="https://quay.io/v2/webhippie/alpine/manifests/3.6"
time="2022-03-09T08:00:20Z" level=debug msg="fetch response received" digest="sha256:316470549e4eb403bf2c67c454cdc644b99dba23d625c299e75ebd00a02fe5e3" mediatype=application/vnd.docker.distribution.manifest.v2+json response.header.content-length=129 response.header.content-type=application/json response.header.date="Wed, 09 Mar 2022 08:00:20 GMT" response.header.server=nginx/1.14.1 response.status="404 Not Found" size=1367 url="https://quay.io/v2/webhippie/alpine/manifests/3.6"
time="2022-03-09T08:00:20Z" level=debug msg="do request" digest="sha256:316470549e4eb403bf2c67c454cdc644b99dba23d625c299e75ebd00a02fe5e3" mediatype=application/vnd.docker.distribution.manifest.v2+json request.header.content-type=application/vnd.docker.distribution.manifest.v2+json request.header.user-agent=containerd/1.5.9+unknown request.method=PUT size=1367 url="https://quay.io/v2/webhippie/alpine/manifests/3.6"
time="2022-03-09T08:00:20Z" level=debug msg="fetch response received" digest="sha256:316470549e4eb403bf2c67c454cdc644b99dba23d625c299e75ebd00a02fe5e3" mediatype=application/vnd.docker.distribution.manifest.v2+json response.header.content-length=112 response.header.content-type=application/json response.header.date="Wed, 09 Mar 2022 08:00:20 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.server=nginx/1.14.1 response.header.www-authenticate="Bearer realm=\"https://quay.io/v2/auth\",service=\"quay.io\",scope=\"repository:webhippie/alpine:pull,push\"" response.status="401 Unauthorized" size=1367 url="https://quay.io/v2/webhippie/alpine/manifests/3.6"
time="2022-03-09T08:00:20Z" level=debug msg=Unauthorized digest="sha256:316470549e4eb403bf2c67c454cdc644b99dba23d625c299e75ebd00a02fe5e3" header="Bearer realm=\"https://quay.io/v2/auth\",service=\"quay.io\",scope=\"repository:webhippie/alpine:pull,push\"" mediatype=application/vnd.docker.distribution.manifest.v2+json size=1367
time="2022-03-09T08:00:20Z" level=fatal msg="Error pushing manifest list/index to registry: sha256:0c2d7d615df8b704e5aaf8fe62e90bf33ec7a0bd0fefca0bc54b77b5d9d720d5: failed commit on ref \"manifest-sha256:316470549e4eb403bf2c67c454cdc644b99dba23d625c299e75ebd00a02fe5e3\": cannot reuse body, request must be retried"

For now I will do a downgrade to latest 1.x.x release to get my pipelines working again, but I would love to use the new version when it works, of course. The same image works fine with 1.0.3:

$ docker run --rm mplatform/manifest-tool:v1.0.3 --username $QUAY_USERNAME --password $QUAY_PASSWORD push from-args --platforms linux/amd64,linux/arm64,linux/arm/v6 --template quay.io/webhippie/alpine:3.6-ARCH --target quay.io/webhippie/alpine:3.6
Digest: sha256:112d1f640d44a844a65b4465183d409b6ac2cdb6ca8038698e93976855712d3c 1079

@estesp estesp mentioned this issue Mar 11, 2022
Closed
@estesp
Copy link
Owner

estesp commented Mar 11, 2022

Looks like #122 is finding the same issue; I don't get the 404 that I see near the end of your debug output when it's trying to assemble the final manifest in the registry:

response.status="404 Not Found" size=1367 url="https://quay.io/v2/webhippie/alpine/manifests/3.6"

and it's from there that the reply/response cycle may actually be reusing a body incorrectly when it triggers an auth flow?

@tboerger
Copy link
Author

If you need any further debug information just let me know :)

@estesp
Copy link
Owner

estesp commented Mar 14, 2022

I was able to reproduce it; there are 2 issues at play--one is the known "cannot reuse body" issue which we have someone assigned to work on and fix in the containerd registry push handler. When that fix is available it should solve the overall problem of retry with auth during a push.

The second is that it looks like maybe there is something with token scopes/auth that makes this only reproducible when you use the same repository for the source(s)/target of the manifest list/index. For example, I had images like quay.io/estesp/amd64_alpine:3.11 that were then populating a manifest list in quay.io/estesp/alpine:3.11 (meaning 5 different repositories were being used; 4 uniquely named repos with the architecture as part of the name and the target "alpine:..." repository.

When I change my configuration to source the architectures and final list from the same repo I get the same 404 on the existence check of the manifest list, followed by a 401 to re-auth, which gets the "cannot reuse body" error.

The 404 is confusing as I would expect that to happen on the existence check HEAD request on the target manifest list in the other flow (where the source repos are all different), but it doesn't happen. Will need to dig further but at least now I can reproduce.

@estesp estesp mentioned this issue Mar 15, 2022
Merged
@estesp
Copy link
Owner

estesp commented Mar 15, 2022

Worked around in #159 until containerd has a complete fix for the retry on auth challenge issue with req body reuse.

I've tested Quay and verified this fixes this issue with using manifest-tool v2 and Quay.io. Interestingly, if you set the repo to private or source any of the platform images from a private repo the 401 auth challenge comes during a "fetch" and you don't get the error. That's why I initially never saw this issue when testing.

@estesp
Copy link
Owner

estesp commented Mar 15, 2022

You can test/verify with the latest release:

$ docker run --rm mplatform/manifest-tool --version
/manifest-tool version 2.0.3 (commit: 65590ecce1d4af199724d235dbb5453e10cad420)

@tboerger
Copy link
Author

Thanks for the effort, I can confirm that the latest release has resolved my issues for Quay.io :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Workaround for body reuse/401 auth errors estesp/manifest-tool
3 participants
@tboerger @estesp @mthalman