oras pull error: empty response Docker-Content-Digest

[johankees]

Pulling a container from AWS ECR fails with error empty response Docker-Content-Digest.

Steps to reproduce:

• login as per https://oras.land/implementors/#amazon-elastic-container-registry-ecr
• oras pull ${ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/${IMAGE}:${TAG}

Expected result:

• Image pulled successfully, and response:

  Pulled ACCOUNT_ID.dkr.ecr.AWS_REGION.amazonaws.com/IMAGE:TAG
  Digest: sha256:<SOME_SHA>
  

Actual result:

• Image failed to be pulled:

  Error: GET "https://ACCOUNT_ID.dkr.ecr.REGION.amazonaws.com/v2/IMAGE/manifests/TAG": empty response Docker-Content-Digest
  

Verbose error message:

-> oras pull ACCOUNT_ID.dkr.ecr.AWS_REGION.amazonaws.com/IMAGE:TAG -d
DEBU[0000]  Request URL: "https://ACCOUNT_ID.dkr.ecr.AWS_REGION.amazonaws.com/v2/IMAGE/manifests/TAG"
DEBU[0000]  Request method: "GET"
DEBU[0000]  Request headers:
DEBU[0000]    "Accept": "application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, application/vnd.cncf.oras.artifact.manifest.v1+json"
DEBU[0000]    "User-Agent": "oras/0.13.0"
DEBU[0000]  Response Status: "401 Unauthorized"
DEBU[0000]  Response headers:
DEBU[0000]    "Docker-Distribution-Api-Version": "registry/2.0"
DEBU[0000]    "Sizes": ""
DEBU[0000]    "Www-Authenticate": "Basic realm=\"https://ACCOUNT_ID.dkr.ecr.AWS_REGION.amazonaws.com/\",service=\"ecr.amazonaws.com\""
DEBU[0000]    "Date": "Mon, 04 Jul 2022 13:06:58 GMT"
DEBU[0000]    "Content-Length": "15"
DEBU[0000]    "Content-Type": "text/plain; charset=utf-8"
DEBU[0000]  Request URL: "https://ACCOUNT_ID.dkr.ecr.AWS_REGION.amazonaws.com/v2/IMAGE/manifests/TAG"
DEBU[0000]  Request method: "GET"
DEBU[0000]  Request headers:
DEBU[0000]    "User-Agent": "oras/0.13.0"
DEBU[0000]    "Accept": "application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, application/vnd.cncf.oras.artifact.manifest.v1+json"
DEBU[0000]    "Authorization": "*****"
DEBU[0001]  Response Status: "200 OK"
DEBU[0001]  Response headers:
DEBU[0001]    "Content-Type": "application/vnd.oci.image.manifest.v1+json"
DEBU[0001]    "Docker-Distribution-Api-Version": "registry/2.0"
DEBU[0001]    "Sizes": ""
DEBU[0001]    "Date": "Mon, 04 Jul 2022 13:06:59 GMT"
DEBU[0001]    "Content-Length": "396"
Error: GET "https://ACCOUNT_ID.dkr.ecr.AWS_REGION.amazonaws.com/v2/IMAGE/manifests/TAG": empty response Docker-Content-Digest

Version Information:
Oras: 0.13.0

Note: I have tested the same command with v0.12.0 and it works fine in this version. So it looks like this issue was introduced in v0.13.0.

[shizhMSFT]

AWS ECR is not compliant with Docker Registry HTTP API V2 where Docker-Content-Digest is required in the response.

In OCI spec,

A successful response SHOULD contain the digest of the uploaded blob in the header Docker-Content-Digest.

/cc @michaelb990

[michaelb990]

Hmm... A client failure based on a registry not returning a header that SHOULD be present seems like a client issue, not a registry issue.

We can still look into fixing this on the registry side, but I'm curious why this is a required header for the ORAS client. Do you know why this changed in v0.13.0?

[shizhMSFT]

Before v0.13.0, oras CLI is based on oras-go v0.1.0, which is further based on containerd, and it seems containerd does not check the Docker-Content-Digest header. In v0.13.0, oras CLI is based on oras-go v2, which is purely implemented according to the Docker and OCI specs.

Since it is a "MUST" in the docker spec and a "SHOULD" in the OCI spec, oras-go MAY handle empty Docker-Content-Digest in the response.

[shizhMSFT]

As we don't have an ECR to test with, @michaelb990 can you ask @nima to help fixing this?

[nima]

Okay I have a repro, and will look into this today/weekend.

Side-questions: any discussions (or existence) of black-box/smoke-testing that would have caught something like this? Even something very basic; single command to test out various common and valid critical permutations of commands that should never fail.

[nima]

I think I need some help with this; I've started a thread here describing my current approach and its shortcomings.

[FeynmanZhou]

Please note that this issue also caused the Notation pull works to fail with ECR since there is a dependency for Notary v2. Can we prioritize it? @nima

[nima]

I've been away for a couple of weeks; taking a look at this now; will update here.