ECR: support manifest lists

[jtoberon]

ECR should support manifest lists (https://docs.docker.com/registry/spec/manifest-v2-2/#manifest-list). This feature allows ECR customers to create and consume multi-architecture images more easily.

[FernandoMiguel]

I strongly support this. The amount of use cases we are blocked by the lack of manifest support when using cache images, is affecting so many developers

[jmeickle]

This lack of support is actually blocking us from adopting Buildkit, since it prevents properly caching in that scenario.

[EduardSergeev]

We had to switch to self-hosted registry:2 because we needed to store multi-arch images. Incidentally we needed it because we wanted to use AWS new A1 instances in production along with T3 instances in development.

[frzifus]

would be great to see that on the roadmap. 🥇

[diclophis]

This would be huge advantage for ECR to support buildkit pulling caching from the registry

[marksantoso]

Looking forward to this.

[omieomye]

This is rolling out now and ready to use in most regions. We'll publish a formal announcement shortly with some more reading. In the meantime, docs are updated and please let us know your feedback.

[nithinvenugopal]

Does AWS batch support multi-architecture compute environment in it's queue with this change?

[omieomye]

@nithinvenugopal I'll let you know.

[omieomye]

Shipped. Blog Announcement.

[jsierles]

I just tried this on eu-west-1, but am still seeing:

#7 ERROR: error writing manifest blob: failed commit on ref "sha256:4ec7073e9d9c619a1320b7db019d1b5cf3b87994f636346bb7ec91165d5f751a": unexpected status: 405 Method Not Allowed

[michaelb990]

Hi @jsierles!

Thanks for trying out the feature! I was not able to reproduce that error in my account. Could you provide us with some more information? Specifically:

• what are the full commands you ran to produce this error?
• what was the full log output of those commands?
• what time (or time range) did this occur?
• what (tool & version) are you using to build & push the image? (e.g. docker build/push, podman, buildah, etc)
• what is the output of docker manifest inspect <tag>?
• are you connecting to ECR through AWS PrivateLink or over the public internet?

[njdancer]

Hi @michaelb990, I'm having the same error as @jsierles although I'm not sure if we're running the same command. Happy to provide some info from my perspective though.

I'm running Buildkit via docker inside GitHub Actions. See https://gist.github.com/njdancer/c8fa02bff61e2b8047fefa5e68af1ee6 for logs.

The logs aren't the cleanest I'm afraid so let me know if you need specific parts pulled out. Here the command though:

  export IMAGE_REPO="***.dkr.ecr.ap-southeast-2.amazonaws.com/drivo-api"
  export SHA=ae6451a6675805e5556c60f4ec23abcc37dc6c98
  docker run \
    --rm \
    --privileged \
    -v $(pwd)/api:/tmp/work \
    -v ${HOME}/.docker:/root/.docker \
    --entrypoint buildctl-daemonless.sh \
    moby/buildkit:master \
      build \
      --frontend dockerfile.v0 \
      --local context=/tmp/work \
      --local dockerfile=/tmp/work \
      --output type=image,name="${IMAGE_REPO}:${SHA}",push=true \
      --export-cache type=registry,mode=max,ref=${IMAGE_REPO}:buildcache,push=true \
      --import-cache type=registry,ref=${IMAGE_REPO}:buildcache

Just like @jsierles, I receive the following error:

#18 ERROR: error writing manifest blob: failed commit on ref "sha256:d8d33b6a5b9538c77d8086e05ec3c1bedd6ad0da5fdc29e105ea07d7ac331e20": unexpected status: 405 Method Not Allowed

I only just ran this in the last 15 minutes so approx. 08:55 UTC. I also don't believe docker manifest inspect will show anything as I don't believe Buildkit to be exporting this to dockers image store in the current setup. Given that this is running inside a container I believe you'd need some kind of DinD setup to get the manifest directly into docker.

I believe GitHub Actions run inside Azure so I imagine this is just running over the internet, I guess there could be some peering but none that should make any difference AFAIK.

Final point I would mention is that if I remove the import and export cache arguments from the command the image builds and pushes to ECR successfully. It is only when adding the cache options that this begins to fail. This usage is supported by Dockers official registry software also.

[jsierles]

I'm using buildx with cache-to. Pushing works fine.

docker buildx build --cache-to=type=registry,ref=450076236152.dkr.ecr.eu-west-1.amazonaws.com/repo,mode=max --push .

docker buildx build -t 450076236152.dkr.ecr.eu-west-1.amazonaws.com/repo --cache-to 450076236152.dkr.ecr.eu-west-1.amazonaws.com/repo --push .
[+] Building 13.4s (6/6) FINISHED
 => [internal] load build definition from Dockerfile                                                                                              0.0s
 => => transferring dockerfile: 31B                                                                                                               0.0s
 => [internal] load .dockerignore                                                                                                                 0.0s
 => => transferring context: 2B                                                                                                                   0.0s
 => [internal] load metadata for docker.io/library/bash:latest                                                                                    4.1s
 => CACHED [1/1] FROM docker.io/library/bash@sha256:145e00f05adb8b2fcf914380f2efeeb0efeda8b3fb4025b1118146311099933f                              0.0s
 => => resolve docker.io/library/bash@sha256:145e00f05adb8b2fcf914380f2efeeb0efeda8b3fb4025b1118146311099933f                                     0.0s
 => exporting to image                                                                                                                            3.9s
 => => exporting layers                                                                                                                           0.0s
 => => exporting manifest sha256:36ac4cc592f7f69f3ff86b2dac0a9b18a96a96a552b623bdc2a16eb81adb238c                                                 0.0s
 => => exporting config sha256:6962fa95e9388a0b1c7222a0890dc8831c8754ac53232c34770da91c6650ee8a                                                   0.0s
 => => pushing layers                                                                                                                             2.6s
 => => pushing manifest for 450076236152.dkr.ecr.eu-west-1.amazonaws.com/repo:latest                                                  1.3s
 => ERROR exporting cache                                                                                                                         5.3s
 => => preparing build cache for export                                                                                                           0.0s
 => => writing config sha256:f94a0907a5d4bd30765dd0aac0c4034b4bf82d712a6a5dff6c3fc9a7c72e0622                                                     3.0s
 => => writing manifest sha256:4ec7073e9d9c619a1320b7db019d1b5cf3b87994f636346bb7ec91165d5f751a                                                   2.3s
------
 > exporting cache:
------
failed to solve: rpc error: code = Unknown desc = error writing manifest blob: failed commit on ref "sha256:4ec7073e9d9c619a1320b7db019d1b5cf3b87994f636346bb7ec91165d5f751a": unexpected status: 405 Method Not Allowed

[bendavies]

@jsierles err, are you sure?

=> ERROR exporting cache      
...
failed to solve: rpc error: code = Unknown desc = error writing manifest blob: failed commit on ref "sha256:4ec7073e9d9c619a1320b7db019d1b5cf3b87994f636346bb7ec91165d5f751a": unexpected status: 405 Method Not Allowed

Are you mean pushing the image works fine, but not the manifest?

[jsierles]

@bendavies Am I sure of what? When I said 'pushing' works, I meant pushing the image itself, not the cache.

The 405 is an error that's previously been reported by people trying to use buildx against ECR and other hosted repositories. As @njdancer reports, this is supported by Docker Hub.

[TBBle]

@jsierles, @njdancer:

I guess the problems with uploading the layer cache are because the cache is another different media-type, compared to the existing container manifests or mainfest lists. That's worth raising in #308, which asked the question:

We're starting to look at this with a view to support, and would love to hear if there use cases for more than Helm charts that should publish to ECR, and what problem that would help solve.

Edit: Actually, looking at moby/buildkit#1143 the problem might be that buildkit's cache manifest-list uses a field that's not in the spec?

[michaelb990]

Hi --

Thank you for reporting this. We are looking into it and will post an update when we have further information.

[aviau]

I am also getting this:

#21 ERROR: error writing manifest blob: failed commit on ref "sha256:8e6b3dad496932ac63cd74fc32f6ab824fc14e650641b2a85c0df0a68a361f91": unexpected status: 400 Bad Request

[pieterza]

I can confirm that pushing the cache is still not working on ECR using mode=max and --push (or even --load)

failed to solve: rpc error: code = Unknown desc = error writing manifest blob: failed commit on ref "unexpected status: 405 Method Not Allowed

[dghubble]

Pushing image manifest lists to ECR works as expected for me. Using buildah (v1.14.8) and us-east-1 after the login dance for ECR.

VERSION=v0.0.1
IMAGE_REPO=ACCOUNTID.dkr.ecr.us-east-1.amazonaws.com/some-image

buildah manifest create dghubble/multiarch:${VERSION}
buildah manifest add dghubble/multiarch:${VERSION} docker://${IMAGE_REPO}@some-digest
buildah manifest add dghubble/multiarch:${VERSION} docker://${IMAGE_REPO}@another-digest
buildah manifest push -f v2s2 dghubble/multiarch:${VERSION} docker://${IMAGE_REPO}:${VERSION}

[michaelb990]

Thanks again for all the comments! We have identified an issue with how we are parsing manifest lists and are working on a fix. We'll let you know when the fix is deployed.

[bgaillard]

Hi @njdancer, i'm also trying to build a Docker image with Builkit and a command similar to the one you provided.

As you explained the command pushes the Docker image to ECR correctly when I remove the --export-cache and --import-cache options. The image cache does not succeed to be pushed.

I encounter a different error than yours when I have the --export-cache and --import-cache options.

...
#67 exporting cache
#67 sha256:2700d4ef94dee473593c5c614b55b2dedcca7893909811a8f2b48291a1f581e4
#67 preparing build cache for export
#67 preparing build cache for export 7.4s done
#67 ERROR: invalid incomplete links
------
 > importing cache manifest from XXXXXXXX.dkr.ecr.eu-west-1.amazonaws.com/myimage:buildcache:
------
------
 > exporting cache:
------
error: failed to solve: rpc error: code = Unknown desc = invalid incomplete links
Failed to build 'myimage' Docker image !

Do you encountered invalid incomplete links error too and succeeded to fix it ? Does somebody know what's the meaning of this error ?

Erratum
After more testing it appears that my problem is also encountered when I use --export-cache type=local so its probably not linked to Amazon ECR.

The line of code associated to my error is https://github.com/moby/buildkit/blob/bcc9df1d0ad2650b25e7e7b58f4732125028ac94/cache/remotecache/v1/utils.go#L151.

As its not linked to Amazon ECR i'll post an issue in the Builkit Github repo directly.

Thanks

[michaelb990]

We have made some fixes to make the annotations and platform fields optional to better support the OCI image index spec. There is an additional issue specifically related to using manifest lists as build caches (which reference layers, instead of referencing images). Due to this, buildkit caching will continue to return '405 Method Not Allowed'. We will track support for this as a separate feature in #876.

Closing this issue now that the manifest list fixes are deployed. Thanks!!

[Aposhian]

Still not working for cache manifests specifically: #876

[vl-shopback]

another year passed...

[systematicguy]

And another!

[TBBle]

If you're waiting for BuildKit layer cache support, see #876, particularly the latest comment.

This ticket was for Docker manifest lists (and OCI image indexes eventually too), and the existing BuildKit layer caching implementation didn't actually match the spec for manifest lists; this is actively being worked on.