Failed to push multi-arch image

[rgl]

Pushing an example multi-arch (aka multi-architecture) image fails with:

source_image='docker.io/ruilopes/example-docker-buildx-go:v1.10.0'
image='keppel.test:9006/ruilopes/example-docker-buildx-go:v1.10.0'
#platform='linux/amd64'
#platform='windows/amd64:10.0.20348.825'
platform='all'
crane copy --insecure --platform "$platform" "$source_image" "$image"

2022/08/27 16:14:20 Copying from docker.io/ruilopes/example-docker-buildx-go:v1.10.0 to keppel.test:9006/ruilopes/example-docker-buildx-go:v1.10.0
2022/08/27 16:14:22 retrying without mount: POST http://keppel.test:9006/v2/ruilopes/example-docker-buildx-go/blobs/uploads/?from=ruilopes%2Fexample-docker-buildx-go&mount=sha256%3A1efc276f4ff952c055dea726cfc96ec6a4fdb8b62d9eed816bd2b788f2860ad7&origin=REDACTED: BLOB_UNKNOWN: blob does not exist in source repository
2022/08/27 16:14:22 retrying without mount: POST http://keppel.test:9006/v2/ruilopes/example-docker-buildx-go/blobs/uploads/?from=ruilopes%2Fexample-docker-buildx-go&mount=sha256%3A460e86bd30d96aa1df7697b4fb8184599645dd748a9c473b3fa921aafe42bf56&origin=REDACTED: BLOB_UNKNOWN: blob does not exist in source repository
2022/08/27 16:14:22 retrying without mount: POST http://keppel.test:9006/v2/ruilopes/example-docker-buildx-go/blobs/uploads/?from=ruilopes%2Fexample-docker-buildx-go&mount=sha256%3Aad526f14af1520ff0694848b88732134fbae909cdef95c02c49efcc7a2d6ab7b&origin=REDACTED: BLOB_UNKNOWN: blob does not exist in source repository
2022/08/27 16:14:22 pushed blob: sha256:ad526f14af1520ff0694848b88732134fbae909cdef95c02c49efcc7a2d6ab7b
2022/08/27 16:14:22 pushed blob: sha256:460e86bd30d96aa1df7697b4fb8184599645dd748a9c473b3fa921aafe42bf56
2022/08/27 16:14:23 pushed blob: sha256:1efc276f4ff952c055dea726cfc96ec6a4fdb8b62d9eed816bd2b788f2860ad7
2022/08/27 16:14:23 keppel.test:9006/ruilopes/example-docker-buildx-go@sha256:2ebebdde436cbbfea50bf5a4eb20b673029dbe7a68577b4fcf42aec122b5988a: digest: sha256:2ebebdde436cbbfea50bf5a4eb20b673029dbe7a68577b4fcf42aec122b5988a size: 740
2022/08/27 16:14:23 retrying without mount: POST http://keppel.test:9006/v2/ruilopes/example-docker-buildx-go/blobs/uploads/?from=ruilopes%2Fexample-docker-buildx-go&mount=sha256%3Ab45a7254211aad549686c27ca941f0d3e9897a56c425fd2bafe3b19e9ea67c4c&origin=REDACTED: BLOB_UNKNOWN: blob does not exist in source repository
2022/08/27 16:14:23 retrying without mount: POST http://keppel.test:9006/v2/ruilopes/example-docker-buildx-go/blobs/uploads/?from=ruilopes%2Fexample-docker-buildx-go&mount=sha256%3Aa9fe95647e78b5516c7e2327355b6996e2ea295cd76ae242cbfe87f016b4e760&origin=REDACTED: BLOB_UNKNOWN: blob does not exist in source repository
2022/08/27 16:14:23 retrying without mount: POST http://keppel.test:9006/v2/ruilopes/example-docker-buildx-go/blobs/uploads/?from=ruilopes%2Fexample-docker-buildx-go&mount=sha256%3A79d4b8007a95f1971e12d2259d85c2e9f845a2c33541c4ea2893bc34d4f4b013&origin=REDACTED: BLOB_UNKNOWN: blob does not exist in source repository
2022/08/27 16:14:23 pushed blob: sha256:79d4b8007a95f1971e12d2259d85c2e9f845a2c33541c4ea2893bc34d4f4b013
2022/08/27 16:14:23 pushed blob: sha256:b45a7254211aad549686c27ca941f0d3e9897a56c425fd2bafe3b19e9ea67c4c
2022/08/27 16:14:24 pushed blob: sha256:a9fe95647e78b5516c7e2327355b6996e2ea295cd76ae242cbfe87f016b4e760
2022/08/27 16:14:24 keppel.test:9006/ruilopes/example-docker-buildx-go@sha256:ffb3b11112406f18c84308db56d2feabd09e88130671c2d23f44981fc8c2f047: digest: sha256:ffb3b11112406f18c84308db56d2feabd09e88130671c2d23f44981fc8c2f047 size: 740
2022/08/27 16:14:24 retrying without mount: POST http://keppel.test:9006/v2/ruilopes/example-docker-buildx-go/blobs/uploads/?from=ruilopes%2Fexample-docker-buildx-go&mount=sha256%3A1dd75a3a9c893a7dc313f683dd62464b7eab6c6d522ee62c8a17022631830f32&origin=REDACTED: BLOB_UNKNOWN: blob does not exist in source repository
2022/08/27 16:14:24 retrying without mount: POST http://keppel.test:9006/v2/ruilopes/example-docker-buildx-go/blobs/uploads/?from=ruilopes%2Fexample-docker-buildx-go&mount=sha256%3A220f1d0b5fbaa132ac1b5a1d18977092c24de291ecb3c673c2fa032084f0079c&origin=REDACTED: BLOB_UNKNOWN: blob does not exist in source repository
2022/08/27 16:14:24 retrying without mount: POST http://keppel.test:9006/v2/ruilopes/example-docker-buildx-go/blobs/uploads/?from=ruilopes%2Fexample-docker-buildx-go&mount=sha256%3A714f60a76c9a842027926b69d9645c4a089c6f1bbfcbd0854d5de090bbc90251&origin=REDACTED: BLOB_UNKNOWN: blob does not exist in source repository
2022/08/27 16:14:24 pushed blob: sha256:714f60a76c9a842027926b69d9645c4a089c6f1bbfcbd0854d5de090bbc90251
2022/08/27 16:14:24 pushed blob: sha256:220f1d0b5fbaa132ac1b5a1d18977092c24de291ecb3c673c2fa032084f0079c
2022/08/27 16:14:24 pushed blob: sha256:1dd75a3a9c893a7dc313f683dd62464b7eab6c6d522ee62c8a17022631830f32
2022/08/27 16:14:24 keppel.test:9006/ruilopes/example-docker-buildx-go@sha256:0e5aef05e2cfc763739f1395de46a862dc21714a1258471c08e55be0d148db0b: digest: sha256:0e5aef05e2cfc763739f1395de46a862dc21714a1258471c08e55be0d148db0b size: 740
2022/08/27 16:14:25 retrying without mount: POST http://keppel.test:9006/v2/ruilopes/example-docker-buildx-go/blobs/uploads/?from=ruilopes%2Fexample-docker-buildx-go&mount=sha256%3Ab5fa2ec8efc4e66fc023da0e4a2915a7eda3dc307c2aeb8b95eb60761d1fc204&origin=REDACTED: BLOB_UNKNOWN: blob does not exist in source repository
2022/08/27 16:14:25 retrying without mount: POST http://keppel.test:9006/v2/ruilopes/example-docker-buildx-go/blobs/uploads/?from=ruilopes%2Fexample-docker-buildx-go&mount=sha256%3Abbe151a29e8136b6d96996e67f266932d724cee589d14cc4ea21aa69dc305c7f&origin=REDACTED: BLOB_UNKNOWN: blob does not exist in source repository
2022/08/27 16:14:25 retrying without mount: POST http://keppel.test:9006/v2/ruilopes/example-docker-buildx-go/blobs/uploads/?from=ruilopes%2Fexample-docker-buildx-go&mount=sha256%3A896a00309dd817a9fda0ecf3ab240aa075f802dd203218cc10b9ec6bdd27aa75&origin=REDACTED: BLOB_UNKNOWN: blob does not exist in source repository
2022/08/27 16:14:25 pushed blob: sha256:896a00309dd817a9fda0ecf3ab240aa075f802dd203218cc10b9ec6bdd27aa75
2022/08/27 16:14:25 pushed blob: sha256:b5fa2ec8efc4e66fc023da0e4a2915a7eda3dc307c2aeb8b95eb60761d1fc204
2022/08/27 16:14:25 pushed blob: sha256:bbe151a29e8136b6d96996e67f266932d724cee589d14cc4ea21aa69dc305c7f
Error: failed to copy index: PUT http://keppel.test:9006/v2/ruilopes/example-docker-buildx-go/manifests/sha256:de7f83381ed864f79e21f2a7a80d89896f88ad8b55c64fbfbe20be6b232ad818: MANIFEST_BLOB_UNKNOWN: manifest blob unknown to registry; sha256:5d24e1a2f5c566b0afb1e46fc24e5cec821c8ebf44220276a95a2b91f44a2f2a

For reference, this is how the manifest looks:

crane manifest --insecure "$source_image" | jq

{
  "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
  "schemaVersion": 2,
  "manifests": [
    {
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "digest": "sha256:2ebebdde436cbbfea50bf5a4eb20b673029dbe7a68577b4fcf42aec122b5988a",
      "size": 740,
      "platform": {
        "architecture": "amd64",
        "os": "linux"
      }
    },
    {
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "digest": "sha256:ffb3b11112406f18c84308db56d2feabd09e88130671c2d23f44981fc8c2f047",
      "size": 740,
      "platform": {
        "architecture": "arm64",
        "os": "linux"
      }
    },
    {
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "digest": "sha256:0e5aef05e2cfc763739f1395de46a862dc21714a1258471c08e55be0d148db0b",
      "size": 740,
      "platform": {
        "architecture": "arm",
        "os": "linux",
        "variant": "v7"
      }
    },
    {
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "digest": "sha256:de7f83381ed864f79e21f2a7a80d89896f88ad8b55c64fbfbe20be6b232ad818",
      "size": 1128,
      "platform": {
        "architecture": "amd64",
        "os": "windows",
        "os.version": "10.0.17763.3165"
      }
    },
    {
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "digest": "sha256:b91cc7beb040a3c607d8000d1f617a16fee2c59f112e66d8ed47cc85c119970a",
      "size": 1128,
      "platform": {
        "architecture": "amd64",
        "os": "windows",
        "os.version": "10.0.20348.825"
      }
    }
  ]
}

[SuperSandro2000]

Does crane first copy the images for all arches and then the multi arch manifest? Due to keppels database schema this is required and cannot be done the other way around.

[rgl]

From the attached logs it seems to do that.

It seems to fail when pushing the de7f83381ed864f79e21f2a7a80d89896f88ad8b55c64fbfbe20be6b232ad818 manifest, which is the first windows image. Please be aware that the windows images have a base foreign layer (non-redistribute) that points to an external Microsoft repository.

[SuperSandro2000]

Please be aware that the windows images have a base foreign layer (non-redistribute) that points to an external Microsoft repository.

That sounds like it could be the problem. I don't think we tested keppel so far with windows images.

[Zhurik]

that looks like intentional behaviour
https://docs.docker.com/reference/cli/dockerd/#allow-push-of-non-distributable-artifacts