Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Push multi-arch manifests to quay.io #4227

Closed
invidian opened this issue Nov 30, 2020 · 6 comments
Closed

Push multi-arch manifests to quay.io #4227

invidian opened this issue Nov 30, 2020 · 6 comments
Labels
kind/enhancement

Comments

@invidian
Copy link
Contributor

Expected Behavior

Replacing calico/cni:v1.17.0 with quay.io/calico/cni:v1.17.0 should work on ARM platforms.

Current Behavior

quay.io registry is returning different manifests than docker/io, even though Quay supports multi-arch manifests.

Manifest from docker.io:

$ docker manifest inspect calico/cni:v3.17.0
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
   "manifests": [
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 946,
         "digest": "sha256:b033b5237cfd233974c5dff2379ad83b81cb938f48083d392c83a1cead537149",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 946,
         "digest": "sha256:5f95f08ee0ee629d29653a90bdbdb4a5fa9d6bc2b251083809dbb781e81fa7a3",
         "platform": {
            "architecture": "arm64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 946,
         "digest": "sha256:5775ca2345692726ede299f038f8c00006edf1ae0e90fecf6dcd59547eeca979",
         "platform": {
            "architecture": "ppc64le",
            "os": "linux"
         }
      }
   ]
}

Manifes from quay.io:

$ docker manifest inspect quay.io/calico/cni:v3.17.0
{
        "schemaVersion": 2,
        "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
        "config": {
                "mediaType": "application/vnd.docker.container.image.v1+json",
                "size": 3364,
                "digest": "sha256:94159b46f8d8265255a5ac26812e70c6199d2d9a8ff01e1b18c13dc1940a515b"
        },
        "layers": [
                {
                        "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
                        "size": 18091,
                        "digest": "sha256:65c8c9bc8b9d70f6caf351e4e5f119a0ace1523b4cb5d7544695b387158e4696"
                },
                {
                        "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
                        "size": 4064,
                        "digest": "sha256:aa83ec2d163535222f2446327c1a6bb1c823fdda81a30b37cd086912d52f3648"
                },
                {
                        "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
                        "size": 44027141,
                        "digest": "sha256:c40275fca228cdb20742bfd4db7d3322c144e6aebfe1f3955d5adcac66f63a22"
                }
        ]
}

Sample multi-arch manifest from quay.io:

$ docker manifest inspect quay.io/kinvolk/checkpoint:43ec4b414e44f202e07bf43e57d2b5ffbcfd4415
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
   "manifests": [
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 740,
         "digest": "sha256:3f2f0340764b1fb7ec432b824316e5e7b9802c960b059679e90d0709b241244d",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 740,
         "digest": "sha256:44e6cb23dda113ec15b67ed5232fc2ac104925cb0da10cd6e17fa9cde1dcd28c",
         "platform": {
            "architecture": "arm64",
            "os": "linux"
         }
      }
   ]
}

Context

Because of new Docker Hub pull limits, we would like to use Quay as a main registry to pull Calico images from, to make cluster creation process more reliable (kinvolk/lokomotive#1225), as currently we sometimes hit limits when creating virtual clusters (so using the same public IP all the time).
@invidian invidian mentioned this issue Nov 30, 2020
Merged
invidian added a commit to kinvolk/lokomotive that referenced this issue Nov 30, 2020
@invidian
assets: pull Calico images from Quay instead of Docker Hub
7d17df6 
To mitigate pulling limits in Docker Hub.

Calico currently offers official images in Quay, however they do not
support multi-arch manifest, so for the time being we need to use
manually built manifests based on images from Quay.

This issue is reported to upstream Calico:
projectcalico/calico#4227

Part of #1225.

Signed-off-by: Mateusz Gozdek <mateusz@kinvolk.io>
@caseydavenport
Copy link
Member

When we implemented our release infrastructure quay.io did not support multi-arch manifests, and so we have separate images pushed to quay.

If quay.io now supports multi-arch manifests, that is great news! We should update our release process to publish those as well.

@caseydavenport caseydavenport changed the title Images in quay.io does not provide multi-arch manifests Push multi-arch manifests to quay.io Nov 30, 2020
@invidian
Copy link
Contributor Author

invidian commented Apr 1, 2021

It seems this is now fixed:

$ docker manifest inspect calico/cni:v3.18.1
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
   "manifests": [
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 946,
         "digest": "sha256:b946fb6149b0dce3324d86eadf2d98ded4931e254d261a92eacf3a5d8441708d",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 946,
         "digest": "sha256:0b32ed73e81e687584b48b3da166e177e0ffa96fbf01d2c6f2dcfc7c11c317cf",
         "platform": {
            "architecture": "arm64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 946,
         "digest": "sha256:5013158080c0625799c6046d17d00e7462c959403cd215115c9a6d29372a5e0f",
         "platform": {
            "architecture": "ppc64le",
            "os": "linux"
         }
      }
   ]
}

@invidian invidian closed this as completed Apr 1, 2021
@invidian invidian mentioned this issue Apr 1, 2021
Open
@surajssd surajssd mentioned this issue Jun 23, 2021
Closed
@aquam8
Copy link

aquam8 commented Jun 28, 2021

@invidian I think you closed this by mistake as the Quay.io images still do not have ARM64 support...

@invidian
Copy link
Contributor Author

Indeed, it seems at least one of the image is still missing ARM64 support:

$ docker manifest inspect quay.io/calico/cni:v3.19.1
{
        "schemaVersion": 2,
        "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
        "config": {
                "mediaType": "application/vnd.docker.container.image.v1+json",
                "size": 3363,
                "digest": "sha256:5749e8b276f9b5aea2683d8a0d59afd05a2be478f5092fbaadc4715f4db76e5a"
        },
        "layers": [
                {
                        "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
                        "size": 18093,
                        "digest": "sha256:740c37ed87bd03bc3193d1382b3b836328022c78911d62d0f9b508d9a37abc1a"
                },
                {
                        "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
                        "size": 4063,
                        "digest": "sha256:5019aa621b5376714f3e9aa04a6f264a857f639c536a587b5a1670b1e2699e62"
                },
                {
                        "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
                        "size": 48310687,
                        "digest": "sha256:b0c76401258249c3fb3308cde0f71a626d225a5ede0dcef81c23b430cbb716da"
                }
        ]
}

@invidian invidian reopened this Jun 28, 2021
@yuha0 yuha0 mentioned this issue Sep 30, 2021
Closed
@frozenprocess
Copy link
Collaborator

frozenprocess commented Feb 23, 2022 •
edited
Loading

Fixed in v3.21.3+ projectcalico/go-build#351

docker manifest inspect calico/cni:release-v3.21
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
   "manifests": [
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 946,
         "digest": "sha256:2fc54a8e3cb84c76f426c4b4ba9751b1b06f9dcbfa3d6acfaebb1363edcc6bd8",
         "platform": {
            "architecture": "arm",
            "os": "linux",
            "variant": "v7"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 1369,
         "digest": "sha256:fa29d367597f2146ead38d8d4b686928569772d883da859e121bdeb0b5aec63e",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 946,
         "digest": "sha256:ca5fc085a81ab94d82f2f47ca48c921782ec3271362b99e830bd42fbd33be4f3",
         "platform": {
            "architecture": "arm64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 946,
         "digest": "sha256:25a15fa9b307c9eedd4e84888db252974eabd39bd2dee062bca7eed303e4da81",
         "platform": {
            "architecture": "ppc64le",
            "os": "linux"
         }
      }
   ]
}


docker manifest inspect quay.io/calico/cni:release-v3.21
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
   "manifests": [
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 946,
         "digest": "sha256:2fc54a8e3cb84c76f426c4b4ba9751b1b06f9dcbfa3d6acfaebb1363edcc6bd8",
         "platform": {
            "architecture": "arm",
            "os": "linux",
            "variant": "v7"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 1369,
         "digest": "sha256:fa29d367597f2146ead38d8d4b686928569772d883da859e121bdeb0b5aec63e",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 946,
         "digest": "sha256:ca5fc085a81ab94d82f2f47ca48c921782ec3271362b99e830bd42fbd33be4f3",
         "platform": {
            "architecture": "arm64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 946,
         "digest": "sha256:25a15fa9b307c9eedd4e84888db252974eabd39bd2dee062bca7eed303e4da81",
         "platform": {
            "architecture": "ppc64le",
            "os": "linux"
         }
      }
   ]
}

@invidian
Copy link
Contributor Author

Awesome!

@vpnachev vpnachev mentioned this issue Jun 9, 2023
Merged
This was referenced Aug 9, 2023
Open
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/enhancement
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@aquam8 @caseydavenport @invidian @frozenprocess