Publish Crossplane image to xpkg.upbound.io

[hasheddan]

What problem are you facing?

A few months ago, many of the packages in the Crossplane ecosystem moved from Dockerhub to xpkg.upbound.io due to a few motivating factors:

• Rate limiting on pulls from Dockerhub.
• Automatic documentation support for all API types in a package on marketplace.upbound.io.
• Ability to search and discover packages based on their metadata on marketplace.upbound.io.

Crossplane itself is published as an OCI image and Helm chart, with the latter referencing the former. This differs from the structure of Crossplane packages, but there are a few commonalities, namely:

• Crossplane bundles its API types in its OCI image.
• Pulling the Crossplane image is still subsequent to rate limiting as imposed by whatever registry it is hosted on.

How could Crossplane help solve your problem?

Moving the Crossplane image to be hosted on xpkg.upbound.io would ensure that the community is not negatively impacted by any rate limiting, and would create the opportunity for the Crossplane API documentation to live alongside Crossplane packages (and similarly be automatically updated when new versions are published).

Moving the image would likely have very minimal impact on existing users, as users install Crossplane by downloading the Helm chart, which would be updated to reference the new location by default. There are two cases where a user could potentially have difficulty in upgrading:

• They have Dockerhub on an allow list in their cluster and do not have xpkg.upbound.io on the list.
• They are automatically syncing Crossplane versions from Dockerhub to an internal registry and then accessing the internal registry from their cluster.

Both of these issues could be mitigated by proper community messaging and a reasonable transition window. I would propose that we dual publish to Dockerhub and xpkg.upbound.io and update to point at xpkg.upbound.io in the Crossplane helm chart by default for the next release (v1.11.0). This would be accompanied by messaging in release notes and Slack prior to the release. At the next release (v1.12.0), Dockerhub could be dropped as an official host. This would give a reasonable window for users that need to update any internal systems to pull from xpkg.upbound.io a full release cycle to do so (they would be required to upgrade with docker.io specified in the values file for the mean time), while going ahead and getting the vast majority of the community over to xpkg.upbound.io on v1.11.0.

Amendment

Instead of moving the default in the Helm chart in v1.11.0, we will add a note in the v1.11.0 release notes that the default will be changing in v1.12.0, then stop publishing to Dockerhub and update the default as part of the v1.12.0 release.

[maximilianbraun]

👋 In my observation many 1* CNCF projects leverage gcr.io as a container registry, and others ghcr.io. I'd assume there are a couple of benefits coming from using gcr.io / ghcr.io like high availability, performance etc. I'm unaware of the infrastructure behind xpkg.upbound.io, maybe xpkg.upbound.io has similar capabilities. Did you evaluate one of those oci registries?

*1:
https://github.com/fluxcd/flux2/pkgs/container/flux-manifests
https://github.com/argoproj/argo-cd/pkgs/container/argo-cd%2Fargocd
https://github.com/etcd-io/etcd/releases/tag/v3.4.23
https://console.cloud.google.com/gcr/images/knative-releases/GLOBAL

[hasheddan]

@maximilianbraun thanks for this feedback! We currently use xpkg.upbound.io for all Crossplane packages due to the additional benefits of marketplace.upbound.io (auto-generated docs, discoverability, etc.). We would like for Crossplane releases to have similar benefits, while also standardizing on where you get all Crossplane images. gcr.io / ghcr.io are great options as well, but do not provide the aforementioned benefits. I believe the success of hosting all Crossplane packages on xpkg.upbound.io has instilled community confidence in the availability and performance of it as a registry.

[negz]

I just realized we never actually switched the Helm chart over to use the new xpkg.crossplane.io release that we've been publishing since v1.11. We should with v1.13.

[jbw976]

For validation and posterity, I just tested the installation of this new Helm chart from the master channel, as well as an upgrade from stable to master. Things seem to be working OK, i.e. an existing Crossplane installation that was using the crossplane/crossplane image from dockerhub goes through the upgrade and starts using the xpkg.upbound.io/crossplane/crossplane image 🎉

Clean install from master

Install master helm chart into a clean cluster:

❯ helm install crossplane \
--namespace crossplane-system \
--create-namespace crossplane-master/crossplane \
--devel

Crossplane pods are in the Running status:

❯ kcs get pod
NAME                                       READY   STATUS    RESTARTS   AGE
crossplane-67984fd6d7-2h2vk                1/1     Running   0          58s
crossplane-rbac-manager-675bb466dd-rfcw8   1/1     Running   0          58s

Pods are using the latest version from master, which is from xpkg.upbound.io:

❯ kcs get pod -o json | jq '.items[] | .metadata.name + " " + .spec.containers[].image'
"crossplane-67984fd6d7-2h2vk xpkg.upbound.io/crossplane/crossplane:v1.14.0-rc.0.663.g0f6b3ccd"
"crossplane-rbac-manager-675bb466dd-rfcw8 xpkg.upbound.io/crossplane/crossplane:v1.14.0-rc.0.663.g0f6b3ccd"

Upgrade scenario

Installing stable

Install stable helm chart into a clean cluster:

❯ helm install crossplane \
--namespace crossplane-system \
--create-namespace crossplane-stable/crossplane

Crossplane pods are in the Running status:

❯ kcs get pods
NAME                                       READY   STATUS    RESTARTS   AGE
crossplane-959448d58-mqp4q                 1/1     Running   0          27s
crossplane-rbac-manager-6994f54b46-t5vqh   1/1     Running   0          27s

Pods are on the stable v1.13.2 version:

❯ kcs get pod -o json | jq '.items[] | .metadata.name + " " + .spec.containers[].image'
"crossplane-959448d58-mqp4q crossplane/crossplane:v1.13.2"
"crossplane-rbac-manager-6994f54b46-t5vqh crossplane/crossplane:v1.13.2"

Upgrading to master

Run the upgrade command to go from stable channel to master channel:

❯ helm --namespace crossplane-system upgrade crossplane crossplane-master/crossplane --devel

New pods are running OK:

❯ kcs get pod
NAME                                       READY   STATUS    RESTARTS   AGE
crossplane-67984fd6d7-t4lxd                1/1     Running   0          4m2s
crossplane-rbac-manager-675bb466dd-b4d8p   1/1     Running   0          4m2s

New pods are running latest versions from master that are from xpkg.upbound.io:

❯ kcs get pod -o json | jq '.items[] | .metadata.name + " " + .spec.containers[].image'
"crossplane-67984fd6d7-t4lxd xpkg.upbound.io/crossplane/crossplane:v1.14.0-rc.0.663.g0f6b3ccd"
"crossplane-rbac-manager-675bb466dd-b4d8p xpkg.upbound.io/crossplane/crossplane:v1.14.0-rc.0.663.g0f6b3ccd"