Unable to serve multiple Composite apiVersions at the same time

[AaronME]

What happened?

I'm currently running several databases on version v1 of a composite. I would like to make v2alpha1 available without disabling v1. (The specific case is to support migrating compositions to the official-providers from Upbound).

My case includes being able to test both versions in my CI pipeline, where both will need to be deployed to a control plane to test behavior.

Considering that only one version may be referenceable, and that compositions must match a referenceable version, it is unclear how to support multiple versions on the same control plane.

How can we reproduce it?

1. Create A Composite (XMysqlDatabase) with two versions like so:

  defaultCompositionRef:
    name: v2.mysqldatabases.platform.example.io
  versions:
    - name: v1
      served: true
      referenceable: false
      schema:
        openAPIV3Schema:
          type: object
          properties:
            ...
    - name: v2alpha1
      served: true
      referenceable: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            ...

1. Prepare two compositions, each referencing a different version:

apiVersion: apiextensions.crossplane.io/v1
kind: Composition
metadata:
  name: v1.mysqldatabases.platform.example.io
spec:
  compositeTypeRef:
    apiVersion: platform.example.io/v1
    kind: XMySQLDatabase
...
---
apiVersion: apiextensions.crossplane.io/v1
kind: Composition
metadata:
  name: v2.mysqldatabases.platform.example.io
spec:
  compositeTypeRef:
    apiVersion: mysqldatabases.platform.example.io/v2alpha1
    kind: XMySQLDatabase
...

1. Deploy this Composite to a cluster and wait for it to become established and offered.
2. Create two claims:

apiVersion: platform.example.io/v1
kind: MySQLDatabase
metadata:
  name: example-database
spec:
  compositionRef:
    name: v1.mysqldatabases.platform.example.io
...
---
apiVersion: platform.example.io/v2alpha1
kind: MySQLDatabase
metadata:
  name: example-database-v2
...

1. Note that both will appear in the cluster as v2alpha1:

$ kubectl describe mysqldatabases
Name:         example-database-v2
API Version:  platform.example.io/v2alpha1
Kind:         MySQLDatabase
...
Name:         example-database
API Version:  platform.example.io/v2alpha1
Kind:         MySQLDatabase

What environment did it happen in?

Crossplane version:

• crossplane:v1.11.1-up.1
• kubernetes v1.24.7 (kind)

[jbw976]

Is this helpful and/or relevant @AaronME? #2608 (comment)

[AaronME]

@jbw976 I don't believe so.

That ticket appears to be discussing the ability to force v1 resources to be converted into v2alpha1 resources, with appropriate changes to the schema.

In our case, we explicitly want to support both versions while we test the new version. And we want to allow users to dictate when they "opt-in" to the new version.

[negz]

Considering that only one version may be referenceable, and that compositions must match a referenceable version, it is unclear how to support multiple versions on the same control plane.

Here's how it should work. Apologies if you already know all this - CRD versioning is unintuitive and none of it was obvious to me at first so I like to make sure folks are on the same page.

• When you have two versions of an XR (or really any Kubernetes custom resource) the API server always serves both versions. Think of it as two different API routes serving the same data with potentially slightly different schemas - like /v1/databases/foo and /v2/databases/foo. So whether you create a v1 XMysqlDatabase or a v2alpha1 XMysqlDatabase it will be served at both versions. You could create a v2alpha1 resource and you'd see it when you listed v1 resources. More details in the Kubernetes docs.
• In order to serve multiple versions (i.e. API shapes) the API server needs to know how to convert between them - e.g. how to turn a v1 resource into a v2alpha resource. Crossplane doesn't support conversion today. That means if you introduce v2alpha1 it needs to have the exact same schema as v1. This is obviously not very useful - supporting conversion is tracked in Conversion support for XRD versions? #2608.
• Crossplane's actual controllers (i.e. the XR and claim controllers) need to pick one version to operate on. No matter whether you create a v1 or v2alpha1 XMysqlDatabase, Crossplane will start a controller for whichever version is referenceable. Crossplane will only be able to use a Composition that targets the referenceable version.

Prepare two compositions, each referencing a different version

This isn't supported - I'm guessing this is where things are falling apart. Unfortunately I think it might not be possible to support this, because our XR controllers can't pick and choose which version of the XR they operate on.

Note that both will appear in the cluster as v2alpha1

They should also exist at v1, but I don't think kubectl describe mysqldatabases will show that. It always uses the newest version. I think if you used a manifest (e.g. kubectl describe -f foo.yaml, where foo.yaml contains apiVersion: foo/v1) you could force it to show you v1 results.

[negz]

The specific case is to support migrating compositions to the official-providers from Upbound

@AaronME could you elaborate a little on why you're introducing a new version? On the surface this doesn't seem like it would need a new XR version, and would instead only need new Compositions. I would only be introducing a new XR version if I needed to make a breaking schema change to my XR (e.g. add a new required field).

[AaronME]

I would only be introducing a new XR version if I needed to make a breaking schema change to my XR (e.g. add a new required field).

We are, in fact, making some breaking changes as we move ahead. New features and "lessons learned" are being included in the shift to official-providers. We believed that gating them behind a version would allow us to freely update the schema during testing without surprising users familiar with the old version.

To clarify:

• Version schemas can be changed, so long as those changes are backwards compatible all currently deployed compositions.
• Compositions can be changed, so long as they are compatible with the current referenceable version.

This means that anyone who wants the new version must change both the Version and the CompositionRef, since the default composition for v1 must remain identical for v2alpha1 to ensure currently-deployed Composites are not disturbed.

Do I have this right?

[AaronME]

I believe #2762 (comment) is the ultimate answer to my issue.

Thanks, @negz!