Support multiple XR versions for handling breaking API changes

[samidbb]

What problem are you facing?

I have a composite resource called XAWSBucket. and when a claim deployed to my cluster it should provision a bucket on AWS in repsonse.

Here is the XRD:

apiVersion: apiextensions.crossplane.io/v1
kind: CompositeResourceDefinition
metadata:
  name: xawsbuckets.storage.crossplane.myorg
spec:
  group: storage.crossplane.myorg
  names:
    kind: XAWSBucket
    plural: xawsbuckets
  claimNames:
    kind: AWSBucket
    plural: awsbuckets
  versions: 
  - name: v1alpha1
    served: true
    referenceable: true
    schema:
      openAPIV3Schema:
        type: object
        properties:                  
          spec:
            type: object
            properties:
              parameters:
                type: object
                properties:
                  acl:
                    description: The canned ACL to apply to the bucket. Note that
                      either canned ACL or specific access permissions are required.
                      If neither (or both) are provided, the creation of the bucket
                      will fail.
                    enum:
                    - private
                    - public-read
                    - public-read-write
                    - authenticated-read
                    type: string
                  locationConstraint:
                    description: LocationConstraint specifies the Region where the
                      bucket will be created. It is a required field.
                    type: string
                  deletionPolicy: 
                    description: Specify whether the actual cloud resource should be deleted when this managed resource is deleted in Kubernetes API server. Possible values are Delete (the default) and Orphan
                    type: string
                    default: "Delete"                     
                required:
                - locationConstraint
            required:
            - parameters

the composition:

apiVersion: apiextensions.crossplane.io/v1
kind: Composition
metadata:
  name: xawsbuckets.storage.crossplane.myorg
  labels:
    version: v1
spec:
  compositeTypeRef:
    apiVersion: storage.crossplane.myorg/v1alpha1
    kind: XAWSBucket

  patchSets:
  - name: configname
    patches:
    - fromFieldPath: spec.claimRef.namespace    
      toFieldPath: spec.providerConfigRef.name 
      transforms:
      - type: string
        string:
          fmt: "%s-aws"
      policy:
        fromFieldPath: Required
  resources:
  - name: bucket
    base:
      apiVersion: s3.aws.crossplane.io/v1beta1
      kind: Bucket
      spec:
        forProvider:
          locationConstraint: eu-west-1
    patches:
    - type: PatchSet
      patchSetName: configname
    - fromFieldPath: metadata.name
      toFieldPath: metadata.name          
    - fromFieldPath: spec.parameters
      toFieldPath: spec.forProvider
    - fromFieldPath: spec.parameters.deletionPolicy
      toFieldPath: spec.deletionPolicy
    - type: FromCompositeFieldPath
      fromFieldPath: "metadata.annotations[crossplane.io/external-name]"      

And the claim:

apiVersion: storage.crossplane.myorg/v1alpha1
kind: AWSBucket
metadata:
  name: s3samdiv1
  namespace: my-namespace
spec:
  parameters:
    locationConstraint: eu-west-1
    acl: public-read
    deletionPolicy: Delete
  compositionSelector:
    matchLabels:
      version: v1

Then I decide to rename locationConstraint field to region. And I consider this to be a breaking change because it requires users to change their claim manifests.
So I add a new schema version to my XRD with the updated field name and a new composition.

Here is my updated XRD:

apiVersion: apiextensions.crossplane.io/v1
kind: CompositeResourceDefinition
metadata:
  name: xawsbuckets.storage.crossplane.myorg
spec:
  group: storage.crossplane.myorg
  names:
    kind: XAWSBucket
    plural: xawsbuckets
  claimNames:
    kind: AWSBucket
    plural: awsbuckets
  versions: 
  - name: v1alpha1
    served: true
    referenceable: false
    schema:
      openAPIV3Schema:
        type: object
        properties:                  
          spec:
            type: object
            properties:
              parameters:
                type: object
                properties:
                  acl:
                    description: The canned ACL to apply to the bucket. Note that
                      either canned ACL or specific access permissions are required.
                      If neither (or both) are provided, the creation of the bucket
                      will fail.
                    enum:
                    - private
                    - public-read
                    - public-read-write
                    - authenticated-read
                    type: string
                  locationConstraint:
                    description: LocationConstraint specifies the Region where the
                      bucket will be created. It is a required field.
                    type: string
                  deletionPolicy: 
                    description: Specify whether the actual cloud resource should be deleted when this managed resource is deleted in Kubernetes API server. Possible values are Delete (the default) and Orphan
                    type: string
                    default: "Delete"                     
                required:
                - locationConstraint
            required:
            - parameters
  - name: v1alpha2
    served: true
    referenceable: true
    schema:
      openAPIV3Schema:
        type: object
        properties:     
          spec:
            type: object
            properties:
              parameters:
                type: object
                properties:
                  region:
                    description: Geographic location of this Database server.
                    type: string
                    enum: ["ireland", "germany"]
                    default: "ireland"                
                  acl:
                    type: string
                  deletionPolicy: 
                    description: Specify whether the actual cloud resource should be deleted when this managed resource is deleted in Kubernetes API server. Possible values are Delete (the default) and Orphan
                    type: string
                    default: "Delete"                    

Here is the added composition:

apiVersion: apiextensions.crossplane.io/v1
kind: Composition
metadata:
  name: xawsbucketsv2.storage.crossplane.myorg
  labels:
    version: v2
spec:
  compositeTypeRef:
    apiVersion: storage.crossplane.myorg/v1alpha2 # v2
    kind: XAWSBucket

  patchSets:
  - name: configname
    patches:
    - fromFieldPath: spec.claimRef.namespace    
      toFieldPath: spec.providerConfigRef.name 
      transforms:
      - type: string
        string:
          fmt: "%s-aws"
      policy:
        fromFieldPath: Required
  resources:
  - name: bucket
    base:
      apiVersion: s3.aws.crossplane.io/v1beta1
      kind: Bucket
      spec:
        forProvider:
          locationConstraint: eu-west-1
    patches:
    - type: PatchSet
      patchSetName: configname
    - fromFieldPath: metadata.name
      toFieldPath: metadata.name
    - type: ToCompositeFieldPath
      fromFieldPath: "metadata.name"
      toFieldPath: "status.createdResources.bucket"
    - fromFieldPath: "spec.parameters.region" # v2
      toFieldPath: "spec.forProvider.locationConstraint"
      transforms:
      - type: map
        map:
          ireland: eu-west-1
          germany: eu-central-1
    - fromFieldPath: "spec.parameters.acl"
      toFieldPath: "spec.forProvider.acl"
    - fromFieldPath: spec.parameters.deletionPolicy
      toFieldPath: spec.deletionPolicy
    - type: FromCompositeFieldPath
      fromFieldPath: "metadata.annotations[crossplane.io/external-name]"      

and the claim for v1alpha2 schema:

apiVersion: storage.crossplane.myorg/v1alpha2
kind: AWSBucket
metadata:
  name: s3samdiv2
  namespace: my-namespace
spec:
  parameters:
    region: ireland
    acl: public-read
    deletionPolicy: Delete    
  compositionSelector:
    matchLabels:
      version: v2

So the migration path for the v1alpha1 composite resources will involve the following steps:

1. Update deletionPolicy on the claim to Orphan
2. Delete v1alpha1 claim
3. Create a claim targeting v1alpha2 and set external name to the name of orphaned resources from step 1)

Following behavior should be supported:

• New bucket resource can only be created using v1alpha2 schema
• It's still possible to update and delete v1alpha1 resources

The problems

1. When I update deletionPolicy field using the v1alpha1 claims the changes is not reflected in the old resources. I can se that when I do kubectl describe bucket. So deletion policy is not changed to Orphan.

When I run kubectl describe composite I can see that there is an error/warning:

Events:
  Type     Reason             Age                    From                                                             Message
  ----     ------             ----                   ----                                                             -------
  Warning  SelectComposition  3m40s (x6 over 6m10s)  defined/compositeresourcedefinition.apiextensions.crossplane.io  no compatible Compositions found
  Normal   SelectComposition  100s (x5 over 3m10s)   defined/compositeresourcedefinition.apiextensions.crossplane.io  Successfully selected composition
  Normal   ComposeResources   100s (x5 over 3m10s)   defined/compositeresourcedefinition.apiextensions.crossplane.io  Successfully composed resources
  Normal   SelectComposition  6s (x4 over 96s)       defined/compositeresourcedefinition.apiextensions.crossplane.io  Successfully selected composition
  Warning  ComposeResources   6s (x4 over 96s)       defined/compositeresourcedefinition.apiextensions.crossplane.io  referenced composition is not compatible with this composite resource

2. It also hangs when I try to delete the v1alpha1 claims. So it means that I cannot delete them

v1alpha2 claims run without issues.

How could Crossplane help solve your problem?

Is there anything I'm missing? We want to understand if multiple version can be run like this or is there a better way to handle breaking changes in the API schema?

[negz]

Thanks for raising this @samidbb! We currently don't really support multiple versions - technically we do but both versions must have identical schemas so it's not very useful. #2608 tracks improving that.

One thing to note is that the way Kubernetes handles multiple versions is sometimes a bit surprising - you wouldn't actually need to delete the old (v1alpha1) version and create a new (v1alpha2) version. When you teach Crossplane/Kubernetes about a new version the resource exists at both versions simultaneously - it helps to think about it as adding a new API version that's accessing the same underlying data.

[negz]

We want to understand if multiple version can be run like this or is there a better way to handle breaking changes in the API schema?

To more directly answer your question - no you're not missing anything. This is just something we don't yet support in Crossplane. We'd like to though. I believe it will be critical.

In the meantime, a workaround would be:

1. Mark the locationConstraint field optional and deprecated in your XRD schema (i.e. update the comment to say so).
2. Introduce an identical, optional region field.
3. Update your Composition such that it will patch from both region and locationConstraint to the same field.

This has some downsides - region effectively becomes optional - but allows you to rename without a breaking change until we support them.

[samidbb]

I couldn't really get it to work unfortunately. What it doesn't work for me is the following:

• I still cannot update deletionPolicy on v1 claims to Orphan. The change does not get propagated to the underlying composite resource. I can see it by describing the bucket and the composite resources
• I cannot delete v1 claims. it just get stuck on deletion request. Then I have to go the other way around by getting the composite and delete it which will then delete the bucket resource and v1 claim that is attached to it.

[negz]

@samidbb I'm not following. Are you seeing the two issues you reported when you create an XRD with two identical schemas? As I mentioned above, we don't support multiple different schemas yet per #2608. My most recent comment was suggesting a way to achieve the change you want to make in a backward compatible way so you can do so without introducing a new version.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[github-actions]

Crossplane does not currently have enough maintainers to address every issue and pull request. This issue has been automatically marked as stale because it has had no activity in the last 90 days. It will be closed in 7 days if no further activity occurs. Leaving a comment starting with /fresh will mark this issue as not stale.

[jbw976]

/fresh I believe this is still relevant as part of supporting effective composition revisions

[github-actions]

Crossplane does not currently have enough maintainers to address every issue and pull request. This issue has been automatically marked as stale because it has had no activity in the last 90 days. It will be closed in 7 days if no further activity occurs. Leaving a comment starting with /fresh will mark this issue as not stale.

[ytsarev]

/fresh multiple XRD versions with different schemas are still desired enhancements

[mithie]

This would be a critical feature for us. It is obvious that schemas/APIs will evolve over time. Not having the chance to implement a stable versioning strategy and running different versions in parallel makes the whole thing pretty useless. Kubernetes already provides that possibility. So I would expect that Crossplane does, too.

[negz]

This is a duplicate of #2608 - let's discuss this there.