Cannot compose resources: cannot apply composed resource: existing object is not controlled by UID

[humoflife]

What happened?

Universal Crossplane cannot create a cluster resource in response to a claim because existing object is not controlled by UID.

How can we reproduce it?

• Create a kind cluster.
• Install a platform-ref-aws package.
• Apply a cluster claim from examples/cluster-claim.yaml.
• Remove the platform-ref-aws package.
• Install another version of the platform-ref-aws package.
• Apply a cluster claim.
  => cannot apply composed resource: existing object is not controlled by UID

What environment did it happen in?

UXP version: 1.11.3-up.1

• Cloud provider: AWS
• Kubectl: v1.25.4
• Kubernetes: v1.25.3
• uname of management cluster: Darwin Rocket.local 22.3.0 Darwin Kernel Version 22.3.0: Mon Jan 30 20:39:46 PST 2023; root:xnu-8792.81.3~2/RELEASE_ARM64_T6020 arm64

[negz]

It may be worth raising this issue against https://github.com/crossplane/crossplane/ - it's almost certainly not specific to UXP.

[jbw976]

edit: I don't know of a specific upstream issue that addresses this behavior - ignore this comment 😇

[turkenh]

• Install a platform-ref-aws package.
• Apply a cluster claim from examples/cluster-claim.yaml.
• Remove the platform-ref-aws package.
• Install another version of the platform-ref-aws package.

Am I understanding correctly that we are removing the platform-ref-aws package while there are instances of claims/composites introduced by that package? If this is the case, we may end up with an inconsistent state since we are deleting CRDs while there are live instances.

Let me try to explain:

• Install a platform-ref-aws package.

• A couple of XRDs and Compositions were created.
• Crossplane created corresponding Claim and Composite CRDs for XRDs, e.g. clusters.aws.platformref.upbound.io/v1alpha1 and xclusters.aws.platformref.upbound.io.
• Provider AWS and Provider Helm installed as dependencies.

• Apply a cluster claim from examples/cluster-claim.yaml.

• We created an instance of Cluster type as Claim.
• Crossplane created corresponding Composite instance, e.g. XCluster
• Crossplane composed other Composites, e.g. XNetwork, XEKS and XServices
• Crosspane composed corresponding managed resources of Provider AWS and Provider Helm.

• Remove the platform-ref-aws package.

• XRDs and Compositions deleted (garbage collected as the package owning them deleted)
• Claim and Composite CRDs got deleted while there were live instances, so all Claims and Composites were garbage collected by K8s.
• I am not sure if composed MRs got a proper deletion request.

@humoflife did you ensure that all MRs composed for your claim were properly cleaned up before installing the new version of the configuration package?

The error message existing object is not controlled by UID signals that there could be remaining composed resources that are owned by the previous composites, hence the new ones cannot take control of them.

[turkenh]

This is likely the upstream issue that addresses this scenario: crossplane/crossplane#3742

@jbw976 I am not sure that I could see the relation.
I am wondering if we are confusing the error message here with the one we are getting when a provider upgrade fails if a CRD is controlled by another ProviderRevision, e.g. is already controlled by ProviderRevision ?

[jbw976]

Yes that's totally possible @turkenh, sorry for the confusion! I'll update my comment above to reduce further confusion.

Remove the platform-ref-aws package.

It would be interesting to see the state of everything when this package is removed while instances of claims that it defines are still alive - I'm curious what gets saved by finalizers or being in use, what gets garbage collected etc.

[turkenh]

Closing as stale, feel free to reopen with enough details 🙏