ArgoCD is unaware of XRC -> XR relationship

[natereid72]

What problem are you facing?

This is likely something that will need to be addressed in Argo CD, creating issue here to describe and track. Argo CD ustilizes Kubernetes owner reference to determine if an Argo CD application is synchronized. XRC being namespaced cannot be the owner of cluster scoped XR.

This issue can be linked to #2773

Overview of issue:

As it stands today, Argo CD accomplishes this by following a chain of owner references. Owner reference is one of two prescribed ways to define relationships between Kubernetes resources (The other being through label selection). Interestingly enough, the Deployment example above uses both. I say ‘interestingly’ because it conveys there is a reason to use one, the other, or both. Let’s take a closer look at owner references, the docs say this about it::

In Kubernetes, some objects are owners of other objects. For example, a ReplicaSet is the owner of a set of Pods. These owned objects are dependents of their owner.
…
Ownership is different from the labels and selectors mechanism that some resources also use.
…
Owner references help different parts of Kubernetes avoid interfering with objects they don’t control.

That last part, “avoid interfering with objects they don’t control” gets to the center of owner reference’s intended purpose. Specifically, it aids Kubernetes garbage collection.

Garbage collection is a collective term for the various mechanisms Kubernetes uses to clean up cluster resources.
…
Many objects in Kubernetes link to each other through owner references. Owner references tell the control plane which objects are dependent on others. Kubernetes uses owner references to give the control plane, and other API clients, the opportunity to clean up related resources before deleting an object. In most cases, Kubernetes manages owner references automatically.

Because owner reference was implemented for garbage collection to prevent unwanted deletion, there are some enforced rules governing where the relationship is permitted.

Cross-namespace owner references are disallowed by design. Namespaced dependents can specify cluster-scoped or namespaced owners. A namespaced owner must exist in the same namespace as the dependent. If it does not, the owner reference is treated as absent, and the dependent is subject to deletion once all owners are verified absent. Cluster-scoped dependents can only specify cluster-scoped owners.

The important take away here:

1. Owner reference conveys relationships between Kubernetes resources, for the purpose of garbage collection.
   Because its purpose is to protect unrelated resources during garbage collection, namepsaced resources cannot own cluster scoped resources or resources in other namespaces.
2. This means Argo CD is bound by the rules imposed on owner reference and therefore cannot derive a relationship with a resource that begins in a namespace and then creates resources at the cluster level or within another namespace.

If we create something that spans any of those lines, Argo CD reports ‘out of sync’ and believes those resources should be ‘pruned’ (i.e. deleted). There is a configuration option to ignore resources in this case, but that leaves you with a sort of ‘black hole’ effect. Take Persistent Volume Claim, a namespaced resource that can cause the creation of a Persistent Volume, a cluster scoped resource. Since these resources and controller pattern are part of core Kubernetes, Argo CD defaults to ignoring the PV created via the PVC.

In the case of Crossplane’s Composite Resource Claim, we create a namespaced resource that results in a cluster scoped resource called Composite Resource. From that single Composite Resource, we can have many additional Composite Resources, all of which go through a Composition, generating Managed Resources that create Provider Config Usages. If we do not configure Argo CD to ignore the Composite Resource (as is the case with PVC -> PV), it will think all of those extra resources are unrelated (namespace resource can’t own cluster resource) and want them pruned. On the other hand, configuring Argo CD to ignore Composite Resource results in a massive ‘black hole’ effect.

How could Crossplane help solve your problem?

I believe this will require collaboration with Argo CD project. There may be alignments within both projects possible.

[negz]

If we do not configure Argo CD to ignore the Composite Resource (as is the case with PVC -> PV), it will think all of those extra resources are unrelated (namespace resource can’t own cluster resource) and want them pruned. On the other hand, configuring Argo CD to ignore Composite Resource results in a massive ‘black hole’ effect.

Can you elaborate on this 'black hole' effect? I think that's the part I don't grok here.

Let me try play-back my understanding.

• We're talking about a case where folks are storing XRCs in git. The XRs and MRs are not in git. ArgoCD syncs the XRCs from git to Kubernetes, after which point Crossplane takes over and syncs the XRCs with XRs, MRs, etc.
• There may be a problem where ArgoCD thinks it (not Crossplane) created the XR directly because Crossplane is propagating ArgoCD specific labels from the XRC to the XR per XRC -> XR Label Propagation Confuses ArgoCD #2121. Let's pretend that's either not true or has been fixed for the purpose of this discussion.
• ArgoCD doesn't understand the relationship between an XRC and an XR, so as far as ArgoCD is concerned when it creates an XRC is synced as long as its spec matches what's in git, and the XR and MRs are just some random resources that exist in the cluster outside of ArgoCD management.

Doesn't this mostly mean that the key problem here is that the ArgoCD UI can't visualize the XR and MRs? They would still be created by Crossplane when the XRC was created, do their thing, then be deleted when the XRC was deleted, right?

[natereid72]

Can you elaborate on this 'black hole' effect? I think that's the part I don't grok here.

Doesn't this mostly mean that the key problem here is that the ArgoCD UI can't visualize the XR and MRs? They would still be created by Crossplane when the XRC was created, do their thing, then be deleted when the XRC was deleted, right?

This is a piece I'm still working on understanding actually. You are correct for sure on the Argo CD UI visualization part. I am still trying to work out, with what combination and if any, Argo CD will evaluate what 'should' be in the cluster via relationship and dry-run compare.

It certainly dies not matter when Crossplane is rectifying state based on its own machinery. So in that case, I agree that just having Argo CD track just the XRC, with the Crossplane controller(s) taking over from there is and should be the right combination. So that takes me back to the place where I first started in my 'this is the way this should be' opinion. But it still doesn't get us away from having to configure Argo CD to ignore each and every 'entry' XR. Perhaps this is an ok burden short of Argo CD adding some better/improved relationship correlation method.

[natereid72]

Ok, I've been continuing to go through the 'start with XRC' vs. 'start with XR' pros and cons. As @negz raised, what is the real impact of starting with XRC and telling Argo CD to ignore the XR and beyond, I think I am back to thinking XRC is fine.

At least from a use of GitOps basis. Everything beyond XRC in that case in the Argo CD UI is just Argo C making an educated guess. And I believe it is better represented in a different dashboard, that will show every thing. For example, if I look at Argo CD UI with XR through MR (starting with XR case), I see the resources associated with the Argo CD app, but not other XR/MR I have created manually by circumventing Argo CD and creating directly against Kubernetes API server.

That is not what I would want as a K8s SRE. I would want to see what ArgoCD thinks should be there from a 'came from Git' perspective (Starting with XRC satisfies this), as well as anything that is running that didn't. To accomplish this, I need a separate/better UI, making the Argo CD UI undesirable.

So that takes me from start with XRC to start with XR and back a total of three tines now. Third time is the charm :)!

I think we should still push for enhancement in Argo CD to understand Crossplane resource relationship. As it is now, there is still a lot of manual configuration required for Argo CD to ignore XRs. If it understood the relationships we wouldn't have to and it would work whether we started with XRC or XR.

Long story short, my opinion is to start with XRC, configure Argo CD to ignore the XRs, use Lens or some other UI to show everything running in cluster, use Argo and Upbound UI to drill into more focused config detail and status.

[natereid72]

I've updated my blog post to reflect this shared opinion. I struggled back and forth with it, but the more I consider it, the more I believe a GitOps platform should only concern itself with what is in the repo, not what an operator created downstream. So I think we benefit from Argo CD improving its platform, whether that be through more flexible relationship modeling, or simply a setting that says it should not even try to track something beyond the 1:1 mapping of resources in repo and cluster (I prefer the latter solution)

[natereid72]

So i think there are two separate issues. One is wrt Argo showing related resources past what it is track from the repo. Those resources are displayed but not factored into the apps sync status. So this is purely cosmetic in the case of XRC not showing XR or anything else.

Then there is the issue of #2121 which has a proposed fix. That fix would solve the primary issue with XRC and ArgoCD.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[natereid72]

this is resolved via argo cd