Objects are not removed when ArgoCD Application is deleted

[gberenice]

Hi!

I'm working on FSA+tf-controller POC now, and have noticed some unexpected behaviour. Will appreciate your help!

1. I have deployed ArgoCD Application infra with finalizers set, and Kustomization object was reconciled:

[
   {
      "group":"kustomize.toolkit.fluxcd.io",
      "health":{
         "message":"ReconciliationSucceeded - Applied revision: poc-1/46ca9138db9a958e9251f951f4168a0e21ef396b",
         "status":"Healthy"
      },
      "kind":"Kustomization",
      "name":"infra",
      "namespace":"infra",
      "status":"Synced",
      "version":"v1beta2"
   },
   {
      "group":"source.toolkit.fluxcd.io",
      "health":{
         "message":"Succeeded - stored artifact for revision 'poc-1/46ca9138db9a958e9251f951f4168a0e21ef396b'",
         "status":"Healthy"
      },
      "kind":"GitRepository",
      "name":"infra",
      "namespace":"infra",
      "status":"Synced",
      "version":"v1beta2"
   }
]

kubectl -n argocd get app infra -o jsonpath="{.metadata.finalizers}"
["resources-finalizer.argocd.argoproj.io"]

If I delete the Kustomization object, all related resources are removed. But destroying the Application didn't work out. In the controller logs I see:

level=info msg="Deleting resources" application=infra
level=info msg="Deleting application's resources with Foreground propagation policy" application=infra
level=info msg="Successfully deleted 0 resources" application=infra

So only the Application was deleted.

2. Also, if you check the Application resources list, there is a GitRepository what is not declared anywhere - there is no such file in the repo that ArgoCD is looking at. In the configuration I reference to another GitRepository object with is created by another ArgoCD Application, but they both are equal. Is is possible that FSA creates this object?

kubectl get gitrepositories -A                                      
NAMESPACE   NAME                URL                                                  AGE   READY   STATUS
infra       bootstrap           https://github.com/masterpointio/tf-controller-poc   20h   True    stored artifact for revision 'poc-1/46ca9138db9a958e9251f951f4168a0e21ef396b'
infra       infra               https://github.com/masterpointio/tf-controller-poc   33m   True    stored artifact for revision 'poc-1/46ca9138db9a958e9251f951f4168a0e21ef396b'

Thanks in advance!

[kingdonb]

Does the ArgoCD Application deletion have any effect at all? I guess it wouldn't, we only use the ArgoCD application to provide a handle and visualize what's going on in the Flux resources under the hood. There is no management or garbage collection from the ArgoCD side, only visualization. ArgoCD doesn't apply anything (and we don't need to give it applier permissions, as a nice side-effect of that.)

Is that a correct behavior? I'm not sure, but I think it's consistent with the way we've been treating ArgoCD applications in the Flamingo project.

[gberenice]

No, it doesn't.
I might understand that wrong, but my expectation was that ArgoCD application role is a bit more influential on resources lifecycle, since we can:

• create the resources by applying an Application
• update the resources by updating its spec.source.helm.values.

Considering that, it might be reasonable to also be able to delete the resources to complete the workflow. Otherwise, what is the recommended way to do that?

[kingdonb]

When a Flux resource is managed by ArgoCD in FSA, the relationship is inverted from the general expectation. ArgoCD isn't managing Flux like it manages its own resources, it's just a pass-through relationship for visualizing. This may not be the consistent experience you were expecting, and these are growing pains I think we expected to see when people started using this in earnest.

The recommended way to delete the resources is for Flux to manage itself (eg. Flux bootstrap) so that all Flux resources are under Flux's watchful eye, then enable garbage collection in the Flux resources that you want to delete something, and delete the resource from Git.

If you delete a resource in the ArgoCD UI when using FSA or Flamingo, your expectation should be that Flux will put it back, unless the definition doesn't exist in Git anymore, (or if it was the root Flux resource, or some critical RBAC permission that you deleted, so that Flux can't put it back.) There are other integrated Flux experiences that also have this difference in design alignment. That's how I'm perceiving it anyway, "there are two camps" as a concept.

For example if you use Azure AKS + Microsoft.Flux, the expectation is that you will create Flux resources through the Azure API, and this is like what you're expecting I think for Argo. I think this is contrary to the Flux-centric ideas that are mainly presented throughout the Flux docs, which all expect that you'll be using only Flux to manage GitOps resources. Flux's docs all anticipate that you start with Bootstrap, and that Flux is the top-level manager of all the resources on the cluster. The ideal for deployment according to the docs should be "git as a single source of truth" and it does not need for Flux to be the only manager. This might be somewhat different than the experience on the ground, now that I write this out in reflection.

You shouldn't be able to turn off GitOps by clicking around in the interface; though you can suspend Flux resources, and when not suspended the reconcilers should replace any resources that got deleted (unless the bootstrap resource itself was deleted.) But the ArgoCD Applications are not owner of Flux resources, they did not create them, so they won't delete them either. That's the state of things right now anyway, whether or not this makes sense may remain an open question.

I think it might be worthwhile to have a meeting if you are evaluating FSA for some production purpose. I have many ideas about the expectation but I do not know how users actually approach FSA in the wild, (and it sounds like you are a live one!)

[gberenice]

Hey @kingdonb, apologies for the delayed response. Thanks a lot for the expanded explanation. We'll discuss it with the team internally and after might be back for a meeting.

[kingdonb]

I appreciate that!

While my response may read like "this is working as designed" I prefer to keep this issue open, until we have collected some use cases and created guidance from real-world about how Argo users that adopt FSA should expect the deployment to be.

My intuition is "Flux manages everything" because in my ideal design, something is managing everything (and I do not think the design of FSA permits at all for Argo to manage Flux... argo is used for the UI, while Flux appliers and sources handle the heavy lifting, and that is the design of Flamingo, as I understand it.)

But as we've seen the dominant pattern across industry leading platforms provided by the integrated service platforms all seem to admit that while "single source of truth" is the documented paradigm, the notion of a centralized "single" source is commonly rejected.

"There can be multiple sources of truth" is really somewhat closer to the reality on the ground in most places, and having multiple independent controllers that interact through a narrow service interface might even be part of certain HA designs.

[kingdonb]

To reiterate, I can imagine some users would expect

• that Argo manages Argo, and Flux manages Flux
• or, that Flux manages itself and Argo, while Argo handles only visualizing

We should provide a clear diagram and example manifests that demonstrate either use case, and explain the difference of operating in either mode, and any implications of that decision that could be important.

And if it's the case that we see users have expectation that Argo resource handles have pass-through effects on the Flux resources (like garbage collection) we may even want to consider that as a possible enhancement request.

(Please do not let me leave you with the impression that, because the design of FSA is a certain way today, that means it is right! This Flamingo project is very young, and it could have lots more growing left to do.)

[reedjosh]

And if it's the case that we see users have expectation that Argo resource handles have pass-through effects on the Flux resources (like garbage collection) we may even want to consider that as a possible enhancement request.

Yes please! : )

[chanwit]

We decided to scope the functionalities of Flamigo to be only the "visualization" layer, so it won't touch the underlying Flux objects.

In recent releases of the flamingo CLI, we introduced the flamingo generate-app command which generates Argo applications from Flux objects. So deletion of Argo / Flamingo applications will not affect the Flux objects they displayed.

The behaviour could be revisited in the future.