chore(breaking)!: Switch to use annotation based resource tracking as the default method. Closes #13981

[terrytangyuan]

Closes #13981. This is a draft PR for now to prevent from getting merged accidentally. This breaking change is intended for v3.0 but we'd like to open up an early discussion.

cc @jannfis and other @argoproj/argocd-approvers

Checklist:

• Either (a) I've created an enhancement proposal and discussed it with the community, (b) this is a bug fix, or (c) this does not need to be in the release notes.
• The title of the PR states what changed and the related issues number (used for the release note).
• The title of the PR conforms to the Toolchain Guide
• I've included "Closes [ISSUE #]" or "Fixes [ISSUE #]" in the description to automatically close the associated issue.
• I've updated both the CLI and UI to expose my feature, or I plan to submit a second PR with them.
• Does this PR require documentation updates?
• I've updated documentation as required by this PR.
• I have signed off all my commits as required by DCO
• I have written unit and/or e2e tests for my change. PRs without these are unlikely to be merged.
• My build is green (troubleshooting builds).
• My new feature complies with the feature status guidelines.
• I have added a brief description of why this PR is necessary and/or what this PR solves.
• Optional. My organization is added to USERS.md.
• Optional. For bug fixes, I've indicated what older releases this fix should be cherry-picked into (this may or may not happen depending on risk/complexity).

[bunnyshell]

🔴 Preview Environment stopped on Bunnyshell

See: Environment Details | Pipeline Logs

Available commands (reply to this comment):

• 🔵 /bns:start to start the environment
• 🚀 /bns:deploy to redeploy the environment
• ❌ /bns:delete to remove the environment

[bunnyshell]

✅ Preview Environment created on Bunnyshell but will not be auto-deployed

See: Environment Details

Available commands (reply to this comment):

• 🚀 /bns:deploy to deploy the environment

[terrytangyuan]

@argoproj/argocd-approvers Are there concerns over this change for v3.0? If so, please comment here so that we can discuss earlier. Thank you!

[crenshaw-dev]

It'll mean some work for Intuit, since we're using a very old tracking method for some instances. But that's okay, this is a good change in default.

[terrytangyuan]

👋🏻 @ironashram You had a thumb down for this PR. Would you mind clarifying? We'd love to hear more inputs from the community.

[ironashram]

a few weeks ago I tried to switch to annotation tracking in one of my clusters and I had tons of issues and out of sync resources, which I was able to solve in a few hours re deploying some apps

so the thumbs down was more related to the heat of the moment and my personal dislike for annotations in general since they are kind of messy to understand by humans compared to labels (maybe I'm alone in this)

Anyway I get the reasons and I've been using annotation tracking in argo for a while now without other issues.

[terrytangyuan]

@ironashram Thanks!

[andrii-korotkov-verkada]

Please, rebase and fix the tests. I also think it may be better to support two methods in parallel as suggested on the issue.

[thesuperzapper]

We should seriously consider the implications if we change the default tracking method to annotations.

While I would love to see wider adoption of the annotation-based tracking, what we see as a bug in the label-based tracking (that other apps can create resources that match the label), is actually a feature for a lot of downstream apps.

Let me lay out a case for label tracking:

1. Argo CD is the center of most people's GitOps on Kubernetes.
2. People like to see all of the resources in their applications in Argo CD.
3. Many charts/apps generate resources (with controllers/jobs) that are not part of their manifests, that users still want to see.
4. It's very common for them to copy or set the label we just happened to use (app.kubernetes.io/instance) onto these resources.
5. That means these resources are visible in Argo CD (as extraneous resources, requiring pruning unless ignored)
6. It also means these resources will be removed on prune/delete, if they don't have an owner (Argo CD will never delete a resource with an owner)

Of course this is also the biggest drawback, because if we used the annotation approach, we could almost guarantee that if a resource has a valid tracking annotation, that ArgoCD created it at some point, and so if it no longer exists in the source, it probably does require pruning.

---

We want to make the migration to Argo CD 3.0 as smooth as possible for users, but doing an in-place migration from labels to only annotations is not safe for most users.

If we are dead-set on doing this, we should probably do a staged change:

1. Change the default to track with BOTH annotation and labels for a few releases (before 3.0).
   • As people re-sync their apps, they will get the new annotations.
2. Put a warning on apps containing resources that only have the label based tracking:
   • We want to consider these apps "out of sync" so they pick up the new tracking on the next sync.
   • We need to be careful not to prune resources unless they have the tracking annotation, as otherwise every resource would be "requiring pruning".
   • This will also help people find these "generated" resources, as they wont pick up valid tracking annotations on sync.
3. Fully remove label-based tracking in 3.0:
   • This lets us make improvements, without needing to consider that some people might still be using label-based tracking.
   • We should also include an example in our docs of how to generate a valid tracking annotation with a Helm template.