ApplicationSet: Add option to preserve manual parameter overrides

[FrittenToni]

Summary

ApplicationSets automatically override manual Application parameter overrides, e.g. performed with argocd client (argocd app set command). The only option to avoid it seems to be to start the applicationset controller with "create-only" policy but this is not always what one wants. Especially in combination with a pull request generator since the pull request apps will never be deleted again once a pull request has been closed.

Motivation

I agree that manual parameter overrides should never be used in production but there is a dev use case where we need them:

1. Inside our pull request pipeline we're creating a temporary staging docker repository on the fly and we push our PR docker image into it
2. The application generated by PR generator should consume images from that prev. created staging repository
3. The name of the staging repository including the credentials to access it are not predictable and therefore we use argocd client to override these parameters

Inside our main pipeline basically the same mechanism happens. We create a staging repository and push our docker image into it. Then, we want to do artifact validation by running a bunch of tests and only if they are successful we promote our docker image from staging into our final release repository.

Proposal

How do you think this should be implemented?

There should be a switch on applicationset level to express that manual parameter overrides should be preserved. If that switch is true and applicationset controller updates an Application object it should merge the parameter overrides instead of just replacing them with whatever is inside the repository.

[jdomag]

I would love to see this feature as well. My use case:
When we build a new image we don't do commit/push to git repo containing the helm chart - we simply do
argocd app set APP_ID --helm-set image.tag=new_image
The same happens when image updater is used (imperative write-back method)

[LeartBeqiraj]

Any update on this?

[crenshaw-dev]

I think it would be a nontrivial change in ApplicationSet. We'd have to do a diff, apply ignored-field rules, and apply the diff as a patch. Not terrible, but also not trivial.

I'm curious if we can just solve the problem one level higher, using an Application, so the already feature-rich Application controller can handle it.

What if you 1) converted your ApplicationSet to a Helm chart, 2) deployed the ApplicationSet with a new, top-level Application, 3) set the parameter override on the new top-level Application and let it propagate down to the target Application.

The main trick would be applying the parameter override to only the target Application. I think that could probably be done with a merge generator in the ApplicationSet.

[raelga]

This feature is also interesting for our team, as we usually play with the branches to test changes in the development / staging Applications, before pushing them to production. We also usually manually stop the automatic syncing / self healing for non-production Applications while testing.

[arturkasperek]

Hi,
our team needs that cause we wanna give a possibility to rollback

[crenshaw-dev]

@raelga would a "pause" label or annotation solve your use case?

[raelga]

Is not as straight forward as the SyncPolicy update, but it can work.

For our use case, we ended reproducing the ApplicationSet outcome using kustomize. So now we have a base with the applications manifests and then several overlays to create all the applications environments. This generates the same combination you can generated with a matrix ApplicationSet from a list of Components (apps from the same workload) and a list of Environments (clusters and repos).

The benefit of using kustomize is that it behaves as an App of Apps with all the benefits of using an Application instead of an ApplicationSet: it can be configured to manual synchronisation, makes really easy to detect apps Out of Sync (not pointing to HEAD or without the automated healing),... etc.

This is how the kustomized App of Apps deployment looks, not as clean as an ApplicationSet, but with all the functionalities of a regular Application.

❯ tree
.
├── application.yaml
├── base
│   ├── applications
│   │   ├── example-component.yaml
│   │   ├── example-component-2.yaml
│   │   ├── example-component-3.yaml
│   │   ├── example-component-4.yaml
│   │   └── example-component-5.yaml
│   └── kustomization.yaml
├── kustomization.yaml
└── overlays
    ├── dev
    │   ├── appproject.yaml
    │   └── kustomization.yaml
    ├── pro
    │   ├── applications
    │   │   ├── pro-only-example-component.yaml
    │   │   └── pro-only-example-component-1.yaml
    │   ├── appproject.yaml
    │   └── kustomization.yaml
    └── stg
        ├── applications
        │   └── stg-only-example-component.yaml
        ├── appproject.yaml
        └── kustomization.yaml

kustomization.yaml

resources:
  - overlays/dev
  - overlays/stg
  - overlays/pro

overlays/pro/kustomization.yaml

bases:
  - ../../base

resources:
  - appproject.yaml
  - applications/pro-only-example-component.yaml
  - applications/pro-only-example-component-1.yaml

transformers:
  - |-
    apiVersion: builtin
    kind: PrefixSuffixTransformer
    metadata:
      name: prefix
    prefix: workload-pro-
    fieldSpecs:
      - kind: Application
        path: metadata/name
  - |-
    apiVersion: builtin
    kind: PrefixSuffixTransformer
    metadata:
      name: source-path-prefix
    prefix: manifests/workload/pro
    fieldSpecs:
      - kind: Application
        path: spec/source/path
patches:
  - patch: |-
      - op: replace
        path: /spec/project
        value: workload-pro
      - op: replace
        path: /spec/destination
        value:
          server: https://api.pro-cluster-fqdn:6443
          namespace: workload-pro
    target:
      kind: Application

And the base are several Application manifests:

/bases/applications/example-component.yaml

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  annotations:
    argocd.argoproj.io/manifest-generate-paths: .
  name: example-component
spec:
  project: kustomize-overlay
  source:
    repoURL: https://github.com/org/workload
    targetRevision: HEAD
    path: example-component     # base path is added as prefix by kustomize with a transfomer
    directory:
      recurse: true
  destination:
    server: kustomize-overlay
    namespace: kustomize-overlay
  syncPolicy:
    automated:
      selfHeal: true

[tenczardavid]

Hi Everyone,

The main problem we have currently with ApplicationSet is the fact that it removes some built-in features of ArgoCD.

One of them is to disable automated sync on Application level. This also has the side-effect that ArgoCD rollbacks can't function in this setup. This causes a lot of confusion for our developers when we would like to on-board them.

As ApplicationSet became a core feature of ArgoCD, I think we should put some effort into trying to integrate it properly into its ecosystem. Was there any planning done for this feature already? How we could pitch in and help to move forward with this issue?

My main concern that in the documentation there is a sentence To some extent this is by design, was this reconsidered or is this still the case today?

https://argo-cd.readthedocs.io/en/stable/operator-manual/applicationset/Controlling-Resource-Modification/#limitation-no-support-for-manual-edits-to-individual-applications

[crenshaw-dev]

ArgoCD rollbacks can't function in this setup

I think we should probably create another issue to track the no-rollbacks problem (if an issue doesn't already exist for that). I'm not sure the solution will be the same as the solution to the parameter preservation issue.

@tenczardavid I'm open to the idea of trying to implement the fix as proposed - basically create a boolean field in ApplicationSet indicating that spec.source fields should be merged rather than overwritten. Precisely which fields should be merged and how they should be merged I think remains an open and important question.

Personally I'd like to see the ApplicationSet spec become fully templateable so that people can use their ApplicationSet source of truth (hopefully git or something else that's stable and has a history mechanism) to do things like disable auto-sync. That would be the more "gitops way" to do things. But I understand that one thing people like about Argo CD is its pragmatism. People like the ability to make a quick change in the UI before going back and making the change in Git. I'm open to expanding that pragmatism to ApplicationSets.

[bakito]

@raelga would a "pause" label or annotation solve your use case?

We are using the Image Updater https://argocd-image-updater.readthedocs.io/en/stable/ and would like to set annotations on certain applications created by the applicationset.
Unless we set the policy to "create-only" the pause annotations/labels would help in our case.

[takumakume]

My team is having trouble because we are using the Pull Request Generator + Image Updater.

The create-only policy alleviates this problem. However, it is difficult to use because it is not deleted when the pull request is closed.

I think the root solution is to keep overwriting or 3-way-merge.

However, this problem has not been solved for a long time because it requires a major modification.

I propose adding a create-delete policy as a way to alleviate this problem quickly.

I think this would be useful for applications like the Pull request Generator, which often have a short lifecycle.

[nhuray]

From my perspective that feature would be useful in certain cases but it probably bring more complexity to the controller code because of the diff computation to make.

One thing I would prefer it's to be able to define the policy (e.g create-only) at ApplicationSet level and not globally at controller level.

This would allow to define different strategy per-environment, so for instance:

• a Preview Environment will use an Application Set with a policy sync in order to create and destroy applications for each Pull Requests (using the Pull Request Generator).
• a Production environment will use an Application Set with a policy create-only to ensure applications are not destroyed by error.

There's an issue open here for that capability: #11073

[lacarvalho91]

not entirely sure if this is the same ask but my use-case is I have a custom config management plugin that will patch the application resource with an annotation that is used by another system. This doesn't seem to work ATM because the application set controller removes the annotation straight away. My understanding is that annotations are commonly used to write temporary state so I'd think the application set should be able to handle this (probably via some ignored fields configuration?)

[lacarvalho91]

this is the same use-case as Argo CD notifications and was solved for that here

can we do the same but from configuration?

[lacarvalho91]

some options i've seen suggested as workarounds are:

• use the create only policy so the app set controller will only create apps
  • this isn't usable for me as i want to be able to make changes to the application set definitions in git and have them reflected on the applications (i have an "app of app sets")
• use a Helm chart to produce the applications
  • i believe this would have the same problem? but even if not, i am leveraging the cluster generator from the application set so i would lose that

[lacarvalho91]

I've raised a PR here that will allow you to configure annotations to be preserved. But only annotations, so not sure if it fully addresses this issue.

[lacarvalho91]

i created a separate issue for preserving annotations as after rereading this I don't believe preserving annotations would be enough to solve the problem described on this issue.

[hbrewster-splunk]

@jessesuen Adding our two cents here:

We'd like to request functionality to 'ignore differences' on all child application parameters. Most notably - commit SHA (string), plugin (object).

In the use case, we have a set of applications across many clusters for each microservice. Pipelines update plugin parameters/git sha on a per application basis

As of now we create applications individually and manage them with a parent app in argocd. There, we use the ignoreDifferences functionality to ignore the updates to the app fields.

would be good to have the same functionality offered with app sets.

[vl-kp]

Thanks @lacarvalho91 for the PR, but unfortunately this feature is not working for me #14027

[crenshaw-dev]

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
spec:
  ignoreDifferences:
  - jsonPointers:
    - /spec/source/helm/parameters
  - jqExpressions:
    - .spec.source.helm.parameters

Is something like that sufficient to resolve the given use cases?

[arturkasperek]

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
spec:
  ignoreDifferences:
  - jsonPointers:
    - /spec/source/helm/parameters
  - jqExpressions:
    - .spec.source.helm.parameters

Is something like that sufficient to resolve the given use cases?

Yes

[crenshaw-dev]

I've built #14743 as a proof of concept. If someone has time to pick up that PR and run with it, I'd really appreciate it. I've added comments to the top describing what work I think remains.

[crenshaw-dev]

I read back through the thread and added unit tests to cover everyone's use cases (with links back to the relevant comments). Please take a look and let me know if we need to add a unit test for another use case: https://github.com/argoproj/argo-cd/pull/14743/files#diff-88fec9c8d881ff513bb04ee8842efcc4f0db0ec0ce631fec3f730633a91ebea6R5694

Looks like the normalizer is behaving well for the current use cases. I'm concerned it'll get trickier for advanced use cases involving ignoring certain list items.

[crenshaw-dev]

How do we feel about the field name ignoreApplicationDifferences?

I'm trying to avoid confusion in the spec where folks indent the field a couple levels too far. Hopefully the slightly different names will avoid confusion.

kind: ApplicationSet
spec:
  template:
    spec:
      ignoreDifferences: # for resources in the app
  ignoreApplicationDifferences: # for applications in the appset

[oisin88]

+1 on this, this would be an awesome feature

[crenshaw-dev]

I expect to get the PR into 2.9, I just have to fix a test failure. :-P If anyone wants to run #14743 internally, I'd appreciate the validation/feedback!