2.6 RC1: multi source application needs a hard refresh to deploy changes

[marcportabellaclotet-mt]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

When creating a multi source application, argocd needs a hard refresh to deploy changes, when changes are applied in the sources.

To Reproduce
Create an application with helm source and values from git repo.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: argo-rollouts
  namespace: argocd
  labels:
    type: deployment
    application: argo-rollouts
spec:
  project: default
  sources:
    - repoURL: 'https://argoproj.github.io/argo-helm'
      targetRevision: 2.21.1
      chart: argo-rollouts
      helm:
        valueFiles:
        - $values/argocd/applications/values/argo-rollouts/values-common.yaml
    - repoURL: https://github.com/xxx/yyy.git
      targetRevision: HEAD
      ref: values
  destination:
    server: 'https://kubernetes.default.svc'
    namespace: argo-rollouts
  syncPolicy:
    syncOptions:
      - CreateNamespace=true
    automated: {}

Once the app is deployed, upgrading the values in the git repo, does not trigger any change in argocd.
In order to make argocd aware of this change, a hard-refresh is needed.

Expected behavior

Argocd controller should note that the repo is out of sync, without the need of a hard-refresh.

[lukas-unity]

It also returns this log line #11770 (now fixed)

[crenshaw-dev]

Just confirmed that order doesn't matter. Even if the values-file source is first, the application controller doesn't catch the new commit.

[endersonmaia]

@marcportabellaclotet-mt what "hard-refresh" means ? you had to run argo CLI or use the web UI

I ask this 'cause in my case, I don't use the CLI nor the web UI, so how to hard-refresh ?

[endersonmaia]

In my case, my Application stays in SyncStatus: OutOfSync and HealthStatus: MIssing and I'm defining the syncPolicy as:

...
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
          - CreateNamespace=true

Before, when I was using a configManagementPlugins workaround from this commen, I could make the Application sync and so on after adding the syncPolicy section.

[crenshaw-dev]

@endersonmaia can you provide your full Application spec?

I think @marcportabellaclotet-mt is probably triggering a hard refresh in the UI or CLI. So if you're running Argo CD headless, I think you're just stuck if the application controller doesn't see the new commit.

[endersonmaia]

So if you're running Argo CD headless, I think you're just stuck if the application controller doesn't see the new commit.

It doesn't work even with the "original" commit, since I'm not even sending a new commit to the values repository. (should I open a new issue?)

[crenshaw-dev]

Yep! Sounds like a Different Thing. 🙂

[endersonmaia]

@endersonmaia can you provide your full Application spec?

This was generated from an ApplicationSet with git generator.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-app
  namespace: argocd
spec:
  destination:
    namespace: my-app
    server: https://kubernetes.default.svc
  project: default
  sources:
  - chart: my-chart
    helm:
      releaseName: my-app
      valueFiles:
      - $values/myorg-aws-dev.values.yaml
      - $values/network.values.yaml
      - $values/my-dapp/network.yaml
      values: |
        ingress:
          subDomain: network.dev.myorg.io
    repoURL: https://myorg.github.io/helm-charts
    targetRevision: 0.4.1
  - path: ./echo-python/
    ref: values
    repoURL: https://github.com/myorg-private-repo/deployments
    targetRevision: poc/argocd-applicationset
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
    - CreateNamespace=true

[crenshaw-dev]

@endersonmaia any chance you can put together an app to reproduce the bug which uses publicly-available sources?

[marcportabellaclotet-mt]

I am using hard-refresh from UI. It also can be done via cli.

[messiahUA]

I'm facing the same problem with a simple app:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: ingress-nginx
spec:
  destination:
    namespace: ingress-nginx
    server: https://kubernetes.default.svc
  project: default
  sources:
  - chart: ingress-nginx
    helm:
      valueFiles:
      - $values/values/ingress-nginx/values.yaml
      - $values/values/ingress-nginx/values-main.yaml
    repoURL: https://kubernetes.github.io/ingress-nginx
    targetRevision: 4.4.0
  - ref: values
    repoURL: https://github.com/messiahUA/argocd
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
    - CreateNamespace=true

[naimo84]

I can second that 😉
Hard refresh works 👍

I can reproduce it with publicly-available sources like from @messiahUA, bt also with my companies apps ;)

Cheers

[logamanig]

I'm also facing this issue

[crenshaw-dev]

Thanks for the confirmations! We'll get this sorted before 2.6 GA.

[yebolenko]

any progress on this issue?
when 2.6 official release will be available?

[crenshaw-dev]

No progress yet.

2.6 GA is currently anticipated Feb. 6.

[yebolenko]

Michael, thank you for this update.
Do I understand correctly, that this fix will be included as a part of 2.6?

[crenshaw-dev]

@yebolenko yep! I consider this a blocking bug for the multi-source feature.

[crenshaw-dev]

Draft PR is up for the fix. This should work for non-webhook refreshes.

Unfortunately, I think webhooks won't be fixed by 2.6 GA. Updates from referenced repos won't go out of sync until the next reconciliation cycle.

[charandas]

Thanks @crenshaw-dev. Do you mean to say that webhooks are broken for multi source applications or across the board in 2.6 GA? I can live with it if it's just multi source ones.

[crenshaw-dev]

Not broken across the board, just for multi-source apps. And I believe the webhook will only be missed if it's for a referenced source. Top-level sources should be fine.

[zenitraM]

@crenshaw-dev I believe I've hit the same issue @endersonmaia did, which does not seem to be fixed by #12217 (or at least it was not fixed by the commit I built earlier today from it). I've opened #12301 about it with a reproducer.

[lsoica]

@crenshaw-dev I'm on v2.6.1+3f143c9 but this issue is still reproducible. Updating the values.yaml in GIT does not trigger a resync.

[crenshaw-dev]

@lsoica is this a case of a missed webhook, or does the 3min reconciliation also miss the change?

[lsoica]

@crenshaw-dev I think my comment was before this merge #12576

With that merge included, it does not reproduce anymore on my current use cases.

[nickyfoster]

I'm experiencing this issue on v2.7.2+cbee7e6.dirty.

Here is the Application manifest:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: multi-source-test
  namespace: argocd
spec:
  project: default
  destination:
    server: https://kubernetes.default.svc
    namespace: multi-source-test
  sources:
    - repoURL: ******.dkr.ecr.us-west-1.amazonaws.com
      chart: my-chart
      targetRevision: 1.0.0
      helm:
        valueFiles:
          - $values/configs/values.yaml
    - repoURL: git@github.com:private/gitops-configs.git
      targetRevision: main
      ref: values
  syncPolicy:
    syncOptions:
      - CreateNamespace=true
    automated:
      prune: true
      selfHeal: true
      allowEmpty: false