Notification subscriptions on AppProject resources not working when using apps-in-any-namespace feature

[ChristianCiach]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

When using the "Applications in any namespace" feature, notification subscriptions added to AppProject resources have no effect if the Application resource is in another namespace.

There are no errors in any argo-cd pod logs. The notification-controller just doesn't seem to honor the annotations on the AppProject if the Application is not in the argocd namespace.

Sanity checks:

• Moving the Application to the argocd namespace makes the notifications work.
• Adding the subscribe-annotation directly to the Application resource (instead of the AppProject) also works.

To Reproduce

Enable app-in-any-namespace-feature:

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cmd-params-cm
data:
  # https://argo-cd.readthedocs.io/en/stable/operator-manual/app-any-namespace/#reconfigure-argo-cd-to-allow-certain-namespaces
  application.namespaces: 'argocd, argocd-*'

Setup notification triggers and templates:

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-notifications-cm
data:
  context: |
    argocdUrl: https://argocd.mycompany.local

  service.mattermost: |
    apiURL: https://mattermost.mycompany.local
    token: $mattermost-argocd-egs-token

  # Optional 'oncePer' property ensure that notification is sent only once per specified field value
  # E.g. following is triggered once per sync revision
  # "oncePer" explanation: https://github.com/argoproj/notifications-engine/issues/71#issuecomment-1605022262
  trigger.on-deployed: |
    - when: app.status.operationState.phase in ['Succeeded'] and app.status.health.status == 'Healthy'
      oncePer: "[app.status?.operationState?.syncResult?.revision, app.status?.operationState?.syncResult?.revisions]"
      send: [app-sync-succeeded]


  template.app-sync-succeeded: |
    message: |
      Application {{.app.metadata.name}} is now running new version of deployments manifests.
    mattermost:
      attachments: |
        [
          {
            "title": "{{.app.metadata.name}}",
            "title_link": "{{.context.argocdUrl}}/applications/{{.app.metadata.name}}",
            "color": "#18be52",
            "fields": [
              {
                "title": "Sync Status",
                "value": "{{.app.status.sync.status}}",
                "short": true
              },
              {
                "title": "Repository",
                "value": "{{.app.spec.source.repoURL}}",
                "short": true
              },
              {
                "title": "Revision",
                "value": "{{.app.status.sync.revision}}",
                "short": true
              }
            ]
          }
        ]

Create an AppProject with notification subscriptions enabled:

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: egs-external
  namespace: argocd
  annotations:
    notifications.argoproj.io/subscribe.on-deployed.mattermost": 'kdqsisuh1b8jdf9zst41jghoqo'
spec:
  sourceNamespaces: ["argocd-egs-external"]
  # ...

Create an application that is part of the AppProject above but does not include its own notification annotations:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: netshoot
  namespace: argocd-egs-external
  finalizers:
    # Delete all managed resources when the app gets deleted
    - resources-finalizer.argocd.argoproj.io
spec:
  project: egs-external

  source:
    repoURL: https://git.mycompany.local/argocd/applications.git
    targetRevision: HEAD
    path: 'netshoot'

  destination:
    name: egs-external
    namespace: default

Then sync the netshoot application. No notifications will be send and the logs will show no errors or warnings.

Expected behavior

According to the documentation at https://argocd-notifications.readthedocs.io/en/stable/subscriptions/, I would expect the above example to "just work". And indeed it does just work when moving the Application to the argocd namespace.

More thoughts

I feel like this may have something to do with:

• fix(notifications): Allow notifications controller to notify on all namespaces #15702.

While this PR made subscribe annotations on Applications in any namespace work, it may have overlooked the case where the annotations are on the AppProject instead, which are always in the argocd system namespace.

Version

argocd: v2.10.9+c071af8
  BuildDate: 2024-04-30T18:33:15Z
  GitCommit: c071af808170bfc39cbdf6b9be4d0212dd66db0c
  GitTreeState: clean
  GoVersion: go1.22.2
  Compiler: gc
  Platform: linux/amd64
argocd-server: v2.10.7+b060053

Logs

No relevant logs :(