The "on-deployed" message is sent as a false positive.

[reverson-farfetch]

Description of the issue

We use an internal Gitlab server as a repository manager. We have a common behaviour that sometimes the Gilab has a short outage due to bunches of requests. Then when it happens we have a curious behaviour with our argocd notifications.
So, the outage is too short (most time less than one minute) the curious behaviour that I mentioned before happen when argocd repo server recover the connections with Gitlab. After the connections are recovered, the notification sends a deployed message for an application with Unkown status (it is because of the last status of the application) and it does not happen all the time when the outage happens. Sometimes the outage happens and this kind of false-positive doesn't happen.

The trigger that we are using is the default one.

trigger.on-deployed: |
    - description: Application is synced and healthy. Triggered once per commit.
      oncePer: app.status.sync.revision
      send:
      - app-deployed
      when: app.status.operationState.phase in ['Succeeded'] and app.status.health.status == 'Healthy'

Tshoot

We made a debugging to try to understand what can be happening. We didn't find a clear cause, but the thing that really get our attention was that the notification is only triggered when somehow in the annotation (notified.notifications.argoproj.io) about notification in the application appears a nil value.

Application annotation (notified.notifications.argoproj.io) values:

{
    "1c1ac356d87a121b49633353119bba0155335fad:on-deployed:[0].FtDfokP5gdvdikpZABRYYnjWhA4:slack:argocd-test-alarms": 1645445901,
    "1c1ac356d87a121b49633353119bba0155335fad:on-deployed:[0].y7b5sbwa2Q329JYH755peeq-fBs:slack:argocd-test-alarms": 1645037020,
    "4241b6d5fa83e81aad482546b8f4c7afd891bb90:on-deployed:[0].y7b5sbwa2Q329JYH755peeq-fBs:slack:argocd-test-alarms": 1645033566,
    "5c5b91bc42441243b917a1717781ea55836ef27c:on-deployed:[0].y7b5sbwa2Q329JYH755peeq-fBs:slack:argocd-test-alarms": 1645019950,
    "71e2d8c1d38ff5342e4e78febe040b8e403e5487:on-deployed:[0].y7b5sbwa2Q329JYH755peeq-fBs:slack:argocd-test-alarms": 1645609690,
    "7f37e82d84f1372c355cedb9fa78c6adfa2541d4:on-deployed:[0].y7b5sbwa2Q329JYH755peeq-fBs:slack:argocd-test-alarms": 1645026667,
    "906710221878765ada7e406129d3cbf47e6ef8ef:on-deployed:[0].y7b5sbwa2Q329JYH755peeq-fBs:slack:argocd-test-alarms": 1645036201,
    "\u003cnil\u003e:on-deployed:[0].y7b5sbwa2Q329JYH755peeq-fBs:slack:argocd-test-alarms": 1645021721,
    "a881f067b2f16a6f0c1f9afc213528fe861c6979:on-deployed:[0].FtDfokP5gdvdikpZABRYYnjWhA4:slack:argocd-test-alarms": 1645447151,
    "a881f067b2f16a6f0c1f9afc213528fe861c6979:on-deployed:[0].y7b5sbwa2Q329JYH755peeq-fBs:slack:argocd-test-alarms": 1645466841,
    "b59235bc2acd7f81a72d1b01cb52bb8c7363300e:on-deployed:[0].y7b5sbwa2Q329JYH755peeq-fBs:slack:argocd-test-alarms": 1645469603,
    "on-sync-status-unknown:[0].6SzWb05EK-0v90hwjyytTbN7S6A:slack:argocd-test-alarms": 1645626835
}

There are many notifications because this is

Notification logs:

time="2022-02-21T18:07:21Z" level=info msg="Trigger on-deployed result: [{[0].y7b5sbwa2Q329JYH755peeq-fBs a881f067b2f16a6f0c1f9afc213528fe861c6979 [app-deployed] true}]" resource=argocd-system/commerce-guest-guestuser-argocd
time="2022-02-21T18:07:21Z" level=info msg="Sending notification about condition 'on-deployed.[0].y7b5sbwa2Q329JYH755peeq-fBs' to '{slack argocd-test-alarms}'" resource=argocd-system/commerce-guest-guestuser-argocd
time="2022-02-21T18:07:22Z" level=info msg="Trigger on-deployed result: [{[0].y7b5sbwa2Q329JYH755peeq-fBs a881f067b2f16a6f0c1f9afc213528fe861c6979 [app-deployed] true}]" resource=argocd-system/commerce-guest-guestuser-argocd

We really don't know how this value is generated. We tried to reproduce it in our dev environment cutting the connection to Gitlab manually and recreating the connection again but we couldn't reproduce that, only once, as is in the above examples, but we couldn't trace it to reproduce again.

So, I'm here to ask for help. If someone had some issue like that or can have some idea what it can be.
We had tried to do many scenarios to reproduce it but until now we have failed.

We are using the argocd-notification v1.2.1 version.

Please, if someone can help us we'll really appreciate it.

[Rohit-Tanwar]

Is this your issue resolved?

[reverson-farfetch]

Not yet, we asked the customer of this feature in our cluster to ignore this kind of message.
And now we are moving to a new argocd version(2.4.2) and we are keeping an eye on that.