Applicationset not refreshed after webhook event using matrix generator when using matrix generator references

[ejhayes]

@KojoRising @crenshaw-dev @sboschman

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

It appears that the changes from #8998 are not compatible with the work done in #9080. This has been confirmed by the author of #8998

@ejhayes , can you please open a new issue for this? This PR was merged before the implementation of the new feat of #9080. I had a quick glance at the code, and you are correct in your assumption that the webhook logic has not been updated to include the new #9080 feature.

To Reproduce

For example if I create an application set like this and setup a webhook for pull requests they will not be refreshed when pull request events happen.

---
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: root-appbundle-test
  namespace: argocd
spec:
  generators:
  - matrix:
      generators:
        - scmProvider:
            cloneProtocol: https
            github: 
              organization: SOME_ORG
              tokenRef:
                secretName: github-token
                key: token
            filters:
              - repositoryMatch: ^argocd-test
        - pullRequest:
            github:
              owner: '{{ organization }}'
              repo: '{{ repository }}'
              labels:
              - preview
              tokenRef:
                secretName: github-token
                key: token
  template:
...

Expected behavior

However if I do not attempt to use {{organization}} and {{repository}} in the pull request generator then everything works as expected:

---
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: root-appbundle-test
  namespace: argocd
spec:
  generators:
  - matrix:
      generators:
        - scmProvider:
            cloneProtocol: https
            github: 
              organization: SOME_ORG
              tokenRef:
                secretName: github-token
                key: token
            filters:
              - repositoryMatch: ^argocd-test
        - pullRequest:
            github:
              owner: 'SOME_ORG'
              repo: 'SOME_REPO'
              labels:
              - preview
              tokenRef:
                secretName: github-token
                key: token
  template:
...

I suspect there may be an issue with using the interpolated values that is causing the webhook event to be discarded.

[crenshaw-dev]

Are there any interesting logs in the applicationset controller when it handles the webhook event?

[ejhayes]

No logs at all @crenshaw-dev -- just appears to be skipped altogether

Here's a recent example of when I removed a label.

Github webhook delivery looks to be returning a 200 so I know the webhook was received:

Request URL: https://***/api/webhook
Request method: POST
Accept: */*
content-type: application/json
User-Agent: GitHub-Hookshot/8e6ed5d
X-GitHub-Delivery: ***
X-GitHub-Event: pull_request
X-GitHub-Hook-ID: 367597581
X-GitHub-Hook-Installation-Target-ID: ***
X-GitHub-Hook-Installation-Target-Type: organization

And in the application set controller logs I see that the application does eventually remove the application but via polling not webhook.

time="2022-07-11T15:33:36Z" level=info msg="Alloc=21385 TotalAlloc=8667431 Sys=51025 NumGC=1871 Goroutines=92"                                                                                                                             │
time="2022-07-11T15:36:15Z" level=info msg="generated 0 applications" generator="{<nil> <nil> <nil> <nil> <nil> <nil> 0xc0002fa420 <nil>}"                                                                                                 │
time="2022-07-11T15:36:15Z" level=debug msg="apps from generator: []" generator="{<nil> <nil> <nil> <nil> <nil> <nil> 0xc0002fa420 <nil>}"                                                                                                 │
time="2022-07-11T15:36:15Z" level=info msg="Deleted application" app=argocd-test-develop-pr-2 appSet=root-appbundle-as                                                                                                                     │
time="2022-07-11T15:36:15Z" level=info msg="end reconcile" requeueAfter=0s                                                                                                                                                                 │
time="2022-07-11T15:36:17Z" level=info msg="generated 0 applications" generator="{<nil> <nil> <nil> <nil> <nil> <nil> 0xc000673760 <nil>}"                                                                                                 │
time="2022-07-11T15:36:17Z" level=debug msg="apps from generator: []" generator="{<nil> <nil> <nil> <nil> <nil> <nil> 0xc000673760 <nil>}"                                                                                                 │
time="2022-07-11T15:36:17Z" level=info msg="Deleted application" app=argocd-test-develop-pr-2 appSet=root-appbundle-as                                                                                                                     │
time="2022-07-11T15:36:17Z" level=info msg="end reconcile" requeueAfter=0s                                                                                                                                                                 │
time="2022-07-11T15:36:18Z" level=info msg="generated 0 applications" generator="{<nil> <nil> <nil> <nil> <nil> <nil> 0xc00041bce0 <nil>}"                                                                                                 │
time="2022-07-11T15:36:18Z" level=debug msg="apps from generator: []" generator="{<nil> <nil> <nil> <nil> <nil> <nil> 0xc00041bce0 <nil>}"                                                                                                 │
time="2022-07-11T15:36:18Z" level=info msg="Deleted application" app=argocd-test-develop-pr-2 appSet=root-appbundle-as                                                                                                                     │
time="2022-07-11T15:36:18Z" level=info msg="end reconcile" requeueAfter=0s                                                                                                                                                                 │
time="2022-07-11T15:36:20Z" level=info msg="generated 0 applications" generator="{<nil> <nil> <nil> <nil> <nil> <nil> 0xc001248c60 <nil>}"                                                                                                 │
time="2022-07-11T15:36:20Z" level=debug msg="apps from generator: []" generator="{<nil> <nil> <nil> <nil> <nil> <nil> 0xc001248c60 <nil>}"                                                                                                 │
time="2022-07-11T15:36:20Z" level=info msg="Deleted application" app=argocd-test-develop-pr-2 appSet=root-appbundle-as                                                                                                                     │
time="2022-07-11T15:36:20Z" level=info msg="end reconcile" requeueAfter=0s                                                                                                                                                                 │
time="2022-07-11T15:36:21Z" level=info msg="generated 0 applications" generator="{<nil> <nil> <nil> <nil> <nil> <nil> 0xc001a94420 <nil>}"                                                                                                 │
time="2022-07-11T15:36:21Z" level=debug msg="apps from generator: []" generator="{<nil> <nil> <nil> <nil> <nil> <nil> 0xc001a94420 <nil>}"                                                                                                 │
time="2022-07-11T15:36:21Z" level=info msg="Deleted application" app=argocd-test-develop-pr-2 appSet=root-appbundle-as                                                                                                                     │
time="2022-07-11T15:36:21Z" level=info msg="end reconcile" requeueAfter=0s

[sboschman]

The webhook code just uses the string as defined in the appset, it does not do any interpolating/transforming/merging/templating as added in #9080. It is however unclear to me how to get the interpolated generator params from the impl of #9080. The Transform method seems to return rendered Application objects.

[ejhayes]

Any thoughts on how this could be moved forward @crenshaw-dev?

[sboschman]

By chance I did have time last week to have a better look at it and was able to make a fix.

[ejhayes]

Much appreciated @sboschman!!!