Webhook is not working for me

[ptchau2003]

I am setting github webhook to trigger the ArgoCD sync, but somehow it is not working.
I am setting time.reconcilation = 1d (to simulate disable the sync and see the webhook can do it)

Issue

1. The webhook arrives to ArgoCD applicationset-controller (this controller is responsible for webhook handling), port 7000
2. The AppSet refresh is triggered by the webhook refresh ApplicationSet argocd/applications-test from webhook

time="2023-01-05T17:30:27Z" level=info msg="Received push event repo: XXXXXXXXXXX, revision: XXXX, touchedHead: false"
time="2023-01-05T17:30:27Z" level=info msg="git fetch origin XXX --tags --force" dir=/tmp/XXXXXXXXXXX execID=0db6f
time="2023-01-05T17:30:27Z" level=info msg="refresh ApplicationSet argocd/applications-test from webhook"
time="2023-01-05T17:30:30Z" level=info msg=Trace args="[git fetch origin XXX --tags --force]" dir=/tmp/XXXXXXXXXXX operation_name="exec git" time_ms=3171.829371
time="2023-01-05T17:30:30Z" level=info msg="git checkout --force XXXX" dir=/tmp/XXXXXXXXXXX execID=717be

3. But I do not see the pipeline trigger. The application is still in old version. In the log it said that, there is no change in application.

Some information
As far as I understand, the ArgoCD is working as:

1. ArgoCD will sync the external repo (like Github) to argocd-repo-server and have cache in local Redis . Reposerver will use Redis as a repository cache. This interval sync is decided by time.hard.reconcilation (it is new parameter at v5.7.0).
2. ArgoCD will sync application live manifest with desired manifest. This interval sync is decided by time.reconcilation
   So if we set the time.hard.reconcilation = time.reconcilation = 3 mins, the maximum delay is 6 mins.
3. To overcome it, webhook is provided. As I think, webhook must do both things: sync the repo (to get the latest manifest) and sync the app (apply the latest repo to application). Because if the webhook trigger app sync, but if the repo is still old (in cache) then nothing new is updated, isn't it? Please correct me if I am wrong.
4. Base on this code, I am not sure that webhook triggers the repo sync and sync the app at the same time https://github.com/argoproj/argo-cd/blob/master/applicationset/webhook/webhook.go#L551-L564
   I am stuck to confirm it. Because if I am setting time.hard.reconcilation = time.reconcilation = 1d for example, nothing is updated.
   We can set both time.hard.reconcilation = time.reconcilation = 30 seconds for example, so the deployment time max is 1 minutes. But it is not the webhook purpose, and still unnecessary increase the load of the system.

Please, anyone can confirm that https://github.com/argoproj/argo-cd/blob/master/applicationset/webhook/webhook.go#L551-L564 can do item 3) on above?

I am using helm for application set, detail

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: applications-test-new
  namespace: argocd
  # annotations:
  #   argocd.argoproj.io/manifest-generate-paths: .
spec:
  syncPolicy:
    preserveResourcesOnDeletion: true
  generators:
  - git:
      repoURL: https://github.com/XXXXXXXXXXX/argocd-example-apps.git
      revision: master
      directories:
      - path: helm-guestbook
  template:
    metadata:
      # There is no longer to escape the bracket as it is not using helm apply anymore.
      name: '{{path.basename}}.test'
      namespace: argocd
      # This is annotation for templates of the application, so the wehhook will look into it for the change
      # Ref: https://argo-cd.readthedocs.io/en/stable/operator-manual/high_availability/#webhook-and-manifest-paths-annotation
      # annotations:
      #   argocd.argoproj.io/manifest-generate-paths: .
      # This will enable cascade delete. Different than Helmfile where
      # you need to specify a flag in order to delete a Helm Chart release
      finalizers:
      - resources-finalizer.argocd.argoproj.io
      labels:
        app: '{{path.basename}}'
        env: test
    spec:
      project: 'applications-test-new'
      source:
        repoURL: 'https://github.com/XXXXXXXXXXX/argocd-example-apps.git'
        targetRevision: master
        path: '{{path}}'
        helm:
          releaseName: '{{path.basename}}'
          valueFiles:
            - values.yaml
      destination:
        server: 'https://kubernetes.default.svc'
        namespace: '{{path.basename}}'
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
        - CreateNamespace=true

[ptchau2003]

I think I found the issue 🎉, the webhook is sent to argocd-server, here: https://github.com/argoproj/argo-cd/blob/master/util/webhook/webhook.go
not the applicationset-controller (which has the same webhook fucntion)

[donRumatta]

Could you please explain, how to get the URL for applicationset-controller webhook? Seems like I faced the same problem: I'm just sending push events from Gitlab to the URL where argocd UI is situated (adding /api/webhook). And argocd immediatly triggers only deletion of an app from appset, but waits 3 minutes before creating it.

I'm using Git Generator and there is no info in docs how to build such URL (https://argocd-applicationset.readthedocs.io/en/stable/Generators-Git/#webhook-configuration)

[donRumatta]

Added an istio-virtualservice for appset-controller service to get a hostname outside k8s, and this new webhook triggers app creation as it should be.

[pre]

At the moment ApplicationSet webhook is broken. Even though the webhook is received and it shows in the logs, the Git generator shows applications "unchanged" until "something in the Redis cache expires":

• Webhook not refreshing applications #15427