Bug: Webhook trigger from Bitbucket does not refresh Applications in 2.11.0

[Alwinius]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

After the update from ArgoCD 2.10.9 to 2.11.0 the sync trigger on webhook from Bitbucket does not work anymore.

To Reproduce

1. Install ArgoCD 2.11.0
2. Connect a repository that is hosted in Bitbucket and create Application for that repo
3. Setup a webhook in Bitbucket to trigger ArgoCD https://argocd.cluster.../api/webhook
4. Push to bitbucket
5. Observe in ArgoCD that the application is not refreshed
6. perform a manual refresh to detect changes

Expected behavior

The refresh is triggered automatically.

Version

argocd: v2.11.0+d3f33c0
  BuildDate: 2024-05-07T16:01:41Z
  GitCommit: d3f33c00197e7f1d16f2a73ce1aeced464b07175
  GitTreeState: clean
  GoVersion: go1.21.9
  Compiler: gc
  Platform: linux/amd64

The version of Bitbucket is v8.19.3

Logs

time="2024-05-20T09:21:51Z" level=info msg="Received push event repo: ssh://git@git.domain.com:7999/incs/k8s-aws-eks-staging.git, revision: master, touchedHead: true"             
time="2024-05-20T09:21:51Z" level=info msg="Received push event repo: https://git.domain.com/scm/incs/k8s-aws-eks-staging.git, revision: master, touchedHead: true"                

Bitbucket receives HTTP 200 and the above log is shown in argocd-server

[kyleli666]

I have same issue on v2.11.0 with bitbucket cloud.
I can see "Received push event log" and Bitbucket get 200 response, but No refresh action log in argocd-server.

[mh013370]

Same here.

[kyleli666]

Tested again in v2.10.10 and v2.10.11, the webhook from Bitbucket works.
But v2.11.1 and v2.11.2 still does not work, argocd-server responses code 200 much faster than v2.10.x while not log any app refresh information in the console.
Here is the debug level log from v2.11.2, it seems doing some work after "Received push event repo" but finally not triggring the app refresh.

time="2024-05-24T03:19:16Z" level=info msg="Received push event repo: https://bitbucket.org/myworkspace/mycoderepo, revision: develop, touchedHead: true"
time="2024-05-24T03:19:16Z" level=debug msg="https://bitbucket.org/myworkspace/mycoderepo.git uses repoURL https://bitbucket.org/myworkspace/mycoderepo"
time="2024-05-24T03:19:16Z" level=debug msg="https://bitbucket.org/myworkspace/mycoderepo.git uses repoURL https://bitbucket.org/myworkspace/mycoderepo"
time="2024-05-24T03:19:16Z" level=debug msg="https://bitbucket.org/myworkspace/mycoderepo.git uses repoURL https://bitbucket.org/myworkspace/mycoderepo"
time="2024-05-24T03:19:16Z" level=debug msg="https://bitbucket.org/myworkspace/mycoderepo.git uses repoURL https://bitbucket.org/myworkspace/mycoderepo"

A normal log from v2.10.11 looks like this

time="2024-05-24T03:26:39Z" level=info msg="Received push event repo: https://bitbucket.org/myworkspace/mycoderepo, revision: develop, touchedHead: true"
time="2024-05-24T03:26:39Z" level=debug msg="https://bitbucket.org/myworkspace/mycoderepo.git uses repoURL https://bitbucket.org/myworkspace/mycoderepo"
time="2024-05-24T03:26:39Z" level=info msg="Requested app 'ba-deployment-api' refresh"
time="2024-05-24T03:26:39Z" level=debug msg="https://bitbucket.org/myworkspace/mycoderepo.git uses repoURL https://bitbucket.org/myworkspace/mycoderepo"
time="2024-05-24T03:26:40Z" level=info msg="Requested app 'cb2-conn-test-org-sk3' refresh"
time="2024-05-24T03:26:40Z" level=debug msg="https://bitbucket.org/myworkspace/mycoderepo.git uses repoURL https://bitbucket.org/myworkspace/mycoderepo"
time="2024-05-24T03:26:40Z" level=info msg="Requested app 'cb2-conn-test-org-sk3-jeff01' refresh"

[de-slalonde]

I'm seeing the same issue. On 2.11.0 and 2.11.1 my Bitbucket webhooks do not trigger a refresh. Things are fine on 2.10.9.

[petrlebedev]

2.11.2 Same issue.

[initialed85]

Hi team, I was having this issue using https://artifacthub.io/packages/helm/argo/argo-cd/7.0.0 and it ended up being that I had enabled the ingress for both argocd-server and argocd-applicationsets / Git Generator webhook for the same FQDN.

It seems that they both have handlers for /api/webhook and therefore similarly error patterns, which adds to confusion.

You can check if this is what's happening to you with the following:

kubectl -n argo-cd port-forward service/argo-cd-argocd-server 8080:80

curl -vvv -L -X POST -H 'X-Event-Key: repo:push' -H 'Host: your-real-argo-fqdn.org' http://localhost:8080/api/webhook

Instead of Unknown webhook event you should get Webhook processing failed: event not defined to be parsed

The fix (for me) was to disable that second ingress. I think Kubernetes ingresses merge and I guess argo-applicationsets is defined second (thus overwriting the /api/webhook handler that we need).

Edit:

• https://github.com/argoproj/argo-cd/blob/master/util/webhook/webhook.go#L386
• https://github.com/argoproj/argo-cd/blob/master/applicationset/webhook/webhook.go#L148

[initialed85]

Hi team, I was having this issue using https://artifacthub.io/packages/helm/argo/argo-cd/7.0.0 and it ended up being that I had enabled the ingress for both argocd-server and argocd-applicationsets / Git Generator webhook for the same FQDN.

It seems that they both have handlers for /api/webhook and therefore similarly error patterns, which adds to confusion.

You can check if this is what's happening to you with the following:

kubectl -n argo-cd port-forward service/argo-cd-argocd-server 8080:80

curl -vvv -L -X POST -H 'X-Event-Key: repo:push' -H 'Host: your-real-argo-fqdn.org' http://localhost:8080/api/webhook

Instead of Unknown webhook event you should get Webhook processing failed: event not defined to be parsed

The fix (for me) was to disable that second ingress. I think Kubernetes ingresses merge and I guess argo-applicationsets is defined second (thus overwriting the /api/webhook handler that we need).

Edit:

• https://github.com/argoproj/argo-cd/blob/master/util/webhook/webhook.go#L386
• https://github.com/argoproj/argo-cd/blob/master/applicationset/webhook/webhook.go#L148

EDIT: Sorry team, I think my message is incorrect- or at least half-incorrect; despite the above fix causing the webhook to land correctly in the argo-cd-argocd-repo-server pod, it was still causing no action (other than a log line) until I forced the image tag to v2.10.9

[cdemi]

Having the same issue. Rolling back to v2.10.9 make it work again

[dmavis]

Same issue here as well. This is currently blocking our upgrade to 2.11.x.

[agaudreault]

Bitbucket and Azure devops do not provide any implementation for changedFiles.

argo-cd/util/webhook/webhook.go

Lines 175 to 217 in ca5caa2

	case bitbucket.RepoPushPayload:
	// See: https://confluence.atlassian.com/bitbucket/event-payloads-740262817.html#EventPayloads-Push
	// NOTE: this is untested
	webURLs = append(webURLs, payload.Repository.Links.HTML.Href)
	// TODO: bitbucket includes multiple changes as part of a single event.
	// We only pick the first but need to consider how to handle multiple
	for _, change := range payload.Push.Changes {
	revision = change.New.Name
	break
	}
	// Not actually sure how to check if the incoming change affected HEAD just by examining the
	// payload alone. To be safe, we just return true and let the controller check for himself.
	touchedHead = true
	// Bitbucket does not include a list of changed files anywhere in it's payload
	// so we cannot update changedFiles for this type of payload
	case bitbucketserver.RepositoryReferenceChangedPayload:
	// Webhook module does not parse the inner links
	if payload.Repository.Links != nil {
	for _, l := range payload.Repository.Links["clone"].([]interface{}) {
	link := l.(map[string]interface{})
	if link["name"] == "http" {
	webURLs = append(webURLs, link["href"].(string))
	}
	if link["name"] == "ssh" {
	webURLs = append(webURLs, link["href"].(string))
	}
	}
	}
	// TODO: bitbucket includes multiple changes as part of a single event.
	// We only pick the first but need to consider how to handle multiple
	for _, change := range payload.Changes {
	revision = parseRevision(change.Reference.ID)
	break
	}
	// Not actually sure how to check if the incoming change affected HEAD just by examining the
	// payload alone. To be safe, we just return true and let the controller check for himself.
	touchedHead = true
	// Bitbucket does not include a list of changed files anywhere in it's payload
	// so we cannot update changedFiles for this type of payload

The current code from #15636 will not trigger a refresh if there are no changed files:

argo-cd/util/app/path/path.go

Lines 117 to 121 in 3dcddee

	// empty slice means there was no changes to any files
	// so we should not refresh
	if len(changedFiles) == 0 {
	return false
	}

If anyone with bitbucket/azure SCM integration wants to implement the changed files feature or update the current implementation to account for SCM without changed files support, it would be a great contribution!