Bug: repocreds do not GitHub App credentials

[abatilo]

Describe the bug
Between version v0.12.2 and v0.13.1, it seems like some bug was introduced which is causing authentication failures when trying to write changes back to the git repository.

To Reproduce
I have the following kind: Application that works perfectly with v0.12.2 and stops working at v0.13.1

---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: sudokurace
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
  annotations:
    argocd-image-updater.argoproj.io/write-back-method: git
    argocd-image-updater.argoproj.io/write-back-target: kustomization
    argocd-image-updater.argoproj.io/image-list: app=911907402684.dkr.ecr.us-west-2.amazonaws.com/sudokurace
    argocd-image-updater.argoproj.io/app.update-strategy: latest
    argocd-image-updater.argoproj.io/git-branch: main
spec:
  project: default

  source:
    repoURL: https://github.com/mentallyanimated/deployments.git
    targetRevision: HEAD
    path: sudokurace/base
    kustomize:

  destination:
    server: https://kubernetes.default.svc
    namespace: default

  syncPolicy:
    automated:
      prune: true
      selfHeal: true
      allowEmpty: false

Expected behavior
I would expect argocd-image-updater could continue to write commits to the spec.source.repoURL after doing the upgrade.

Additional context
I'm using a GitHub App for authentication as documented here

Version
v0.13.1

Logs
Here are trace level logs:

argocd-image-updater-7959c4f976-db76q argocd-image-updater time="2024-06-22T20:44:28Z" level=info msg="Setting new image to 911907402684.dkr.ecr.us-west-2.amazonaws.com/sudokurace:0004" alias=app application=sudokurace image_name=sudokurace image_tag=0003 registry=911907402684.dkr.ecr.us-west-2.amazonaws.com
argocd-image-updater-7959c4f976-db76q argocd-image-updater time="2024-06-22T20:44:28Z" level=trace msg="Setting Kustomize parameter 911907402684.dkr.ecr.us-west-2.amazonaws.com/sudokurace:0004" application=sudokurace
argocd-image-updater-7959c4f976-db76q argocd-image-updater time="2024-06-22T20:44:28Z" level=info msg="Successfully updated image '911907402684.dkr.ecr.us-west-2.amazonaws.com/sudokurace:0003' to '911907402684.dkr.ecr.us-west-2.amazonaws.com/sudokurace:0004', but pending spec update (dry run=false)" alias=app application=sudokurace image_name=sudokurace image_tag=0003 registry=911907402684.dkr.ecr.us-west-2.amazonaws.com
argocd-image-updater-7959c4f976-db76q argocd-image-updater time="2024-06-22T20:44:28Z" level=debug msg="Using commit message: build: automatic update of sudokurace\n\nupdates image sudokurace tag '0003' to '0004'\n"
argocd-image-updater-7959c4f976-db76q argocd-image-updater time="2024-06-22T20:44:28Z" level=info msg="Committing 1 parameter update(s) for application sudokurace" application=sudokurace
argocd-image-updater-7959c4f976-db76q argocd-image-updater time="2024-06-22T20:44:28Z" level=info msg="Starting configmap/secret informers"
argocd-image-updater-7959c4f976-db76q argocd-image-updater time="2024-06-22T20:44:30Z" level=info msg="Configmap/secret informer synced"
argocd-image-updater-7959c4f976-db76q argocd-image-updater time="2024-06-22T20:44:30Z" level=info msg="Initializing https://github.com/mentallyanimated/deployments.git to /tmp/git-sudokurace1680072880"
argocd-image-updater-7959c4f976-db76q argocd-image-updater time="2024-06-22T20:44:30Z" level=info msg="rm -rf /tmp/git-sudokurace1680072880" dir= execID=77aa9
argocd-image-updater-7959c4f976-db76q argocd-image-updater time="2024-06-22T20:44:30Z" level=info msg="configmap informer cancelled"
argocd-image-updater-7959c4f976-db76q argocd-image-updater time="2024-06-22T20:44:30Z" level=info msg="secrets informer cancelled"
argocd-image-updater-7959c4f976-db76q argocd-image-updater time="2024-06-22T20:44:30Z" level=info msg=Trace args="[rm -rf /tmp/git-sudokurace1680072880]" dir= operation_name="exec rm" time_ms=1.340393
argocd-image-updater-7959c4f976-db76q argocd-image-updater time="2024-06-22T20:44:30Z" level=info msg="git fetch origin --tags --force" dir=/tmp/git-sudokurace1680072880 execID=97636
argocd-image-updater-7959c4f976-db76q argocd-image-updater time="2024-06-22T20:44:30Z" level=error msg="`git fetch origin --tags --force` failed exit status 128: error: cannot run argocd: No such file or directory\nfatal: could not read Username for 'https://github.com': terminal prompts disabled" execID=97636
argocd-image-updater-7959c4f976-db76q argocd-image-updater time="2024-06-22T20:44:30Z" level=info msg=Trace args="[git fetch origin --tags --force]" dir=/tmp/git-sudokurace1680072880 operation_name="exec git" time_ms=124.54551599999999
argocd-image-updater-7959c4f976-db76q argocd-image-updater time="2024-06-22T20:44:30Z" level=info msg="Processing results: applications=8 images_considered=1 images_skipped=7 images_updated=0 errors=1"
argocd-image-updater-7959c4f976-db76q argocd-image-updater time="2024-06-22T20:44:30Z" level=error msg="Could not update application spec: `git fetch origin --tags --force` failed exit status 128: error: cannot run argocd: No such file or directory\nfatal: could not read Username for 'https://github.com': terminal prompts disabled" application=sudokurace

[abatilo]

After doing some more digging, it appears that the following statement is no longer being respected:

By default Argo CD Image Updater re-uses the credentials you have configured
in Argo CD for accessing the repository.

I say this because as soon as I update my write-back-method to explicitly use the credentials from the kind: Secret that ArgoCD itself is configured with:

    argocd-image-updater.argoproj.io/write-back-method: git:secret:argocd/argocd-repo-creds-github-mentallyanimated

Then argocd-image-updater starts to work again.

[abatilo]

Sorry, I did not mean to close the issue!

[abatilo]

After doing quite a bit of investigation, it appears that the bugs I was finding actually had to do with the fact that I was defining my GitHub app credentials via an ArgoCD credential template.

However, likewise, even after switching to a repository set of github credentials, it looks like the legacy repo provider doesn't support loading the GitHub app credentials at all.

[jannfis]

This should have been fixed by #737, which is not yet released. If you feel like it, you could test from a latest image though.

[benfuu]

@jannfis even with the latest image, the error becomes this issue:

time="2024-06-23T15:41:43Z" level=info msg=Trace args="[git fetch origin --tags --force --prune]" dir=/tmp/git-<alias>2547502857 operation_name="exec git" time_ms=208.298996
time="2024-06-23T15:41:43Z" level=error msg="Could not update application spec: `git fetch origin --tags --force --prune` failed exit status 128: time=\"2024-06-23T15:41:43Z\" level=fatal msg=\"ARGOCD_GIT_ASKPASS_NONCE is not set\"\nerror: unable to read askpass response from '/usr/local/bin/argocd-image-updater'\nfatal: could not read Username for 'https://github.com': terminal prompts disabled" application=<alias>                                                                                                                                                                                                      
time="2024-06-23T15:41:43Z" level=info msg="Processing results: applications=1 images_considered=1 images_skipped=0 images_updated=0 errors=1"

[benfuu]

@abatilo can you try with the latest image now? This should be fixed now.

[abatilo]

@benfuu Thank you so much for looking into this.

It looks like image quay.io/argoprojlabs/argocd-image-updater:latest@sha256:c9134fb6873a89ff21b49dee6834505d4c3620f321bde8406371b601b3a93694 fixes the problem that I reported at the top of this issue, but now there's a different error.

I'm happy to close this and open a different issue if you'd prefer?

Here's the new set of logs.

argocd-image-updater-768f74dc58-7mcs5 argocd-image-updater time="2024-06-24T13:41:46Z" level=info msg="Starting image update cycle, considering 8 annotated application(s) for update"
argocd-image-updater-768f74dc58-7mcs5 argocd-image-updater time="2024-06-24T13:41:46Z" level=info msg=/scripts/ecr-login.sh dir= execID=3cf43
argocd-image-updater-768f74dc58-7mcs5 argocd-image-updater time="2024-06-24T13:41:46Z" level=warning msg="\"latest\" strategy has been renamed to \"newest-build\". Please switch to the new convention as support for the old naming convention will be removed in future versions." image_alias=app image_name=911907402684.dkr.ecr.us-west-2.amazonaws.com/sudokurace registry_url=911907402684.dkr.ecr.us-west-2.amazonaws.com
argocd-image-updater-768f74dc58-7mcs5 argocd-image-updater time="2024-06-24T13:41:48Z" level=info msg="Setting new image to 911907402684.dkr.ecr.us-west-2.amazonaws.com/sudokurace:0006" alias=app application=sudokurace image_name=sudokurace image_tag=0005 registry=911907402684.dkr.ecr.us-west-2.amazonaws.com
argocd-image-updater-768f74dc58-7mcs5 argocd-image-updater time="2024-06-24T13:41:48Z" level=info msg="Successfully updated image '911907402684.dkr.ecr.us-west-2.amazonaws.com/sudokurace:0005' to '911907402684.dkr.ecr.us-west-2.amazonaws.com/sudokurace:0006', but pending spec update (dry run=false)" alias=app application=sudokurace image_name=sudokurace image_tag=0005 registry=911907402684.dkr.ecr.us-west-2.amazonaws.com
argocd-image-updater-768f74dc58-7mcs5 argocd-image-updater time="2024-06-24T13:41:48Z" level=info msg="Committing 1 parameter update(s) for application sudokurace" application=sudokurace
argocd-image-updater-768f74dc58-7mcs5 argocd-image-updater time="2024-06-24T13:41:48Z" level=info msg="Starting configmap/secret informers"
argocd-image-updater-768f74dc58-7mcs5 argocd-image-updater time="2024-06-24T13:41:48Z" level=info msg="Configmap/secret informer synced"
argocd-image-updater-768f74dc58-7mcs5 argocd-image-updater time="2024-06-24T13:41:48Z" level=info msg="Initializing https://github.com/mentallyanimated/deployments.git to /tmp/git-sudokurace3636392443"
argocd-image-updater-768f74dc58-7mcs5 argocd-image-updater time="2024-06-24T13:41:48Z" level=info msg="configmap informer cancelled"
argocd-image-updater-768f74dc58-7mcs5 argocd-image-updater time="2024-06-24T13:41:48Z" level=info msg="secrets informer cancelled"
argocd-image-updater-768f74dc58-7mcs5 argocd-image-updater time="2024-06-24T13:41:49Z" level=info msg="git fetch origin --tags --force --prune" dir=/tmp/git-sudokurace3636392443 execID=c8907
argocd-image-updater-768f74dc58-7mcs5 argocd-image-updater time="2024-06-24T13:41:50Z" level=info msg=Trace args="[git fetch origin --tags --force --prune]" dir=/tmp/git-sudokurace3636392443 operation_name="exec git" time_ms=1530.45609
argocd-image-updater-768f74dc58-7mcs5 argocd-image-updater time="2024-06-24T13:41:50Z" level=info msg="git config user.name argocd-image-updater" dir=/tmp/git-sudokurace3636392443 execID=6a487
argocd-image-updater-768f74dc58-7mcs5 argocd-image-updater time="2024-06-24T13:41:50Z" level=info msg=Trace args="[git config user.name argocd-image-updater]" dir=/tmp/git-sudokurace3636392443 operation_name="exec git" time_ms=3.894587
argocd-image-updater-768f74dc58-7mcs5 argocd-image-updater time="2024-06-24T13:41:50Z" level=info msg="git config user.email noreply@argoproj.io" dir=/tmp/git-sudokurace3636392443 execID=014a9
argocd-image-updater-768f74dc58-7mcs5 argocd-image-updater time="2024-06-24T13:41:50Z" level=info msg=Trace args="[git config user.email noreply@argoproj.io]" dir=/tmp/git-sudokurace3636392443 operation_name="exec git" time_ms=3.198214
argocd-image-updater-768f74dc58-7mcs5 argocd-image-updater time="2024-06-24T13:41:50Z" level=info msg="git checkout --force main" dir=/tmp/git-sudokurace3636392443 execID=c1178
argocd-image-updater-768f74dc58-7mcs5 argocd-image-updater time="2024-06-24T13:41:50Z" level=info msg=Trace args="[git checkout --force main]" dir=/tmp/git-sudokurace3636392443 operation_name="exec git" time_ms=11.987477
argocd-image-updater-768f74dc58-7mcs5 argocd-image-updater time="2024-06-24T13:41:50Z" level=info msg="git clean -ffdx" dir=/tmp/git-sudokurace3636392443 execID=f5fdb
argocd-image-updater-768f74dc58-7mcs5 argocd-image-updater time="2024-06-24T13:41:50Z" level=info msg="updating base /tmp/git-sudokurace3636392443/sudokurace/base" application=sudokurace
argocd-image-updater-768f74dc58-7mcs5 argocd-image-updater time="2024-06-24T13:41:50Z" level=info msg=Trace args="[git clean -ffdx]" dir=/tmp/git-sudokurace3636392443 operation_name="exec git" time_ms=2.0949940000000002
argocd-image-updater-768f74dc58-7mcs5 argocd-image-updater time="2024-06-24T13:41:50Z" level=info msg="git -c gpg.format=openpgp commit -a -S -F /tmp/image-updater-commit-msg2517640946" dir=/tmp/git-sudokurace3636392443 execID=5811c
argocd-image-updater-768f74dc58-7mcs5 argocd-image-updater time="2024-06-24T13:41:50Z" level=error msg="`git -c gpg.format=openpgp commit -a -S -F /tmp/image-updater-commit-msg2517640946` failed exit status 128: error: cannot run gpg: No such file or directory\nerror: gpg failed to sign the data:\n(no gpg output)\nfatal: failed to write commit object" execID=5811c
argocd-image-updater-768f74dc58-7mcs5 argocd-image-updater time="2024-06-24T13:41:50Z" level=info msg=Trace args="[git -c gpg.format=openpgp commit -a -S -F /tmp/image-updater-commit-msg2517640946]" dir=/tmp/git-sudokurace3636392443 operation_name="exec git" time_ms=5.892401
argocd-image-updater-768f74dc58-7mcs5 argocd-image-updater time="2024-06-24T13:41:50Z" level=error
argocd-image-updater-768f74dc58-7mcs5 argocd-image-updater time="2024-06-24T13:41:50Z" level=error msg="Could not update application spec: `git -c gpg.format=openpgp commit -a -S -F /tmp/image-updater-commit-msg2517640946` failed exit status 128: error: cannot run gpg: No such file or directory\nerror: gpg failed to sign the data:\n(no gpg output)\nfatal: failed to write commit object" application=sudokurace
argocd-image-updater-768f74dc58-7mcs5 argocd-image-updater time="2024-06-24T13:41:50Z" level=info msg="Processing results: applications=8 images_considered=1 images_skipped=7 images_updated=0 errors=1"

Perhaps there's another pass through required somewhere for gpg settings? I am NOT configuring anything related to commit signing whatsoever.

[benfuu]

It seems gpg params are added to the git client even when disabled by default:

argocd-image-updater/ext/git/writer.go

Line 31 in 9ad9e4f

func (m *nativeGitClient) Commit(pathSpec string, opts *CommitOptions) error {