Not able to use GIT write back with chart from helm chart repo in repoURL

[flozzone]

Describe the bug

Using GIT write back together with Helm Chart from Helm Chart repository in repoURL isn't supported, because the ArgoCD image-updater tries to resolve the URL under spec.source.repoURL as the GIT repository to create update-commits, but since its a Helm repository, this fails with:

time="2022-04-22T09:47:06Z" level=error msg="Could not update application spec: could not get creds for repo 'https://nexus.helm.repo.test-app.nwe.org/repository/helm-charts': credentials for 'https://nexus.helm.repo.test-app.nwe.org/repository/helm-charts' are not configured in Argo CD settings" application=test-app

To Reproduce

Create a Application resource like the following one:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: test-app
  namespace: argocd-gitops
  annotations:
    # configure ArgoCD image updater which updates image.name and image.tag according to latest tag in container registry
    argocd-image-updater.argoproj.io/image-list: frontend=test-repo/test-app
    argocd-image-updater.argoproj.io/frontend.update-strategy: latest
    argocd-image-updater.argoproj.io/frontend.helm.image-name: image.name
    argocd-image-updater.argoproj.io/frontend.helm.image-tag: image.tag
    argocd-image-updater.argoproj.io/frontend.force-update: "true"

    # write ArgoCD image updater updates back to a separate GIT branch
    argocd-image-updater.argoproj.io/write-back-method: git
    argocd-image-updater.argoproj.io/git-branch: :update-next
spec:
  destination:
    namespace: test-app
    server: https://kubernetes.default.svc
  project: apps
  source:
    chart: room-catalogue
    repoURL: 'https://nexus.helm.repo.test-app.nwe.org/repository/helm-charts'
    targetRevision: 0.1.5
    helm:
      parameters:
        - name: global.hostname
          value: test-app.nwe.org
        - name: image.name
          value: test-repo/test-app

Expected behavior

Having ArgoCD image updates committed to GIT

Additional context

One would expect to have a annotation based configuration alternative to set repoURL to a separate GIT repository url, looking like:

argocd-image-updater.argoproj.io/git-repository: https://github.com/nwe/argocd-gitops-repo.git

Version

quay.io/argoprojlabs/argocd-image-updater:v0.12.0

Logs

time="2022-04-22T09:47:06Z" level=error msg="Could not update application spec: could not get creds for repo 'https://nexus.helm.repo.test-app.nwe.org/repository/helm-charts': credentials for 'https://nexus.helm.repo.test-app.nwe.org/repository/helm-charts' are not configured in Argo CD settings" application=test-app

[ugurarpaci]

+1

[tomsozolins]

+1

[karma-git]

I have the same issue.

Found several threads in slack related to the issue: 1, 2

And I also think that this feature (git writeback method + external helm repo) could be implemented via additional anotations.

[filiprafaj]

hi, my workaround is using app-of-apps pattern
(disclamer - the behavior is not ideal as it is randomly removing the tags and adding them again - this is something I want to investigate)

• add image updater annotations to app-of-apps.yaml
• source of app-of-apps.yaml is kustomize repo containing:
  • application.yaml ( Helm source and image specified using helm parameter)
  • kustomconfig.yaml
  • kustomization.yaml

app-of-apps.yaml

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: app-of-apps
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
  annotations:
    argocd-image-updater.argoproj.io/write-back-target: kustomization
    argocd-image-updater.argoproj.io/image-list: myalias=registry.gitlab.com/...
    argocd-image-updater.argoproj.io/myalias.pull-secret: pullsecret:argocd/regcreds
    argocd-image-updater.argoproj.io/myalias.force-update: "true"
    argocd-image-updater.argoproj.io/write-back-method: git:secret:argocd/image-updater-repocreds
    argocd-image-updater.argoproj.io/git-branch: main
spec:
  ...
  source:
    repoURL: https://gitlab.com/some/repo.git
    targetRevision: HEAD
    path: app-prod
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd
  syncPolicy:
    automated:
      selfHeal: true
      prune: false

application.yaml

apiVersion: argoproj.io/v1alpha1
kind: Application
...
spec:
  ...
  source:
    ...
    helm:
      parameters:
      - name: image
        value: image_name:image_tag

kustomconfig.yaml - to tell kustomize to overwrite those values use following configuration

# kustomconfig.yaml 
images:
- path: spec/source/helm/parameters[]/value
  kind: Application

kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
configurations:
- kustomconfig.yaml

[flozzone]

Looks interesting! @filiprafaj could you please complete your example, its kind of hard to understand:

• you refer to app-of-apps.yaml but none is marked as such
• could you put the respective image updater annotations with the git-writeback setting inside the app-of-apps.yaml resource

[filiprafaj]

@flozzone app-of-apps.yaml is something like this

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: app-prod
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
  annotations:
    argocd-image-updater.argoproj.io/write-back-target: kustomization
    argocd-image-updater.argoproj.io/image-list: myalias=registry.gitlab.com/...
    argocd-image-updater.argoproj.io/myalias.pull-secret: pullsecret:argocd/regcreds
    argocd-image-updater.argoproj.io/myalias.force-update: "true"
    argocd-image-updater.argoproj.io/write-back-method: git:secret:argocd/image-updater-repocreds
    argocd-image-updater.argoproj.io/git-branch: main
spec:
  project: prod
  source:
    repoURL: https://gitlab.com/some/repo.git
    targetRevision: HEAD
    path: app-prod
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd
  syncPolicy:
    automated:
      selfHeal: true
      prune: false

[etiennetremel]

@filiprafaj nice hack but this work around would probably not work if the Helm chart define tags and image as separate values.

Most of the charts write the image structure like this:

image:
  repository: nginx
  tag: 1.23.1

[filiprafaj]

@filiprafaj nice hack but this work around would probably not work if the Helm chart define tags and image as separate values.

Most of the charts write the image structure like this:

image:
  repository: nginx
  tag: 1.23.1

you're right :) I use it only on my own charts

[AnhQKatalon]

Is there any workaround for this? I am using the helm chart from AWS ECR OCI, and it seems the same error occurs. Argo Image Updater cannot commit to git since it is a helm repo.

time="2022-11-25T15:01:03Z" level=error msg="`git fetch origin --tags --force` failed exit status 128: fatal: '550842083568.dkr.ecr.us-east-1.amazonaws.com' does not appear to be a git repository\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists." execID=8c2ea
time="2022-11-25T15:01:03Z" level=info msg=Trace args="[git fetch origin --tags --force]" dir=/tmp/git-gen5-backend-api-release765755698 operation_name="exec git" time_ms=6.568499
time="2022-11-25T15:01:03Z" level=error msg="Could not update application spec: `git fetch origin --tags --force` failed exit status 128: fatal: '550842083568.dkr.ecr.us-east-1.amazonaws.com' does not appear to be a git repository\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists." application=gen5-backend-api-release
time="2022-11-25T15:01:03Z" level=info msg="Processing results: applications=2 images_considered=2 images_skipped=0 images_updated=0 errors=1"

This issue makes the image updater so bad since we are using helm repo to deploy instead of directly pointing to the Git repo containing the chart manifests

[portswigger-tim]

@AnhQKatalon - I worked around this by creating a "local" chart (in the ArgoCD repo) that has the upstream chart as a dependency. I also had to override the image tag and image repository keys to target the dependency.

The override values get created inside the local helm chart directory and Argo applies them.

[rakeshramakrishnan-hbk]

@filiprafaj - can you please elaborate your setup and solution? Not able to follow.
The app of apps is not getting picked up by the image updater as it is of kind Directory - so it skips it.

[filiprafaj]

hi @rakeshramakrishnan-hbk, I have updated my comment to be more clear
I have the app-of-apps.yaml of kind kustomize

[devopsidiot]

Hi @filiprafaj - I'm trying to make this work in my environments (thank you so much btw for this work around). If you don't mind, are you using an app-of-apps pattern w/ multiple applications inside and all are sharing the

images:
- path: spec/source/helm/parameters[]/value
   kind: Application

value? Would you be so kind as to share how you store your image value in your values.yaml file in your helm repo? Do you encompass your child applications in the same directory as your app-of-apps application?

[filiprafaj]

Hi @devopsidiot , at the moment I use something like this https://github.com/filiprafaj/argocd-image-updater-helm-demo

The new thing there is the Deployment in the app folder - thanks to this I don't need the force-update annotation anymore - it behaves better when there are multiple images to update in a single helmchart.

(btw for some large images I use DaemonSet instead of Deployment together with argocd.argoproj.io/hook: PreSync, so it also work as a pre-pull)

[devopsidiot]

@filiprafaj - thank you SO MUCH. Just one final question and I think you've given me the keys to my kingdom: The image updater HAS to be installed in the app-of-apps? I currently have it installed elsewhere but it has access to all namespaces.

[filiprafaj]

@devopsidiot I have image uptater running in the argocd namespace where I also deploy the Application resources, but I don't think it is necessary