ApplicationSet does not support private repos configured using SSH

[chamarakera]

Getting below error with ApplicationSet although the repository has been configured with ArgoCD to use SSH Private Key Credentials. Same repository works fine with Application objects though.

time="2021-03-18T09:48:03Z" level=error msg="`git fetch origin --tags --force` failed exit status 128: load pubkey \"/dev/shm/919020132\": invalid format\r\nNo RSA host key is known for github.com and you have requested strict checking.\r\nHost key verification failed.\r\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists." execID=uyS7R
time="2021-03-18T09:48:03Z" level=info msg=Trace args="[git fetch origin --tags --force]" dir=/tmp/git@github.com_mycompany_k8s-config-repo operation_name="exec git" time_ms=36.487388
time="2021-03-18T09:48:03Z" level=error msg="error generating params" error="Error during fetching repo: `git fetch origin --tags --force` failed exit status 128: load pubkey \"/dev/shm/919020132\": invalid format\r\nNo RSA host key is known for github.com and you have requested strict checking.\r\nHost key verification failed.\r\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists." generator="&{0xc0001c3140}"
2021-03-18T09:48:03.835Z        ERROR   controller-runtime.manager.controller.applicationset    Reconciler error        {"reconciler group": "argoproj.io", "reconciler kind": "ApplicationSet", "name": "cluster-addons", "namespace": "argocd", "error": "Error during fetching repo: `git fetch origin --tags --force` failed exit status 128: load pubkey \"/dev/shm/919020132\": invalid format\r\nNo RSA host key is known for github.com and you have requested strict checking.\r\nHost key verification failed.\r\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists.", "errorVerbose": "`git fetch origin --tags --force` failed exit status 128: load pubkey \"/dev/shm/919020132\": invalid format\r\nNo RSA host key is known for github.com and you have requested strict checking.\r\nHost key verification failed.\r\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists.\nError during fetching repo\ngithub.com/argoproj-labs/applicationset/pkg/services.checkoutRepo\n\t/workspace/pkg/services/repo_service.go:159\ngithub.com/argoproj-labs/applicationset/pkg/services.(*argoCDService).GetDirectories\n\t/workspace/pkg/services/repo_service.go:84\ngithub.com/argoproj-labs/applicationset/pkg/generators.(*GitGenerator).generateParamsForGitDirectories\n\t/workspace/pkg/generators/git.go:78\ngithub.com/argoproj-labs/applicationset/pkg/generators.(*GitGenerator).GenerateParams\n\t/workspace/pkg/generators/git.go:62\ngithub.com/argoproj-labs/applicationset/pkg/controllers.(*ApplicationSetReconciler).generateApplications\n\t/workspace/pkg/controllers/applicationset_controller.go:349\ngithub.com/argoproj-labs/applicationset/pkg/controllers.(*ApplicationSetReconciler).Reconcile\n\t/workspace/pkg/controllers/applicationset_controller.go:93\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.0/pkg/internal/controller/controller.go:263\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.0/pkg/internal/controller/controller.go:235\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.1\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.0/pkg/internal/controller/controller.go:198\nk8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.19.2/pkg/util/wait/wait.go:185\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.19.2/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.19.2/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.19.2/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext\n\t/go/pkg/mod/k8s.io/apimachinery@v0.19.2/pkg/util/wait/wait.go:185\nk8s.io/apimachinery/pkg/util/wait.UntilWithContext\n\t/go/pkg/mod/k8s.io/apimachinery@v0.19.2/pkg/util/wait/wait.go:99\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
github.com/go-logr/zapr.(*zapLogger).Error
        /go/pkg/mod/github.com/go-logr/zapr@v0.2.0/zapr.go:132
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.0/pkg/internal/controller/controller.go:267
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.0/pkg/internal/controller/controller.go:235
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.1
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.0/pkg/internal/controller/controller.go:198
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1
        /go/pkg/mod/k8s.io/apimachinery@v0.19.2/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
        /go/pkg/mod/k8s.io/apimachinery@v0.19.2/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
        /go/pkg/mod/k8s.io/apimachinery@v0.19.2/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
        /go/pkg/mod/k8s.io/apimachinery@v0.19.2/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext
        /go/pkg/mod/k8s.io/apimachinery@v0.19.2/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.UntilWithContext
        /go/pkg/mod/k8s.io/apimachinery@v0.19.2/pkg/util/wait/wait.go:99

My ApplicationSet manifest

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: cluster-addons
  namespace: argocd
spec:
  generators:
  - git:
      repoURL: git@github.com:mycompany/k8s-config-repo.git
      revision: HEAD
      directories:
      - path: kustomize/addons/*
  template:
    metadata:
      name: '{{path.basename}}'
    spec:
      project: default
      source:
        repoURL: git@github.com:mycompany/k8s-config-repo.git
        targetRevision: HEAD
        path: '{{path}}'
      destination:
        server: https://kubernetes.default.svc
        namespace: '{{path.basename}}'

[jgwest]

I can reproduce the issue and see the same error, thanks @ckeragala!

[fardin01]

Hi @jgwest. I don't understand how #164 fixes the issue. As far as I understand, ArgoCD repo server finds private repo credentials in a secret labeled with argocd.argoproj.io/secret-type: repo-creds. It's not mounted in the container. I'm still getting Could not read from remote repository with v0.2.0 of ApplicationSet and v2.1.7 of ArgoCD. I'd appreciate any suggestions.

[fardin01]

Well, it started working after a couple of minutes. Not sure what it was that the controller didn't like 🤷🏼