SSH auth not working with git

[cyclingwithelephants]

Checklist:

• I've included the version.
• I've included reproduction steps.
• I've included the workflow YAML.
• I've included the logs.

What happened:
following the example at https://github.com/argoproj/argo/blob/master/examples/input-artifact-git.yaml
I am unable to make this workflow pull my git repository.
I executed the workflow using argo submit -n argo --watch <path-to_workflow>

What you expected to happen:
I expected to be able to pull from my repository
How to reproduce it (as minimally and precisely as possible):
my workflow looks exactly as above except with the appropriate git address, and with

sshPrivateKeySecret:
            name: argo-secrets
            key: git-ssh-private-key

uncommented and filled in with my secret details. I also changed the revision field to master

My secret looks like this

apiVersion: v1
kind: Secret
metadata:
  namespace: argo
  name: argo-secrets
type: Opaque
data:
  tf-api-key: <redacted>
  gcp-service-account: <redacted>
  git-ssh-private-key: <redacted> (I made sure there was no newline and of course base64 encoded it)

Anything else we need to know?:

Environment:

• Argo version:

argo: v2.8.1
  BuildDate: 2020-05-28T23:40:32Z
  GitCommit: 0fff4b21c21c5ff5adbb5ff62c68e67edd95d6b8
  GitTreeState: clean
  GitTag: v2.8.1
  GoVersion: go1.13.4
  Compiler: gc
  Platform: darwin/amd64

• Kubernetes version :

clientVersion:
  buildDate: "2020-03-12T23:41:24Z"
  compiler: gc
  gitCommit: 8d8aa39598534325ad77120c120a22b3a990b5ea
  gitTreeState: clean
  gitVersion: v1.17.4
  goVersion: go1.14
  major: "1"
  minor: "17"
  platform: darwin/amd64
serverVersion:
  buildDate: "2020-05-01T21:47:04Z"
  compiler: gc
  gitCommit: 9cabee15e0922c3b36724de4866a98f6c2da5e6a
  gitTreeState: clean
  gitVersion: v1.16.8-gke.15
  goVersion: go1.13.8b4
  major: "1"
  minor: 16+
  platform: linux/amd64

Other debugging information (if applicable)
I wasn't able to understand the line about installing on GKE, I have no idea what YOURNAME has to do in this line since the deployment shouldn't be tied to my name?

NOTE: On GKE, you may need to grant your account the ability to create new clusterroles

kubectl create clusterrolebinding YOURNAME-cluster-admin-binding --clusterrole=cluster-admin --user=YOUREMAIL@gmail.com

Additionally, the containers are still intialising on the GKE console, I think because this git magic happens in an init container?

• workflow result:

DEBU[0000] CLI version                                   version="{v2.8.1 2020-05-28T23:40:32Z 0fff4b21c21c5ff5adbb5ff62c68e67edd95d6b8 v2.8.1 clean go1.13.4 gc darwin/amd64}"
DEBU[0000] Client options                                opts="{{ false false} 0x2175d40 0xc00004d900}"
FATA[0000] workflows.argoproj.io "input-artifact-git-kx22d" not found

• executor logs:

Error from server (NotFound): pods "input-artifact-git-kx22d" not found
Error from server (NotFound): pods "input-artifact-git-kx22d" not found

• workflow-controller logs:

time="2020-06-17T15:19:48Z" level=info msg="config map" name=workflow-controller-configmap
time="2020-06-17T15:19:48Z" level=info msg="Configuration:\nartifactRepository:\n  gcs:\n    bucket: adam-argo-test\n    serviceAccountKeySecret:\n      key: gcp-service-account\n      name: argo-secrets\nfeatureFlags: {}\nmetricsConfig:\n  disableLegacy: false\npodSpecLogStrategy: {}\ntelemetryConfig:\n  disableLegacy: false\n"
time="2020-06-17T15:19:48Z" level=info msg="Persistence configuration disabled"
time="2020-06-17T15:19:48Z" level=info msg="Starting CronWorkflow controller"
time="2020-06-17T15:19:48Z" level=info msg="Starting workflow TTL controller (resync 20m0s)"
time="2020-06-17T15:19:48Z" level=info msg="Starting Workflow Controller" version=v2.8.1+0fff4b2.dirty
time="2020-06-17T15:19:48Z" level=info msg="Workers: workflow: 32, pod: 32"
time="2020-06-17T15:19:48Z" level=info msg="Persistence disabled - so archived workflow GC disabled - you must restart the controller if you enable this"
time="2020-06-17T15:19:48Z" level=info msg="Performing periodic GC every 5m0s"
time="2020-06-17T15:19:48Z" level=info msg="Started workflow TTL worker"
time="2020-06-17T15:20:07Z" level=info msg="Processing workflow" namespace=argo workflow=input-artifact-git-7lttk
time="2020-06-17T15:20:07Z" level=info msg="Updated phase  -> Running" namespace=argo workflow=input-artifact-git-7lttk
time="2020-06-17T15:20:07Z" level=info msg="Pod node {input-artifact-git-7lttk input-artifact-git-7lttk input-artifact-git-7lttk Pod git-clone nil   local/input-artifact-git-7lttk Pending   2020-06-17 15:20:07.064148991 +0000 UTC 0001-01-01 00:00:00 +0000 UTC   <nil> nil nil [] [] } initialized Pending" namespace=argo workflow=input-artifact-git-7lttk
time="2020-06-17T15:20:07Z" level=info msg="Created pod: input-artifact-git-7lttk (input-artifact-git-7lttk)" namespace=argo workflow=input-artifact-git-7lttk
time="2020-06-17T15:20:07Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=4327025 workflow=input-artifact-git-7lttk
time="2020-06-17T15:20:08Z" level=info msg="Processing workflow" namespace=argo workflow=input-artifact-git-7lttk
time="2020-06-17T15:20:08Z" level=info msg="Updating node &NodeStatus{ID:input-artifact-git-7lttk,Name:input-artifact-git-7lttk,DisplayName:input-artifact-git-7lttk,Type:Pod,TemplateName:git-clone,TemplateRef:nil,Phase:Pending,BoundaryID:,Message:,StartedAt:2020-06-17 15:20:07 +0000 UTC,FinishedAt:0001-01-01 00:00:00 +0000 UTC,PodIP:,Daemoned:nil,Inputs:&Inputs{Parameters:[]Parameter{},Artifacts:[]Artifact{Artifact{Name:argo-source,Path:/src,Mode:nil,From:,ArtifactLocation:ArtifactLocation{ArchiveLogs:nil,S3:nil,Git:&GitArtifact{Repo:git@github.com:esqimo/ziglu.git,Revision:master,Depth:*1,Fetch:[],UsernameSecret:nil,PasswordSecret:nil,SSHPrivateKeySecret:&v1.SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:argo-secrets,},Key:git-ssh-private-key,Optional:nil,},InsecureIgnoreHostKey:true,},HTTP:nil,Artifactory:nil,HDFS:nil,Raw:nil,OSS:nil,GCS:nil,},GlobalName:,Archive:nil,Optional:false,},},},Outputs:nil,Children:[],OutboundNodes:[],StoredTemplateID:,WorkflowTemplateName:,TemplateScope:local/input-artifact-git-7lttk,ResourcesDuration:ResourcesDuration{},HostNodeName:,} message: PodInitializing"
time="2020-06-17T15:20:08Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=4327031 workflow=input-artifact-git-7lttk
time="2020-06-17T15:20:09Z" level=info msg="Processing workflow" namespace=argo workflow=input-artifact-git-7lttk
time="2020-06-17T15:20:09Z" level=info msg="Updating node &NodeStatus{ID:input-artifact-git-7lttk,Name:input-artifact-git-7lttk,DisplayName:input-artifact-git-7lttk,Type:Pod,TemplateName:git-clone,TemplateRef:nil,Phase:Pending,BoundaryID:,Message:PodInitializing,StartedAt:2020-06-17 15:20:07 +0000 UTC,FinishedAt:0001-01-01 00:00:00 +0000 UTC,PodIP:,Daemoned:nil,Inputs:&Inputs{Parameters:[]Parameter{},Artifacts:[]Artifact{Artifact{Name:argo-source,Path:/src,Mode:nil,From:,ArtifactLocation:ArtifactLocation{ArchiveLogs:nil,S3:nil,Git:&GitArtifact{Repo:git@github.com:esqimo/ziglu.git,Revision:master,Depth:*1,Fetch:[],UsernameSecret:nil,PasswordSecret:nil,SSHPrivateKeySecret:&v1.SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:argo-secrets,},Key:git-ssh-private-key,Optional:nil,},InsecureIgnoreHostKey:true,},HTTP:nil,Artifactory:nil,HDFS:nil,Raw:nil,OSS:nil,GCS:nil,},GlobalName:,Archive:nil,Optional:false,},},},Outputs:nil,Children:[],OutboundNodes:[],StoredTemplateID:,WorkflowTemplateName:,TemplateScope:local/input-artifact-git-7lttk,ResourcesDuration:ResourcesDuration{},HostNodeName:gke-cluster-1-default-pool-84a24b04-873f,} status Pending -> Error"
time="2020-06-17T15:20:09Z" level=info msg="Updating node &NodeStatus{ID:input-artifact-git-7lttk,Name:input-artifact-git-7lttk,DisplayName:input-artifact-git-7lttk,Type:Pod,TemplateName:git-clone,TemplateRef:nil,Phase:Error,BoundaryID:,Message:,StartedAt:2020-06-17 15:20:07 +0000 UTC,FinishedAt:0001-01-01 00:00:00 +0000 UTC,PodIP:,Daemoned:nil,Inputs:&Inputs{Parameters:[]Parameter{},Artifacts:[]Artifact{Artifact{Name:argo-source,Path:/src,Mode:nil,From:,ArtifactLocation:ArtifactLocation{ArchiveLogs:nil,S3:nil,Git:&GitArtifact{Repo:git@github.com:esqimo/ziglu.git,Revision:master,Depth:*1,Fetch:[],UsernameSecret:nil,PasswordSecret:nil,SSHPrivateKeySecret:&v1.SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:argo-secrets,},Key:git-ssh-private-key,Optional:nil,},InsecureIgnoreHostKey:true,},HTTP:nil,Artifactory:nil,HDFS:nil,Raw:nil,OSS:nil,GCS:nil,},GlobalName:,Archive:nil,Optional:false,},},},Outputs:nil,Children:[],OutboundNodes:[],StoredTemplateID:,WorkflowTemplateName:,TemplateScope:local/input-artifact-git-7lttk,ResourcesDuration:ResourcesDuration{},HostNodeName:gke-cluster-1-default-pool-84a24b04-873f,} message: failed to load artifacts: ssh: no key found"
time="2020-06-17T15:20:09Z" level=info msg="Updated phase Running -> Error" namespace=argo workflow=input-artifact-git-7lttk
time="2020-06-17T15:20:09Z" level=info msg="Updated message  -> failed to load artifacts: ssh: no key found" namespace=argo workflow=input-artifact-git-7lttk
time="2020-06-17T15:20:09Z" level=info msg="Marking workflow completed" namespace=argo workflow=input-artifact-git-7lttk
time="2020-06-17T15:20:09Z" level=info msg="Checking daemoned children of " namespace=argo workflow=input-artifact-git-7lttk
time="2020-06-17T15:20:09Z" level=info msg="Workflow update successful" namespace=argo phase=Error resourceVersion=4327038 workflow=input-artifact-git-7lttk
time="2020-06-17T15:20:10Z" level=info msg="Labeled pod argo/input-artifact-git-7lttk completed"

Stackdriver logs yield the below for the init container

"time="2020-06-17T15:20:08Z" level=fatal msg="ssh: no key found
github.com/argoproj/argo/errors.Wrap
	/go/src/github.com/argoproj/argo/errors/errors.go:88
github.com/argoproj/argo/errors.InternalWrapError
	/go/src/github.com/argoproj/argo/errors/errors.go:71
github.com/argoproj/argo/workflow/artifacts/git.(*GitArtifactDriver).Load
	/go/src/github.com/argoproj/argo/workflow/artifacts/git/git.go:36
github.com/argoproj/argo/workflow/executor.(*WorkflowExecutor).LoadArtifacts
	/go/src/github.com/argoproj/argo/workflow/executor/executor.go:165
github.com/argoproj/argo/cmd/argoexec/commands.loadArtifacts
	/go/src/github.com/argoproj/argo/cmd/argoexec/commands/init.go:34
github.com/argoproj/argo/cmd/argoexec/commands.NewInitCommand.func1
	/go/src/github.com/argoproj/argo/cmd/argoexec/commands/init.go:14
github.com/spf13/cobra.(*Command).execute
	/go/pkg/mod/github.com/spf13/cobra@v0.0.4-0.20181021141114-fe5e611709b0/command.go:766
github.com/spf13/cobra.(*Command).ExecuteC
	/go/pkg/mod/github.com/spf13/cobra@v0.0.4-0.20181021141114-fe5e611709b0/command.go:852
github.com/spf13/cobra.(*Command).Execute
	/go/pkg/mod/github.com/spf13/cobra@v0.0.4-0.20181021141114-fe5e611709b0/command.go:800
main.main
	/go/src/github.com/argoproj/argo/cmd/argoexec/main.go:17
runtime.main
	/usr/local/go/src/runtime/proc.go:203
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1357"

I have tried also using the username/password and https address to no success, when I do that it returns the message "failed to load artifacts: authentication required"

---

Message from the maintainers:

If you are impacted by this bug please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.

[alexec]

Can you confirm if this was working in older version please?

[cyclingwithelephants]

this is my first time using argo, I've not used an older version before

[alexec]

"ssh: no key found" seems to indicate that no SSH private key was specified, interestingly the field must exist - so presumably the contents of the field (and therefore the secret) in not a private key.

Can you try creating using stringData and the YAML pipe operator for multi-line sting?:

apiVersion: v1
kind: Secret
metadata:
  namespace: argo
  name: argo-secrets
type: Opaque
stringData:
  git-ssh-private-key: | 
    -----BEGIN OPENSSH PRIVATE KEY-----
    b3BlbnNzaC1rZXktdjEAAAAABG5vbmUAAAA...
    -----END OPENSSH PRIVATE KEY-----

[cyclingwithelephants]

It was in this moment I realised I gave argo the public key. This one's completely on me. Thanks for responding so quickly! :)

[alexec]

v2.10