Unable to clone the github repo

[skdubey]

Hi I am very new to Concourse and trying my hands dirty on this, but has stuck at very first level.

What am I doing?
I am trying to pull a private repository from github.

My config files -

main.yml

---
resources:
- name: pullcode
  type: git
  source:
    uri: git@github.com:user/test.git
    branch: master
    username: git
    password: xyz
    private_key: ((keys.private_key))   

jobs:
- name: buildcode
  plan:
  - get: pullcode
    trigger: false
  - task: create-image
    file: build-image.yml

private.yml -

keys:
    private_key: |
      -----BEGIN RSA PRIVATE KEY-----
      xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
      -----END RSA PRIVATE KEY-----

Command: .\fly.exe -t devops set-pipeline --pipeline hello -c .\main.yml -l .\private.yml

git pull is working locally for me but not when I try to un my pipeline in Concourse. Getting error below -

_selected worker: ip-10-30-0-27
Identity added: /tmp/git-resource-private-key (/tmp/git-resource-private-key)
Cloning into '/tmp/git-resource-repo-cache'...
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists._

Not sure what is missing.

If I try with url (uri: https://github.com/user/test.git), getting error below -
selected worker: ip-10-30-0-27
Identity added: /tmp/git-resource-private-key (/tmp/git-resource-private-key)
Cloning into '/tmp/git-resource-repo-cache'...
fatal: unable to access 'https://github.com/user/test.git/': Could not resolve host: github.com

System details -

1. self hosted concouser on EC2 instance (Linux)
2. Running command from Windows10 using Powershell
3. Concourse version: 7.0.0
4. Fly.exe version: 7.0.0

[crsimmons]

What you have looks mostly correct. Two things:

1. You don't need both username/password and private_key in the git resource source. Use the former with uri: https://foo.git and the latter with uri: git@github.com/foo.git.
2. Whitespace matters a lot with YAML especially in the vars file. It's hard to tell with the formatting in your post if you got it right. I would expect the following to work:

---
# main.yml
resources:
- name: pullcode
  type: git
  source:
    uri: git@github.com:user/test.git
    branch: master
    private_key: ((keys.private_key))

jobs:
- name: buildcode
  plan:
  - get: pullcode
    trigger: false
  - task: create-image
    file: build-image.yml
---
# private.yml
keys:
  private_key: |
    -----BEGIN RSA PRIVATE KEY-----
    xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    -----END RSA PRIVATE KEY-----

[skdubey]

Thanks @crsimmons for the reply.
I validated with same config, another repository from with my one of the colleague, it's throwing the same error.
It seem that the setup I have done on an EC2 instance has some issue but not able to figure it out what.

[evanchaoli]

You may check your worker configuration. I hit once this error in my dev environment, where I badly set CONCOURSE_RUNTIME: containerd but CONCOURSE_GARDEN_DNS_PROXY_ENABLE: "true".

[evanchaoli]

As you set runtime to containerd, you should set export CONCOURSE_CONTAINERD_DNS_PROXY_ENABLE=true. Can you try that?

[skdubey]

I have updated the config as below -

#!/bin/bash
export CONCOURSE_WORK_DIR=/var/lib/concourse
export CONCOURSE_TSA_WORKER_PRIVATE_KEY=/etc/concourse/worker_key
export CONCOURSE_TSA_PUBLIC_KEY=/etc/concourse/tsa_host_key.pub
export CONCOURSE_TSA_HOST=127.0.0.1:2222
export CONCOURSE_RUNTIME=containerd
#export CONCOURSE_GARDEN_DNS_PROXY_ENABLE=true
export CONCOURSE_CONTAINERD_DNS_PROXY_ENABLE=true_

This time I am getting error in message -

root@ip-10-30-0-27:/etc/concourse# source worker_environment
root@ip-10-30-0-27:/etc/concourse# /usr/local/concourse/bin/concourse worker &
[1] 886728
root@ip-10-30-0-27:/etc/concourse# {"timestamp":"2021-04-01T08:59:52.637386598Z","level":"info","source":"baggageclaim","message":"baggageclaim.using-driver","data":{"driver":"btrfs"}}
{"timestamp":"2021-04-01T08:59:52.653341510Z","level":"info","source":"baggageclaim","message":"baggageclaim.listening","data":{"addr":"127.0.0.1:7788"}}
{"timestamp":"2021-04-01T08:59:52.666590591Z","level":"error","source":"worker","message":"worker.garden-runner.logging-runner-exited","data":{"error":"Exit trace for group:\ncontainerd-garden-backend exited with error: client init: failed to connect to addr /run/containerd/containerd.sock: failed to dial \"/run/containerd/containerd.sock\": connection error: desc = \"transport: error while dialing: dial unix /run/containerd/containerd.sock: connect: connection refused\"\ncontainerd exited with error: signal: interrupt\n","session":"8"}}
{"timestamp":"2021-04-01T08:59:52.666717851Z","level":"info","source":"worker","message":"worker.debug-runner.logging-runner-exited","data":{"session":"10"}}
{"timestamp":"2021-04-01T08:59:52.666776490Z","level":"info","source":"worker","message":"worker.healthcheck-runner.logging-runner-exited","data":{"session":"11"}}
{"timestamp":"2021-04-01T08:59:52.666816877Z","level":"info","source":"worker","message":"worker.container-sweeper.sweep-cancelled-by-signal","data":{"session":"6","signal":2}}
{"timestamp":"2021-04-01T08:59:52.666872233Z","level":"info","source":"worker","message":"worker.container-sweeper.logging-runner-exited","data":{"session":"13"}}
{"timestamp":"2021-04-01T08:59:52.666908906Z","level":"info","source":"worker","message":"worker.volume-sweeper.sweep-cancelled-by-signal","data":{"session":"7","signal":2}}
{"timestamp":"2021-04-01T08:59:52.666936482Z","level":"info","source":"worker","message":"worker.volume-sweeper.logging-runner-exited","data":{"session":"14"}}
{"timestamp":"2021-04-01T08:59:52.667003305Z","level":"info","source":"worker","message":"worker.baggageclaim-runner.logging-runner-exited","data":{"session":"9"}}
{"timestamp":"2021-04-01T08:59:52.688863879Z","level":"error","source":"worker","message":"worker.beacon-runner.beacon.forward-conn.failed-to-dial","data":{"addr":"127.0.0.1:7777","error":"dial tcp 127.0.0.1:7777: connect: connection refused","network":"tcp","session":"4.1.4"}}
{"timestamp":"2021-04-01T08:59:53.689095451Z","level":"info","source":"worker","message":"worker.beacon-runner.beacon.forward-conn.retrying","data":{"addr":"127.0.0.1:7777","network":"tcp","session":"4.1.4"}}
{"timestamp":"2021-04-01T08:59:53.689314055Z","level":"error","source":"worker","message":"worker.beacon-runner.beacon.forward-conn.failed-to-dial","data":{"addr":"127.0.0.1:7777","error":"dial tcp 127.0.0.1:7777: connect: connection refused","network":"tcp","session":"4.1.4"}}_

Should I replace this line "CONCOURSE_TSA_HOST=127.0.0.1:2222" with the an internal ip of the instance?

[taylorsilva]

CONCOURSE_TSA_HOST should point to the IP or hostname of where ever concourse web is running. What you have will only work if the web and worker are on the same machine.

[skdubey]

Yes, concourse web is running.
I have singe EC2 instance so web + worker both are on same system.

[taylorsilva]

Ah I see, I think I missed seeing this error: failed to connect to addr /run/containerd/containerd.sock and containerd exited with error: signal: interrupt

It looks like the containerd daemon is failing to come up, so the client is being sent an interrupt signal, which is the expected behaviour. So the real question is why is the containerd daemon failing to start? Wonder why no errors are being printed if it's failing to start?