SSH support for submodule

[jpmorin]

SSH Support for submodule

When submodules url are SSH (or relative to a main SSH url), it was only possible to clone them if the main repository was also cloned via SSH and with the same private_key. Otherwise it was not possible to clone the submodule via SSH.

We use relative url for all our submodules to allows HTTPS and SSH cloning. Our Concourse instance works with SSH and every repository has it's how private_key.

I expended upon the existing submodule_credentials parameter by adding a second optionnal structure for ssh private keys:

• submodule_credentials: Optional. List of credentials for HTTP(s) or SSH auth when pulling git submodules which are not stored in the same git server as the container repository.
• http(s) credentials
  • host : The host to connect too. Note that host is specified with no protocol extensions.
  • username : Username for HTTP(S) auth when pulling submodule.
  • password : Password for HTTP(S) auth when pulling submodule.
• ssh credentials
  • url : Submodule url, as specified in the .gitmodule file. Support full or relative ssh url.
  • private_key : Private key for SSH auth when pulling submodule.
  • private_key_passphrase : Optional. To unlock private_key if it is protected by a passphrase.
• exemples:

  submodule_credentials:
    # http(s) credentials
  - host: github.com
    username: git-user
    password: git-password
    # ssh credentials
  - url: git@github.com:org-name/repo-name.git
    private_key: |
      -----BEGIN RSA PRIVATE KEY-----
      MIIEowIBAAKCAQEAtCS10/f7W7lkQaSgD/mVeaSOvSF9ql4hf/zfMwfVGgHWjj+W
      <Lots more text>
      DWiJL+OFeg9kawcUL6hQ8JeXPhlImG6RTUffma9+iGQyyBMCGd1l
      -----END RSA PRIVATE KEY-----
    private_key_passphrase: ssh-passphrase # (optionnal)
    # ssh credentials with relative url
  - url: ../org-name/repo-name.git
    private_key: |
      -----BEGIN RSA PRIVATE KEY-----
      MIIEowIBAAKCAQEAtCS10/f7W7lkQaSgD/mVeaSOvSF9ql4hf/zfMwfVGgHWjj+W
      <Lots more text>
      DWiJL+OFeg9kawcUL6hQ8JeXPhlImG6RTUffma9+iGQyyBMCGd1l
      -----END RSA PRIVATE KEY-----
    private_key_passphrase: ssh-passphrase # (optionnal)

Since adding multiple identities to an ssh-agent does not work with git, I implemented a retry mecanism where a short-lived ssh-agent is created and used to retry the git submodule update ... command. The short-lived ssh-agent processes are killed after use to ensure the ressource terminate correctly.

Since adding multiple identities to an ssh-agent does not work with git, I implemented a mecanism where a single ssh-agent is re-initialised with a single identity for every submodule clone via ssh. Once the submodule has been cloned, the ssh-agent is again re-initialised with the main repo identity (if it exist).

Linked issues

• Resolves private_key for submodule_credentials #357
• Resolves Submodule with different credentials #270
• Resolves Failure when cloning submodule #149

[jpmorin]

I have published a docker image of this pull-request on dockerhub : jpmorin/git-resource. It is already use it in our pipelines.

Looking forward for review / feedback.

Thank you!

[luis-nagaki]

Will this be getting any attention soon? I have a big blocker on this and I tried to use your PR @jpmorin but not sure if its working for me

[taylorsilva]

Sorry for the insane delay here, but I'm going to take a look at this and will merge if it's all good.

[taylorsilva]

LGTM - I rebased your branch locally against master since I just merged some other PR's in. I ran the tests locally and everything passed.

Thanks again for the PR!