Feat: Support retrieving modules in private git repo through SSH

[motilayo]

This is a PR to Support SSH for retrieving terraform modules in private git repo for this issue: #292
There is a corresponding PR in kubevela repo: kubevela/kubevela#5059

This change require a secret which hold the SSH private key and known hosts for the git repo:
known_hosts can be generated using ssh-keyscan <git-url>

apiVersion: v1
kind: Secret
metadata:
  name: git-ssh
  namespace: vela-system
type: kubernetes.io/ssh-auth
stringData:
  ssh-privatekey: |
   <SSH Private Key> # the ssh private key used for authenticating git
  known_hosts: |
   <SSH known_hosts>  # use `ssh-keyscan github.com`  to generate known_hosts

[codecov]

Codecov Report

Base: 79.31% // Head: 79.19% // Decreases project coverage by -0.12% ⚠️

Coverage data is based on head (7e1eeb5) compared to base (3a96f68).
Patch coverage: 76.47% of modified lines in pull request are covered.

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #349      +/-   ##
==========================================
- Coverage   79.31%   79.19%   -0.13%     
==========================================
  Files          23       23              
  Lines        1750     1807      +57     
==========================================
+ Hits         1388     1431      +43     
- Misses        278      290      +12     
- Partials       84       86       +2     

Flag	Coverage Δ
e2e	?
unit	79.19% <76.47%> (-0.13%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
controllers/configuration_controller.go	77.24% <76.47%> (-0.12%)	⬇️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

[chivalryq]

Let's add some e2e test. Maybe this repo will helps: https://github.com/isomorphic-git/git-http-mock-server
Also please fix the CI.

[motilayo]

Let's add some e2e test. Maybe this repo will helps: https://github.com/isomorphic-git/git-http-mock-server Also please fix the CI.

Couldn't get https://github.com/isomorphic-git/git-http-mock-server to work. Looked into different options, did not find anything lightweight to use. Decided to go with creating a private git server in a pod, with a terraform module repo in that git.
e2e test output:

❯ go test -v ./e2e/normal/... -run TestGitCredentialsSecretReference
=== RUN   TestGitCredentialsSecretReference
I1129 12:05:53.773202    6759 configuration_test.go:71] random-e2e-git-creds-secret-ref test begins……
I1129 12:05:53.774256    6759 configuration_test.go:94] 1. Applying Configuration
I1129 12:05:53.925994    6759 configuration_test.go:311] - Checking git-server pod status
I1129 12:05:53.937536    6759 configuration_test.go:319] - pod=git-server Ready=True
I1129 12:05:54.467514    6759 configuration_test.go:102] 2. Checking Configuration status
I1129 12:06:40.326000    6759 configuration_test.go:125] 3. Checking the status of Configs and Secrets
I1129 12:06:40.326021    6759 configuration_test.go:127] - Checking ConfigMap which stores .tf
I1129 12:06:40.328671    6759 configuration_test.go:132] - Checking Secret which stores Backend
I1129 12:06:40.331256    6759 configuration_test.go:137] - Checking Secret which stores outputs
I1129 12:06:40.333018    6759 configuration_test.go:141] - Checking Secret which stores variables
I1129 12:06:40.334731    6759 configuration_test.go:145] 4. Deleting Configuration
I1129 12:06:40.336358    6759 configuration_test.go:150] 5. Checking Configuration is deleted
I1129 12:06:55.744077    6759 configuration_test.go:182] 6. Checking Secrets and ConfigMap which should all be deleted
I1129 12:06:55.749020    6759 configuration_test.go:198] random-e2e-git-creds-secret-ref test ends……
--- PASS: TestGitCredentialsSecretReference (62.08s)
PASS

[chivalryq]

@motilayo You did a very great job! The test methods is impressive. And can you explain some of the files which is not used in test?
Don't worry about the code coverage CI item. It seems not to calculate the e2e test correctly.

[motilayo]

Those files in examples/git-credentials are prereqs for support git ssh credentials test. The git-server which requires ssh auth for git push & pull. The git-push job which will push the terraform module to a repo on the git server pod, the secret with the ssh public key & private key. The public key is added to authorized_keys file on the git server, the private key is added to the git-push for the initial repo push. Then a generated secret which holds the private key & known_hosts - this secret is mounted to the terraform apply/destroy job so that it can pull the terraform module from a private repo.
Best effort to keep everything self contained for e2e test.

[motilayo]

Hi @chivalryq & @wonderflow , hoping for a review. I've made the suggested changes and added some tests. Let me know if there's any pending tasks for me to do.
Thanks!

[chivalryq]

LGTM after some code cleanup

[chivalryq]

@motilayo Great Job! You can fix the conflicts within PR in vela's repo and we try to merge it into master.