Feat: Support SSH for retrieving terraform modules in private git repo #5059

motilayo · 2022-11-14T04:35:17Z

Support SSH for retrieving terraform modules in private git repo

Enable support for terraform modules in private git repo through SSH. A kubernetes secret which holds the ssh private-key and known_hosts should be created and referenced in the component definition.

Terraform controller PR: kubevela/terraform-controller#349

Feature for kubevela/terraform-controller#292

I have:

Read and followed KubeVela's contribution process.
Related Docs updated properly. In a new feature or configuration option, an update to the documentation is necessary.
Run make reviewable to ensure this PR is ready for review.
Added backport release-x.y labels to auto-backport this PR if necessary.

How has this code been tested

Tested in a local cluster.

Create a kubernetes secret of type kubernetes.io/ssh-auth for the git ssh credentials & known-hosts

apiVersion: v1
kind: Secret
metadata:
  name: git-ssh
  namespace: vela-system
type: kubernetes.io/ssh-auth
stringData:
  ssh-privatekey: |
   <SSH Private Key> # the ssh private key used for authenticating git
  known_hosts: |
   <SSH known_hosts>  # use `ssh-keyscan github.com`  to generate known_hosts

Create a component definition

apiVersion: core.oam.dev/v1beta1
kind: ComponentDefinition
metadata:
  annotations:
    definition.oam.dev/description: Terraform module to provision a KMS key with alias
  labels:
    type: terraform
  name: aws-kms-key
  namespace: vela-system
spec:
  schematic:
    terraform:
      configuration: git@github.com:motilayo/terraform-aws-kms-key.git # use terraform configuration in private git repo to test
      gitCredentialsSecretReference: # reference to the secret created earlier
        name: git-ssh # secret name
        namespace: vela-system  # secret namespace
      type: remote

Create an application and verify

Special notes for your reviewer

codecov · 2022-11-14T04:41:49Z

Codecov Report

Base: 49.60% // Head: 61.08% // Increases project coverage by +11.47% 🎉

Coverage data is based on head (8785823) compared to base (c7c6009).
Patch coverage: 71.42% of modified lines in pull request are covered.

Additional details and impacted files

@@             Coverage Diff             @@
##           master    #5059       +/-   ##
===========================================
+ Coverage   49.60%   61.08%   +11.47%     
===========================================
  Files         304      305        +1     
  Lines       45429    45499       +70     
===========================================
+ Hits        22537    27792     +5255     
+ Misses      20544    14846     -5698     
- Partials     2348     2861      +513

Flag	Coverage Δ
apiserver-e2etests	`35.05% <0.00%> (?)`
apiserver-unittests	`36.84% <ø> (-0.05%)`	⬇️
core-unittests	`55.12% <62.50%> (-0.02%)`	⬇️
e2e-multicluster-test	`18.87% <14.28%> (-0.03%)`	⬇️
e2e-rollout-tests	`20.53% <16.07%> (+0.01%)`	⬆️
e2etests	`25.91% <16.07%> (-0.12%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
pkg/controller/utils/capability.go	`80.64% <68.62%> (-0.57%)`	⬇️
pkg/appfile/appfile.go	`74.31% <100.00%> (+0.16%)`	⬆️
...ponentdefinition/componentdefinition_controller.go	`91.35% <100.00%> (ø)`
pkg/addon/registry.go	`38.65% <0.00%> (-5.05%)`	⬇️
pkg/cue/definition/template.go	`64.16% <0.00%> (-3.47%)`	⬇️
...troller/core.oam.dev/v1alpha2/application/apply.go	`87.84% <0.00%> (-2.09%)`	⬇️
pkg/oam/util/helper.go	`61.91% <0.00%> (-0.53%)`	⬇️
pkg/apiserver/utils/container/container.go	`83.33% <0.00%> (ø)`
pkg/velaql/providers/query/tree.go	`85.27% <0.00%> (+0.26%)`	⬆️
pkg/multicluster/cluster_management.go	`36.70% <0.00%> (+0.26%)`	⬆️
... and 113 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

Signed-off-by: motilayo <44736801+motilayo@users.noreply.github.com>

Signed-off-by: motilayo <joshuaagboola@live.ca>

…date secret, improve known_hosts logic Signed-off-by: motilayo <joshuaagboola@live.ca>

Signed-off-by: motilayo <joshuaagboola@live.ca>

motilayo · 2022-12-23T01:51:25Z

@wonderflow Please review when you get a chance. I've added some tests and rebased my branch with the master branch.

go.mod

Somefive

LGTM after nit.

go.mod

pkg/controller/utils/capability.go

wonderflow · 2022-12-23T02:32:44Z

pkg/appfile/appfile.go

@@ -724,6 +724,10 @@ func generateTerraformConfigurationWorkload(wl *Workload, ns string) (*unstructu
 		configuration.Spec.ProviderReference = wl.FullTemplate.ComponentDefinition.Spec.Schematic.Terraform.ProviderReference
 	}

+	if configuration.Spec.GitCredentialsSecretReference == nil {
+		configuration.Spec.GitCredentialsSecretReference = wl.FullTemplate.ComponentDefinition.Spec.Schematic.Terraform.GitCredentialsSecretReference


what if the value is still nil?

Then it's ok to have no credentials. Like when it is a public repo.

wonderflow

great job, generally LGTM!

pkg/appfile/appfile.go

pkg/controller/utils/capability.go

pkg/controller/utils/capability_test.go

Signed-off-by: motilayo <joshuaagboola@live.ca>

chivalryq

LGTM! Please fix tests.

pkg/controller/utils/capability_test.go

Signed-off-by: motilayo <joshuaagboola@live.ca>

kubevela#5059) * Feat: Support SSH for retrieving terraform modules in private git repo Signed-off-by: motilayo <44736801+motilayo@users.noreply.github.com> * fix lint errors Signed-off-by: motilayo <44736801+motilayo@users.noreply.github.com> * fix gofmt lint error Signed-off-by: motilayo <joshuaagboola@live.ca> * update gitCredentialsReference to gitCredentialsSecretReference, validate secret, improve known_hosts logic Signed-off-by: motilayo <joshuaagboola@live.ca> * SImplify logic to get publickey Signed-off-by: motilayo <joshuaagboola@live.ca> * Set known_hosts and export SSH_KNOWN_HOSTS Signed-off-by: motilayo <joshuaagboola@live.ca> * Call GetGitSSHPublicKey for cli Signed-off-by: motilayo <joshuaagboola@live.ca> * fix parser.go - nil check for ref.Client Signed-off-by: motilayo <joshuaagboola@live.ca> * Unit test for GetGitSSHPublicKey Signed-off-by: motilayo <joshuaagboola@live.ca> * improve test coverage Signed-off-by: motilayo <joshuaagboola@live.ca> * make reviewable Signed-off-by: motilayo <joshuaagboola@live.ca> * minor improvements & cleanup Signed-off-by: motilayo <joshuaagboola@live.ca> * update secret name in test Signed-off-by: motilayo <joshuaagboola@live.ca> Signed-off-by: motilayo <44736801+motilayo@users.noreply.github.com> Signed-off-by: motilayo <joshuaagboola@live.ca>

motilayo requested review from Somefive, FogDong, barnettZQG, wonderflow, leejanee and jefree-cat as code owners November 14, 2022 04:35

motilayo mentioned this pull request Nov 14, 2022

Feat: Support retrieving modules in private git repo through SSH kubevela/terraform-controller#349

Merged

motilayo force-pushed the master branch from 9601747 to 872ce13 Compare November 14, 2022 23:27

motilayo force-pushed the master branch 2 times, most recently from 0667c40 to bcd07a8 Compare December 17, 2022 21:07

motilayo force-pushed the master branch from a406e3f to 36570f1 Compare December 20, 2022 01:37

motilayo and others added 9 commits December 21, 2022 22:50

Feat: Support SSH for retrieving terraform modules in private git repo

19ff7d9

Signed-off-by: motilayo <44736801+motilayo@users.noreply.github.com>

fix lint errors

c58cda8

Signed-off-by: motilayo <44736801+motilayo@users.noreply.github.com>

fix gofmt lint error

4c43f5b

Signed-off-by: motilayo <joshuaagboola@live.ca>

update gitCredentialsReference to gitCredentialsSecretReference, vali…

86f47b9

…date secret, improve known_hosts logic Signed-off-by: motilayo <joshuaagboola@live.ca>

SImplify logic to get publickey

970c4d7

Signed-off-by: motilayo <joshuaagboola@live.ca>

Set known_hosts and export SSH_KNOWN_HOSTS

ee7e2d0

Signed-off-by: motilayo <joshuaagboola@live.ca>

Call GetGitSSHPublicKey for cli

bb1d68f

Signed-off-by: motilayo <joshuaagboola@live.ca>

fix parser.go - nil check for ref.Client

2a3309e

Signed-off-by: motilayo <joshuaagboola@live.ca>

Unit test for GetGitSSHPublicKey

827e9cb

Signed-off-by: motilayo <joshuaagboola@live.ca>

motilayo force-pushed the master branch from 36570f1 to 827e9cb Compare December 22, 2022 03:51

motilayo added 2 commits December 22, 2022 14:04

improve test coverage

6eba05e

Signed-off-by: motilayo <joshuaagboola@live.ca>

make reviewable

639b649

Signed-off-by: motilayo <joshuaagboola@live.ca>

wonderflow reviewed Dec 23, 2022

View reviewed changes

go.mod Outdated Show resolved Hide resolved

Somefive reviewed Dec 23, 2022

View reviewed changes

go.mod Outdated Show resolved Hide resolved

pkg/controller/utils/capability.go Outdated Show resolved Hide resolved

pkg/controller/utils/capability.go Outdated Show resolved Hide resolved

wonderflow reviewed Dec 23, 2022

View reviewed changes

wonderflow approved these changes Dec 23, 2022

View reviewed changes

chivalryq reviewed Dec 23, 2022

View reviewed changes

pkg/appfile/appfile.go Show resolved Hide resolved

chivalryq reviewed Dec 23, 2022

View reviewed changes

pkg/controller/utils/capability.go Outdated Show resolved Hide resolved

chivalryq reviewed Dec 23, 2022

View reviewed changes

pkg/controller/utils/capability.go Show resolved Hide resolved

pkg/controller/utils/capability_test.go Outdated Show resolved Hide resolved

minor improvements & cleanup

1292126

Signed-off-by: motilayo <joshuaagboola@live.ca>

chivalryq reviewed Dec 23, 2022

View reviewed changes

pkg/controller/utils/capability_test.go Show resolved Hide resolved

update secret name in test

8785823

Signed-off-by: motilayo <joshuaagboola@live.ca>

wonderflow merged commit a9bc43a into kubevela:master Dec 23, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feat: Support SSH for retrieving terraform modules in private git repo #5059

Feat: Support SSH for retrieving terraform modules in private git repo #5059

motilayo commented Nov 14, 2022 •

edited

Loading

codecov bot commented Nov 14, 2022 •

edited

Loading

motilayo commented Dec 23, 2022

Somefive left a comment

wonderflow Dec 23, 2022

chivalryq Dec 23, 2022

wonderflow left a comment

chivalryq left a comment •

edited

Loading

Feat: Support SSH for retrieving terraform modules in private git repo #5059

Feat: Support SSH for retrieving terraform modules in private git repo #5059

Conversation

motilayo commented Nov 14, 2022 • edited Loading

Support SSH for retrieving terraform modules in private git repo

How has this code been tested

Special notes for your reviewer

codecov bot commented Nov 14, 2022 • edited Loading

Codecov Report

motilayo commented Dec 23, 2022

Somefive left a comment

Choose a reason for hiding this comment

wonderflow Dec 23, 2022

Choose a reason for hiding this comment

chivalryq Dec 23, 2022

Choose a reason for hiding this comment

wonderflow left a comment

Choose a reason for hiding this comment

chivalryq left a comment • edited Loading

Choose a reason for hiding this comment

motilayo commented Nov 14, 2022 •

edited

Loading

codecov bot commented Nov 14, 2022 •

edited

Loading

chivalryq left a comment •

edited

Loading