Support for Terraform Cloud / Terraform Enterprise #458

ayshiff · 2021-04-25T22:23:14Z

Q	A
🐛 Bug fix?	no
🚀 New feature?	yes
⚠ Deprecations?	yes/no
❌ BC Break	no
🔗 Related issues	#434
❓ Documentation	yes snyk/driftctl-docs#47

Description

This PR adds support for Terraform Cloud / Terraform Enterprise.

As described in the issue the logic is as follow:

Fetch hosted-state-download-url from the API with the provided $WORKSPACE_ID (tfstate+tfcloud://$WORKSPACE_ID) and the API token through the headers (--headers 'Authorization=Bearer $API_TOKEN')
Use HTTPReader with the retrieved hosted-state-download-url

Note that the logic is the same for Terraform Cloud and Terraform Enterprise as the API is the same for both.

Here is a complete command example:

driftctl scan --from tfstate+tfcloud://$WORKSPACE_ID --headers 'Authorization=Bearer $API_TOKEN'

I also added some test the different cases we can have:

The provided workspace_id is wrong (404)
The provided authentication api_token is wrong (401)
The workspace_id and api_token are good

I have as well updated the documentation in snyk/driftctl-docs#47 to fit this new state reader.

codecov · 2021-04-25T22:39:45Z

Codecov Report

Merging #458 (ae65fb4) into v0.8 (2ef6593) will decrease coverage by 0.00%.
The diff coverage is 70.37%.

@@            Coverage Diff             @@
##             v0.8     #458      +/-   ##
==========================================
- Coverage   70.73%   70.73%   -0.01%     
==========================================
  Files         286      287       +1     
  Lines        6445     6472      +27     
==========================================
+ Hits         4559     4578      +19     
- Misses       1516     1521       +5     
- Partials      370      373       +3

Impacted Files	Coverage Δ
pkg/iac/terraform/state/backend/backend.go	`40.00% <0.00%> (-4.45%)`	⬇️
pkg/iac/terraform/state/backend/tfcloud_reader.go	`68.42% <68.42%> (ø)`
pkg/cmd/scan.go	`82.12% <100.00%> (+0.46%)`	⬆️

wbeuil · 2021-04-26T08:35:34Z

Thanks @ayshiff for your contribution. You did great on a first step to support TF Cloud. But we should converge to what they do in Terraform for our user to use the support quite the same way.

You can find all the information for the credentials part here. They rely on their .terraformrc file which can be created with terraform login which is simply:

credentials "app.terraform.io" {
  token = "XXX"
}

We should, thus, support 2 ways:

Get the token from .terraformrc (maybe in another filename as well)
Inline token from CLI (what you did but would rather write simply --token or something else)

I didn't go further into terraform package but we could re-use one of their functions/methods to fetch the credentials from the file. Maybe they even have a function/method the fetch the state with the token and the workspace_id.

sundowndev

Great job! 👍🏽

Left few comments about error handling and tests. We also might discuss a bit about authentication, as there are multiple ways to authenticate to TFC. We usually don't perform integration tests for backends, so we have to discuss about that too.

pkg/iac/terraform/state/backend/cloud_reader.go

pkg/iac/terraform/state/backend/cloud_reader_test.go

ayshiff · 2021-04-27T12:12:17Z

I fixed the various issues that came up and added a new --terraform-cloud-token flag to provide the Terraform Cloud API token.

I still have to work on the way to retrieve the token from .terraformrc as I didn't have the time to look at it in detail.

eliecharra · 2021-04-27T12:15:00Z

I still have to work on the way to retrieve the token from .terraformrc as I didn't have the time to look at it in detail.

@wbeuil @sundowndev Maybe this could be splitted in another PR ? I'm fine with that

sundowndev · 2021-04-27T12:22:59Z

I still have to work on the way to retrieve the token from .terraformrc as I didn't have the time to look at it in detail.

@wbeuil @sundowndev Maybe this could be splitted in another PR ? I'm fine with that

Well I don't really know much about how credentials management work in TFC and what it implies to support the .terraformrc file, but yeah we could introduce this feature with the flag only for now.

wbeuil · 2021-04-27T12:36:20Z

We could indeed split this PR into two.

Just for the record, there is this function which seems to read and load everything there is inside a config file (.terraformrc in linux, terraform.rc in windows).

sundowndev

This seems great, I just left some comments about the code

pkg/cmd/scan.go

pkg/iac/terraform/state/backend/cloud_reader.go

pkg/cmd/scan_test.go

pkg/iac/terraform/state/backend/backend.go

wbeuil

Small comments to be fully consistent with the new wording. Can you also update your branch to have the latest changes from @sundowndev ? Thanks !

wbeuil · 2021-05-03T09:25:38Z

pkg/iac/terraform/state/backend/backend.go

 }

 type Backend io.ReadCloser

 type Options struct {
-	Headers map[string]string
+	Headers             map[string]string
+	TerraformCloudToken string


Since we reworded everything with TFCloud, I would suggest to do it everywhere to be consistent. Thus, let's call it TFCloudToken.

wbeuil · 2021-05-03T09:25:59Z

pkg/iac/terraform/state/backend/tfcloud_reader.go

+)
+
+const BackendKeyTFCloud = "tfcloud"
+const TerraformCloudAPI = "https://app.terraform.io/api/v2"


TFCloudAPI

wbeuil

LGTM !

eliecharra · 2021-05-03T15:57:29Z

Hey @all-contributors please add @ayshiff for code

allcontributors · 2021-05-03T15:57:37Z

@eliecharra

I've put up a pull request to add @ayshiff! 🎉

ayshiff requested a review from a team as a code owner April 25, 2021 22:23

ayshiff mentioned this pull request Apr 25, 2021

Support for Terraform Cloud / Terraform Enterprise snyk/driftctl-docs#47

Merged

sundowndev suggested changes Apr 26, 2021

View reviewed changes

sundowndev added kind/enhancement New feature or improvement priority/1 labels Apr 27, 2021

sundowndev added this to Review in driftctl via automation Apr 27, 2021

sundowndev added this to the v0.8.0 milestone Apr 27, 2021

sundowndev suggested changes Apr 27, 2021

View reviewed changes

ayshiff changed the base branch from main to v0.8 April 28, 2021 10:34

ayshiff force-pushed the feature/tfcloud branch from a501e01 to d964347 Compare April 28, 2021 11:41

sundowndev reviewed Apr 30, 2021

View reviewed changes

pkg/iac/terraform/state/backend/backend.go Outdated Show resolved Hide resolved

sundowndev mentioned this pull request May 3, 2021

[v0.8.0] documentation updates snyk/driftctl-docs#52

Closed

5 tasks

wbeuil suggested changes May 3, 2021

View reviewed changes

ayshiff added 9 commits May 3, 2021 12:08

feat(schemes): add tfcloud

e0a295e

feat(state): add cloud reader

55a84a2

fix(cloud_reader): make tests pass

5dbf569

fix(scan_test): add tfcloud

3d52107

fix(cloud_reader): update tests + tfcloud token flag

73df960

fix(cloud_reader): requested changes

05dd544

fix(tfcloud): rename tfcloud -> tfc

c40896b

fix(tfcloud): rename tfcloud -> tfc

8d65adb

refactor(tfcloud): rebase

ae65fb4

ayshiff force-pushed the feature/tfcloud branch from a300e2c to ae65fb4 Compare May 3, 2021 12:02

ayshiff requested a review from sundowndev May 3, 2021 13:18

sundowndev approved these changes May 3, 2021

View reviewed changes

ayshiff requested a review from wbeuil May 3, 2021 13:32

wbeuil approved these changes May 3, 2021

View reviewed changes

sundowndev merged commit b64276a into snyk:v0.8 May 3, 2021

driftctl automation moved this from Review to Done May 3, 2021

allcontributors bot mentioned this pull request May 3, 2021

docs: add ayshiff as a contributor #479

Merged

This was referenced May 3, 2021

merge v0.8.0 into main #477

Closed

merge v0.8.0 into main #503

Merged

sundowndev mentioned this pull request Jul 1, 2021

Use an existing Terraform Cloud configuration #780

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support for Terraform Cloud / Terraform Enterprise #458

Support for Terraform Cloud / Terraform Enterprise #458

ayshiff commented Apr 25, 2021 •

edited

codecov bot commented Apr 25, 2021 •

edited

wbeuil commented Apr 26, 2021

sundowndev left a comment •

edited

ayshiff commented Apr 27, 2021

eliecharra commented Apr 27, 2021

sundowndev commented Apr 27, 2021 •

edited

wbeuil commented Apr 27, 2021 •

edited by sundowndev

sundowndev left a comment

wbeuil left a comment

wbeuil May 3, 2021

wbeuil May 3, 2021

wbeuil left a comment

eliecharra commented May 3, 2021

allcontributors bot commented May 3, 2021

Support for Terraform Cloud / Terraform Enterprise #458

Support for Terraform Cloud / Terraform Enterprise #458

Conversation

ayshiff commented Apr 25, 2021 • edited

Description

codecov bot commented Apr 25, 2021 • edited

Codecov Report

wbeuil commented Apr 26, 2021

sundowndev left a comment • edited

Choose a reason for hiding this comment

ayshiff commented Apr 27, 2021

eliecharra commented Apr 27, 2021

sundowndev commented Apr 27, 2021 • edited

wbeuil commented Apr 27, 2021 • edited by sundowndev

sundowndev left a comment

Choose a reason for hiding this comment

wbeuil left a comment

Choose a reason for hiding this comment

wbeuil May 3, 2021

Choose a reason for hiding this comment

wbeuil May 3, 2021

Choose a reason for hiding this comment

wbeuil left a comment

Choose a reason for hiding this comment

eliecharra commented May 3, 2021

allcontributors bot commented May 3, 2021

ayshiff commented Apr 25, 2021 •

edited

codecov bot commented Apr 25, 2021 •

edited

sundowndev left a comment •

edited

sundowndev commented Apr 27, 2021 •

edited

wbeuil commented Apr 27, 2021 •

edited by sundowndev