Comply with the Google Cloud impersonate library and implement integration test #2550

yamadayutaka · 2023-05-02T08:27:17Z

Description

Fixes #1997.
Fixes #2477.

Based on #2052.

Change the scope to ScopeFullControl for label updates. (refer to here)
Implement test for the following scenario.

Test senario

Preparation

Prepare the following two service accounts.
Account A: Terragrunt test runner account. This requires storage administrator privileges.
Account B: Impersonation test runner account. This requires token creation privileges without storage admin privileges.
Set the following environment variables.

Name	Value	Note
GOOGLE_APPLICATION_CREDENTIALS	path to service account key json of `Account A`	(Optional) Not required in either case - User credentials set up by using the Google Cloud CLI - The attached service account, returned by the metadata server
GCLOUD_SERVICE_KEY_IMPERSONATOR	service account key json content of `Account B`
GOOGLE_IDENTITY_EMAIL	e-mail address of `Account A`

Execution

Change runner account to Account B.
Run terragrunt without impersonation.
Confirm failure to create backend bucket.
Run terragrunt with impersonation. (impersonate account A)
Confirm successful to create backend bucket.

Result

https://gist.github.com/yamadayutaka/bf774ecf249ac17d08f41a5e397d2e85

TODOs

Read the Gruntwork contribution guidelines.

Run the relevant tests successfully, including pre-commit checks.
Include release notes. If this PR is backward incompatible, include a migration guide.

Release Notes (draft)

Comply with the Google Cloud impersonate library.

denis256 · 2023-05-02T12:46:36Z

test/integration_test.go

+	remoteStateConfig := remote.RemoteStateConfigGCS{Bucket: bucketName}
+
+	gcsClient, err := remote.CreateGCSClient(remoteStateConfig)
+	if err != nil {


I was wondering why can't be used assert.NoError()

assert.NoError(t, err)

I made it the same as other implementations such as here.

denis256 · 2023-05-02T12:53:12Z

remote/remote_state_gcs.go

-		opts = append(opts, option.ImpersonateCredentials(
-			gcsConfigRemote.ImpersonateServiceAccount,
-			gcsConfigRemote.ImpersonateServiceAccountDelegates...))
+		ts, err := impersonate.CredentialsTokenSource(context.Background(), impersonate.CredentialsConfig{


Why can't be used existing context from ctx variable?

Modified to use ctx variable.

denis256 · 2023-05-02T12:54:37Z

Noticed failed integration tests:

TestTerragruntWorksWithImpersonateGCSBackend

You can apply this plan to save these new output values to the Terraform
state, without changing any real infrastructure.
�[0m�[1m�[32m
Apply complete! Resources: 0 added, 0 changed, 0 destroyed.
�[0m�[0m�[1m�[32m
Outputs:

�[0mrendered_template = "Hello, I am a template. My sample_var value = hello"
    integration_test.go:4298: 
runTerragruntVersionCommand after split
[terragrunt apply -auto-approve --terragrunt-non-interactive --terragrunt-config /tmp/terragrunt-test124429562/terragrunt.hcl --terragrunt-working-dir fixture-gcs-impersonate/]
    integration_test.go:3864: Failed to run Terragrunt command 'terragrunt apply -auto-approve --terragrunt-non-interactive --terragrunt-config /tmp/terragrunt-test124429562/terragrunt.hcl --terragrunt-working-dir fixture-gcs-impersonate/' due to error: google: could not find default credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.
        
        Stdout: (see log output above)
        
        Stderr: (see log output above)
    integration_test.go:4350: Deleting test GCS bucket terragrunt-test-bucket-rqxgp2

yamadayutaka · 2023-05-04T08:35:00Z

Thank you for your review!
I'll check it.

yamadayutaka · 2023-05-04T09:12:22Z

Noticed failed integration tests:

Maybe you don't have enough environment variables set.
I added an environment variable check to my test code.

84a9eb5#diff-288bea63faddba4478b115bc89ce4fc23b9fab5f9a17dabce38fbb69033a0f7aR368-R373

denis256 · 2023-05-08T19:15:05Z

Definitely GCP environment variables should be added to CICD
Also, documentation should be updated to list required environment variables

Change environment variable name Restore default credentials before validate

yamadayutaka · 2023-05-09T01:55:32Z

Changed to set the contents of the service account key to an environment variable considering running in CICD.
Along with that, I changed the environment variable name.
Also added processing to revert to default credentials before validation.

yamadayutaka · 2023-05-09T02:07:50Z

Also, documentation should be updated to list required environment variables

Sorry, I couldn't find any documentation that lists the environment variables required for integration testing.
Could you tell me where in the documentation I should update?

findmyname666 · 2023-05-29T08:03:33Z

Is there anything I can help with to get PR reviewed & merged pls ?

FYI We cannot upgrade Terragrunt because impersonation is broken in GCP.

Thx a lot.

denis256 · 2023-06-06T20:36:01Z

Conflicting files

test/integration_test.go

yamadayutaka · 2023-06-08T02:28:25Z

Conflicts resolved.

jmesterh · 2023-06-23T20:54:15Z

Is this PR still waiting on the documentation update? We would like to adopt terragrunt, but this bug is a show-stopper since our backends all use impersonation to switch to a Terraform service account.

denis256 · 2023-06-25T19:37:54Z

This PR has not passing integration tests:

=== RUN   TestTerragruntWorksWithImpersonateGCSBackend
    integration_serial_test.go:369: Required environment variable `GCLOUD_SERVICE_KEY_IMPERSONATOR` - not found
--- FAIL: TestTerragruntWorksWithImpersonateGCSBackend (0.00s)

jmesterh · 2023-06-29T04:49:15Z

It is not passing integration tests because the CICD attached to this repo needs to be updated to include a second service account key. Yamadayutaka mentioned that above and in the test code. I assume this is something that must be done by a project admin who has access to the CircleCI account attached to this repo?

Pardeep009 · 2023-07-12T08:36:55Z

Hi @denis256 any updates on this, we are also blocked on using the latest terraform version in our environment because of this broken impersonation thing, do anyone here know other way-out for using the latest terraform version using terragrunt ?

denis256 · 2023-07-12T20:15:22Z

test/integration_serial_test.go

+	// run without impersonation
+	tmpTerragruntGCSConfigPath := createTmpTerragruntGCSConfig(t, TEST_FIXTURE_GCS_PATH, project, TERRAFORM_REMOTE_STATE_GCP_REGION, gcsBucketName, config.DefaultTerragruntConfigPath)
+	// run terregrunt ignore errors
+	runTerragruntCommand(t, fmt.Sprintf("terragrunt apply -auto-approve --terragrunt-non-interactive --terragrunt-config %s --terragrunt-working-dir %s", tmpTerragruntGCSConfigPath, TEST_FIXTURE_GCS_PATH), os.Stdout, os.Stderr)


why error should be ignored? why can't be used assert.Error()?

Since the command is executed without impersonation, the command will fail due to lack of privileges.
This is expected behavior and the error should be ignored.

Lack of privileges should prevent the bucket from being created, so checking for that here.
https://github.com/gruntwork-io/terragrunt/pull/2550/files/7956720c2c9cc4a17fb6d8de61a54494e4962655#diff-288bea63faddba4478b115bc89ce4fc23b9fab5f9a17dabce38fbb69033a0f7aR393

If is expected to have an error, then should be checked that the error is returned, it will help in the future if for some reason behavior of code will change and error will not be thrown, it will also be more to find that error is not returned instead if debugging why GCS bucket wasn't created.

Thanks for explaining.
I have modified it to use assert.Error().

cmeury · 2023-07-26T10:14:28Z

Do I understand correctly, the "only" thing missing is a second service account key in the Circle CI configuration (as per this comment?

denis256 · 2023-07-26T19:03:33Z

Hello,
yes, I asked for help with adding an additional service account to CICD pipeline which will be used in impersonation tests

robmorgan · 2023-08-14T10:18:46Z

Sorry for the delay on our side. I've added the GCLOUD_SERVICE_KEY_IMPERSONATOR environment variable to our CI/CD pipeline, but I still have an issue running the tests. Here is the output:

circleci@33ac7ec63d42:~/project/test$ go test -v -run TestTerragruntWorksWithImpersonateGCSBackend 
=== RUN   TestTerragruntWorksWithImpersonateGCSBackend
runTerragruntVersionCommand after split
[terragrunt apply -auto-approve --terragrunt-non-interactive --terragrunt-config /tmp/terragrunt-test3955787892/terragrunt.hcl --terragrunt-working-dir fixture-gcs/]
ERRO[0000] Create GCS bucket terragrunt-test-bucket-q6uj9t returned an error: googleapi: Error 403: circleci-impersonator@XYZ does not have storage.buckets.create access to the Google Cloud project. Permission 'storage.buckets.create' denied on resource (or it may not exist)., forbidden. Sleeping for 10s and will try again.  prefix=[/tmp/terragrunt-test3955787892] 
ERRO[0011] Create GCS bucket terragrunt-test-bucket-q6uj9t returned an error: googleapi: Error 403: circleci-impersonator@XYZ does not have storage.buckets.create access to the Google Cloud project. Permission 'storage.buckets.create' denied on resource (or it may not exist)., forbidden. Sleeping for 10s and will try again.  prefix=[/tmp/terragrunt-test3955787892] 
^Csignal: interrupt
FAIL	github.com/gruntwork-io/terragrunt/test	19.620s
circleci@33ac7ec63d42:~/project/test$ go test -v -run TestTerragruntWorksWithImpersonateGCSBackend
=== RUN   TestTerragruntWorksWithImpersonateGCSBackend
runTerragruntVersionCommand after split
[terragrunt apply -auto-approve --terragrunt-non-interactive --terragrunt-config /tmp/terragrunt-test3275918446/terragrunt.hcl --terragrunt-working-dir fixture-gcs/]
ERRO[0000] Create GCS bucket terragrunt-test-bucket-kxderq returned an error: googleapi: Error 403: circleci-impersonator@XYZ does not have storage.buckets.create access to the Google Cloud project. Permission 'storage.buckets.create' denied on resource (or it may not exist)., forbidden. Sleeping for 10s and will try again.  prefix=[/tmp/terragrunt-test3275918446] 
ERRO[0011] Create GCS bucket terragrunt-test-bucket-kxderq returned an error: googleapi: Error 403: circleci-impersonator@XYZ does not have storage.buckets.create access to the Google Cloud project. Permission 'storage.buckets.create' denied on resource (or it may not exist)., forbidden. Sleeping for 10s and will try again.  prefix=[/tmp/terragrunt-test3275918446] 
ERRO[0021] Create GCS bucket terragrunt-test-bucket-kxderq returned an error: googleapi: Error 403: circleci-impersonator@XYZ does not have storage.buckets.create access to the Google Cloud project. Permission 'storage.buckets.create' denied on resource (or it may not exist)., forbidden. Sleeping for 10s and will try again.  prefix=[/tmp/terragrunt-test3275918446] 
ERRO[0032] Create GCS bucket terragrunt-test-bucket-kxderq returned an error: googleapi: Error 403: circleci-impersonator@XYZ does not have storage.buckets.create access to the Google Cloud project. Permission 'storage.buckets.create' denied on resource (or it may not exist)., forbidden. Sleeping for 10s and will try again.  prefix=[/tmp/terragrunt-test3275918446] 
    integration_test.go:4390: Error creating GCS client: dialing: google: could not find default credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.
    integration_test.go:4443: Error creating GCS client: dialing: google: could not find default credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.

yamadayutaka · 2023-08-17T01:59:50Z

Hi @robmorgan, Thank you for trying.

The first call of terragrunt should result in an error, could you try waiting a little longer without interrupting?

And perhaps the GOOGLE_APPLICATION_CREDENTIALS environment variable is set incorrectly.
The CI/CD pipeline outputs the value of the GCLOUD_SERVICE_KEY environment variable to a file and sets it to the GOOGLE_APPLICATION_CREDENTIALS environment variable, so I think the same setting is necessary when running tests on the command line.

https://github.com/gruntwork-io/terragrunt/blob/master/.circleci/config.yml#L95-L97

For example, as follows.

echo $GCLOUD_SERVICE_KEY > ${HOME}/gcloud-service-key.json
export GOOGLE_APPLICATION_CREDENTIALS=${HOME}/gcloud-service-key.json
go test -v -run TestTerragruntWorksWithImpersonateGCSBackend

denis256 · 2023-08-17T05:50:54Z

Will take a look on this error

bjgbeelen · 2023-10-04T08:12:35Z

For what it is worth, just expressing my interest in this feature! (eagerly been keeping an eye on this PR for a while)

denis256 · 2023-10-04T08:36:25Z

Changes from this PR merged and released as part of: #2679

If the issue still occurs, please open a separate issue

fgateuil and others added 14 commits May 2, 2023 06:58

Comply with the Google Cloud impersonate library

6d91442

Impersonate test

e7585fc

fetching of gcp email

48441a2

Reverted test changes

fd53a4e

test for not existing account

4e4ea14

updated service account

fc9daaf

341219

9717869

testing project url

d1e0656

Updated account email

27b8998

Updated account url

2089ec2

cleanup

5a4721d

Comply with the Google Cloud impersonate library

41668e7

Change scope for label update

1650b25

Add test for gcs backend with impersonation

5745563

yamadayutaka requested review from zackproser and denis256 as code owners May 2, 2023 08:27

yamadayutaka mentioned this pull request May 2, 2023

Comply with the Google Cloud impersonate library #2052

Closed

denis256 reviewed May 2, 2023

View reviewed changes

yamadayutaka added 2 commits May 4, 2023 17:54

Fixed to use predefined ctx variable

198ee5e

Add required environment variable check

84a9eb5

Set impersonator service account key as contents

786ecaf

Change environment variable name Restore default credentials before validate

Merge branch 'master' into bug/issue-1997

7956720

denis256 reviewed Jul 12, 2023

View reviewed changes

fix: add error checking

4ca6098

yamadayutaka force-pushed the bug/issue-1997 branch from 556674c to 4ca6098 Compare July 17, 2023 11:24

denis256 closed this Oct 4, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Comply with the Google Cloud impersonate library and implement integration test #2550

Comply with the Google Cloud impersonate library and implement integration test #2550

yamadayutaka commented May 2, 2023 •

edited

Loading

denis256 May 2, 2023

yamadayutaka May 4, 2023

denis256 May 2, 2023

yamadayutaka May 4, 2023

denis256 commented May 2, 2023 •

edited

Loading

yamadayutaka commented May 4, 2023

yamadayutaka commented May 4, 2023

denis256 commented May 8, 2023

yamadayutaka commented May 9, 2023

yamadayutaka commented May 9, 2023

findmyname666 commented May 29, 2023

denis256 commented Jun 6, 2023

yamadayutaka commented Jun 8, 2023

jmesterh commented Jun 23, 2023

denis256 commented Jun 25, 2023

jmesterh commented Jun 29, 2023

Pardeep009 commented Jul 12, 2023

denis256 Jul 12, 2023

yamadayutaka Jul 16, 2023

denis256 Jul 17, 2023

yamadayutaka Jul 17, 2023

cmeury commented Jul 26, 2023

denis256 commented Jul 26, 2023 •

edited

Loading

robmorgan commented Aug 14, 2023

yamadayutaka commented Aug 17, 2023

denis256 commented Aug 17, 2023

bjgbeelen commented Oct 4, 2023

denis256 commented Oct 4, 2023

Comply with the Google Cloud impersonate library and implement integration test #2550

Comply with the Google Cloud impersonate library and implement integration test #2550

Conversation

yamadayutaka commented May 2, 2023 • edited Loading

Description

Test senario

Preparation

Execution

Result

TODOs

Release Notes (draft)

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

denis256 commented May 2, 2023 • edited Loading

yamadayutaka commented May 4, 2023

yamadayutaka commented May 4, 2023

denis256 commented May 8, 2023

yamadayutaka commented May 9, 2023

yamadayutaka commented May 9, 2023

findmyname666 commented May 29, 2023

denis256 commented Jun 6, 2023

yamadayutaka commented Jun 8, 2023

jmesterh commented Jun 23, 2023

denis256 commented Jun 25, 2023

jmesterh commented Jun 29, 2023

Pardeep009 commented Jul 12, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cmeury commented Jul 26, 2023

denis256 commented Jul 26, 2023 • edited Loading

robmorgan commented Aug 14, 2023

yamadayutaka commented Aug 17, 2023

denis256 commented Aug 17, 2023

bjgbeelen commented Oct 4, 2023

denis256 commented Oct 4, 2023

yamadayutaka commented May 2, 2023 •

edited

Loading

denis256 commented May 2, 2023 •

edited

Loading

denis256 commented Jul 26, 2023 •

edited

Loading