Improve discovery and loading of credentials from Terraform config files #360

nfagerlund · 2021-09-04T00:45:48Z

Description

This provider is somewhat special, because the API token it uses is also used
by Terraform CLI itself. We can give a nicer experience on the local CLI by
auto-discovering Terraform's copy of the token (if one isn't provided in the
provider config or the TFE_TOKEN environment variable).

However, we weren't quite matching Terraform's credential-finding behavior:

We were respecting the TERRAFORM_CONFIG environment variable for the main
config file location, but not TF_CLI_CONFIG_FILE. (Terraform will use
either, but prefers the latter since it matches the other environment variable
names it uses.)
We weren't ever consulting the credentials file
(~/.terraform.d/credentials.tfrc.json, on unix), which is written to by
terraform login and/or credential helper scripts. The HCL library
understands the credentials JSON, so you could work around this by pointing
TERRAFORM_CONFIG at the credentials file, but if there was anything else
important in the config file you also had to set TF_CLI_CONFIG_FILE so that
Terraform (but not the provider) could still find it.

This commit fixes both of those: we'll accept a non-default config file location
via either environment variable (preferring TF_CLI_CONFIG_FILE), and we'll
also consult the credentials file if present.

If both sources include a token for the desired hostname, the provider will
follow Terraform's behavior and prefer the token from the CLI config file.

Testing plan

Create a basic config that configures the tfe provider and maybe manages one resource. In the provider config, provide a hostname (if you aren't using app.terraform.io) but not a token.
Ensure that your environment doesn't have any values set for TFE_TOKEN, TF_CLI_CONFIG_FILE, or TERRAFORM_CONFIG.
Ensure that neither your ~/.terraformrc or ~/.terraform.d/credentials.tfrc.json files include a token for the configured TFE host, to start out with.
Then:

Note: for the environment variable columns, "x" means "unset"; read any other value as "points to a loadable .terraformrc config file containing _____ for the configured hostname".

.terraformrc	credentials json	`TF_CLI_ CONFIG_FILE`	`TERRAFORM_ CONFIG`	Outcome
valid token	no token	x	x	Works (finds default config file)
no token	valid token	x	x	Works (finds credentials file, merges it with config file)
valid token	bad token	x	x	Works (config file beats credentials file)
no token	no token	valid token	x	Works (new env var works)
no token	no token	x	valid token	Works (old env var works)
bad token	no token	valid token	x	Works (env var beats config file)
no token	bad token	valid token	x	Works (env var beats credentials file)
no token	no token	valid token	bad token	Works (new env var beats old)
valid token	no token	no token	x	Fails (env var replaces config file instead of merging with it)
no token	valid token	no token	x	Works (credentials file is still merged with config file, even if config file is non-default)

Output from acceptance tests

See CI output in the checks below.

This provider is somewhat special, because the API token it uses is also used by Terraform CLI itself. We can give a nicer experience on the local CLI by auto-discovering Terraform's copy of the token (if one isn't provided in the provider config or the `TFE_TOKEN` environment variable). However, we weren't quite matching Terraform's credential-finding behavior: - We were respecting the `TERRAFORM_CONFIG` environment variable for the main config file location, but not `TF_CLI_CONFIG_FILE`. (Terraform will use either, but prefers the latter since it matches the other environment variable names it uses.) - We weren't ever consulting the credentials file (`~/.terraform.d/credentials.tfrc.json`, on unix), which is written to by `terraform login` and/or credential helper scripts. The HCL library understands the credentials JSON, so you could work around this by pointing `TERRAFORM_CONFIG` at the credentials file, but if there was anything else important in the config file you also had to set `TF_CLI_CONFIG_FILE` so that Terraform (but not the provider) could still find it. This commit fixes both of those: we'll accept a non-default config file location via either environment variable (preferring `TF_CLI_CONFIG_FILE`), and we'll _also_ consult the credentials file if present. If both sources include a token for the desired hostname, the provider will follow Terraform's behavior and prefer the token from the CLI config file.

website/docs/index.html.markdown

chrisarcand

This LGTM, though admittedly I don't have the time to properly go through the testing plan at the moment. Really really great PR!

chrisarcand · 2021-09-07T20:09:40Z

tfe/provider.go

+
+	// There might be credentials in the main CLI config file (manually entered)
+	// AND/OR the credentials file (auto-configured by terraform login or a
+	// credentials helper), so we need to consult both. As per the behavior of


This doesn't work with a credentials helper, no?

FWIW we could support them, the logic is here: https://github.com/hashicorp/terraform-svchost/blob/f050f53b97348772833a8f7fc1b297ec71f9fc7c/auth/helper_program.go

But this PR is already 💯 without all that, for sure 👍🏻 I'd just amend this comment.

Oh! I thought credentials helpers used the credentials file as a cache, though now I've forgotten where I would have gotten that idea.

I'll just amend the comment to only refer to login. 👍🏼

tfe/provider.go

omarismail

Functionally this is great! My main concern is just adding tests.

I put an example here:

c23bbb5

omarismail · 2021-09-07T21:11:58Z

tfe/provider.go

+	configFilePath := os.Getenv("TF_CLI_CONFIG_FILE")
+	if configFilePath == "" {
+		configFilePath = os.Getenv("TERRAFORM_CONFIG")
+	}


What do you think about refactoring this a tad bit for readability? I added an example here c23bbb5

The way the previous implementation combined the configs was throwing away the Hosts member from the one Config that might have anything in it. This makes the combining behavior a bit more explicit, and also pulls out some wordy conditionals into a separate function per Omar's suggestion.

Caught by tests.

More info about what this undocumented feature is used for over at hashicorp/terraform#28309

omarismail · 2021-09-10T15:54:40Z

tfe/provider_test.go

+	hasCredentials := "test-fixtures/env/home"
+	noCredentials := "test-fixtures/env/empty"
+	terraformrc := "test-fixtures/env/terraformrc"
+	noTerraformrc := "test-fixtures/env/nothing"


I know i'm the one who made this directory path, so I blame my older self 😆 . I think if I was looking at the directory path, i'd be confused what "env" referred to. Could we change these paths to be test-fixtures/tf-env-files/*

And nit comment, could we be more specific for the empty and nothing? So like test-fixtures/tf-env-files/no-terraformrc and test-fixtures/tf-env-files/no-credentials

👍🏼 !

I went with test-fixtures/cli-config-files/* for the path name.

omarismail · 2021-09-10T15:56:05Z

tfe/provider_test.go

+	noCredentials := "test-fixtures/env/empty"
+	terraformrc := "test-fixtures/env/terraformrc"
+	noTerraformrc := "test-fixtures/env/nothing"
+	fromTerraformrc := "something.atlasv1.prod_rc_file"


Nit as well, s/fromTerraformrc/tokenFromTerraformrc

omarismail

A few minor nits, but overall amazing job! once tests pass

nfagerlund requested a review from a team as a code owner September 4, 2021 00:45

nfagerlund commented Sep 7, 2021

View reviewed changes

website/docs/index.html.markdown Outdated Show resolved Hide resolved

docs: Remove reference to TERRAFORM_CONFIG

9abbbb0

chrisarcand approved these changes Sep 7, 2021

View reviewed changes

nfagerlund commented Sep 7, 2021

View reviewed changes

tfe/provider.go Outdated Show resolved Hide resolved

Amend a comment about the credentials file

14f07f5

omarismail requested changes Sep 7, 2021

View reviewed changes

omarismail reviewed Sep 7, 2021

View reviewed changes

nfagerlund added 5 commits September 9, 2021 10:21

Add tests for locateConfigFile()

2202ead

Fix assignment to entry in nil map with missing credentials.tfrc.json

d59ebba

Caught by tests.

Add loading/merging tests for cliConfig()

7f107f4

Extend cliConfig tests to check for host blocks

b69486b

More info about what this undocumented feature is used for over at hashicorp/terraform#28309

omarismail reviewed Sep 10, 2021

View reviewed changes

omarismail approved these changes Sep 10, 2021

View reviewed changes

Use more descriptive names for some files and variables

0cc08b2

nfagerlund merged commit 2a079b9 into main Sep 10, 2021

nfagerlund deleted the nf/sep21-core-credentials-file branch September 10, 2021 20:01

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve discovery and loading of credentials from Terraform config files #360

Improve discovery and loading of credentials from Terraform config files #360

nfagerlund commented Sep 4, 2021 •

edited

Loading

chrisarcand left a comment

chrisarcand Sep 7, 2021

nfagerlund Sep 7, 2021

omarismail left a comment •

edited

Loading

omarismail Sep 7, 2021 •

edited

Loading

omarismail Sep 10, 2021

omarismail Sep 10, 2021

nfagerlund Sep 10, 2021

omarismail Sep 10, 2021

omarismail left a comment •

edited

Loading

Improve discovery and loading of credentials from Terraform config files #360

Improve discovery and loading of credentials from Terraform config files #360

Conversation

nfagerlund commented Sep 4, 2021 • edited Loading

Description

Testing plan

Output from acceptance tests

chrisarcand left a comment

Choose a reason for hiding this comment

chrisarcand Sep 7, 2021

Choose a reason for hiding this comment

nfagerlund Sep 7, 2021

Choose a reason for hiding this comment

omarismail left a comment • edited Loading

Choose a reason for hiding this comment

omarismail Sep 7, 2021 • edited Loading

Choose a reason for hiding this comment

omarismail Sep 10, 2021

Choose a reason for hiding this comment

omarismail Sep 10, 2021

Choose a reason for hiding this comment

nfagerlund Sep 10, 2021

Choose a reason for hiding this comment

omarismail Sep 10, 2021

Choose a reason for hiding this comment

omarismail left a comment • edited Loading

Choose a reason for hiding this comment

nfagerlund commented Sep 4, 2021 •

edited

Loading

omarismail left a comment •

edited

Loading

omarismail Sep 7, 2021 •

edited

Loading

omarismail left a comment •

edited

Loading