Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

config: validate backend configuration can't contain interpolations #12067

Merged
merged 1 commit into from Feb 27, 2017

Conversation

@mitchellh
Copy link
Member

commented Feb 18, 2017

Validate that backend configuration doesn't contain interpolations.

@mitchellh mitchellh added this to the Terraform 0.9 milestone Feb 18, 2017
@mitchellh mitchellh requested a review from jbardin Feb 18, 2017
@nanoz

This comment has been minimized.

Copy link

commented Feb 20, 2017

Do you think basic variables provided using a tfvars file could be interpolated at some point ?

It would be extremely useful to pipeline code through different environments (ref below) without requiring an external logic for -backend-config.

https://medium.com/@kief/https-medium-com-kief-using-pipelines-to-manage-environments-with-infrastructure-as-code-b37285a1cbf5#.ve3bqwrgw

@mitchellh

This comment has been minimized.

Copy link
Member Author

commented Feb 21, 2017

@nanoz Yes its possible.

I find it important to start with a smaller surface area: you can always add things but you can't really take things away. For the variable issue, I'm concerned with restricting the TYPES of interpolations that are allowed in different areas of a Terraform config. My fear is that that leads to a future where a user is unclear what kind of interpolation they can use. You can claim (and I do) that we're already there with users unclear when they can/can't use interpolations. I view partial support as exacerbating that problem. I don't think there is an easy solution to it but without adding effectively unremovable features to the config, I'd rather ship with a more limited solution, consider this, and make it support more in the future.

All that is to say: I want to head into a future where almost all of Terraform is parametrizable, I really do. But I want to do so with some more thought first.

@apparentlymart

This comment has been minimized.

Copy link
Member

commented Feb 21, 2017

@mitchellh I saw some initial work towards adding a first-class "environment" concept to Terraform a few weeks ago, but didn't yet see enough of it to have a good mental model for how that construct will work. My hope was that it would include a means to either specify a separate remote configuration per environment within the config or to use the environment name as an interpolation within a single configuration, both of which would go some way to address the issue raised here.

Currently at work we have a systematic way of constructing a remote state configuration given two pieces of information: a codebase name and an environment name. This allows us to centralize this convention in a single place, which for us is the wrapper script we use to run Terraform from a continuous integration/deployment tool. At the moment I find myself unsure how our current approach will adapt to the new architecture; if possible I'd like to avoid needing to hand-write the remote config for each environment+codebase pair into each repo, but it seems like for now that's going to be the recommended strategy. Is that right?

I'm sure I can make something work with generating on the fly an additional config file that gives the remote config before running terraform init, but I'd love to understand the design intent here so that I don't end up implementing something that goes against the grain of the intended workflow.

(I realize this is a bit of a big question to shove on the end of a closed issue. If there's documentation coming that will answer this question I'm happy to just wait and get my answers there... mainly just wanted to share the use-case.)

@mitchellh

This comment has been minimized.

Copy link
Member Author

commented Feb 21, 2017

@apparentlymart Scripting terraform init is expected/recommended. The backend config file is HCL so you can just as easily generate JSON. And the backend config only needs to merge with what is in the actual TF config to make a full backend config, it doesn't need to specify everything.

@apparentlymart

This comment has been minimized.

Copy link
Member

commented Feb 23, 2017

@mitchellh personally I agree that this is best kept for a later release, since this extra complexity should be approached with caution. In the long run it feels pragmatic to have three different possibilities for any configuration attribute:

  • No expressions are allowed at all
  • Only non-computed expressions are allowed
  • Any valid expression is allowed.

The second case here is the awkward one, but when 0.9 ships we'll see with the count meta-attribute whether this situation is intuitive for users. I expect it'll take some iteration on UX and terminology to help users understand what really is valid in that scenario, and why.

I think the trick will be guiding users up the complexity curve so that they learn supporting concepts along the way. For example, a user should have learned what computed means before they are introduced to data sources, so that they can understand the ways in which a data source behaves differently than a resource.

Based on earlier questions on this subject it's felt to me like the term "computed" isn't resonating with all users, so it might be worth spending some time researching alternative terminology here that might resonate better.

With that said, none of the stuff in this comment is intended to disagree with this PR... just musing about possible future work. For now, dynamically generating a configuration file containing the backend config should work for anyone who was previously scripting calls to terraform remote config ....

@michalmedvecky

This comment has been minimized.

Copy link

commented Feb 23, 2017

@apparentlymart can you please give me some hint on what do you mean by 'generate one-off configuration file that contains the desired settings'?

I am experienced just with the local terraform.tfstate file (and the -state and -var-file parameters), but now I'm trying to migrate the workflow from locally-run terraform apply to CD pipeline. As this will run in an ephemeral environment (docker container) I surely need to hand off the state handling to a different backend.

From my little experience, when I configure the backend as mentioned in #12200 and run terraform init, it creates some .terraform/terraform.tfstate file with a reference to the swift container and offers me to copy the current terraform.tfstate there. Do I understand correctly that you're suggesting to generate this file automatically (out of terraform) every run of CD pipeline and refer to the correct container? If yes, what about lineage and hash parameters that look like some reference numbers which are absolutely unclear to me?

@apparentlymart

This comment has been minimized.

Copy link
Member

commented Feb 24, 2017

The configuration I mean to generate is the terraform block that specifies the backend:

terraform {
  backend "foo" {
    # ...
  }
}

From 0.9 the settings that were formerly passed to terraform remote config must now be provided via configuration, so anyone that was scripting Terraform to mechanically configure remote state must replace the terraform remote config command with a call to generate a configuration file like above and then run terraform init.

@mitchellh

This comment has been minimized.

Copy link
Member Author

commented Feb 24, 2017

@apparentlymart They don't have to specifically, they just need to generate terraform { backend "type" {}} and then pass in a -backend-config file that is just JSON K/V.

@mitchellh mitchellh merged commit b502643 into master Feb 27, 2017
1 check passed
1 check passed
continuous-integration/travis-ci/pr The Travis CI build passed
Details
@mitchellh mitchellh deleted the b-backend-interp branch Feb 27, 2017
@nanoz

This comment has been minimized.

Copy link

commented Mar 1, 2017

I think #12182 (Environments) could actually mitigate my concern. Needs testing :)

@ozbillwang

This comment has been minimized.

Copy link

commented Mar 16, 2017

I will be happy if terraform init can support this

/bin/terraform init -backend=s3 -backend-config="bucket=com_mydomain_terraform_prod" -backend-config="key=iam/terraform.tfstate" -backend-config="region=us-east-1"

With that, I can avoid the interpolations limits.

@ozbillwang

This comment has been minimized.

Copy link

commented Mar 16, 2017

Vote down for this PR. Whatever, I hope to get interpolations support back. I like it work as old terraform remove config

And hope this type of PR can be reviewed not only one reviewer. It generates incompatible problem a lot.

@mitchellh

This comment has been minimized.

Copy link
Member Author

commented Mar 17, 2017

@SydOps We never supported interpolations in backend configurations (new in 0.9) and this was protecting people from using a feature we never supported. :) As I said above, we'd love to extend this support in the future, but we didn't take anything away to give "back": it never worked!

In 0.9.0 we added more places you can use interpolations, we didn't roll any back at all, so I believe there is some confusion happening!

@mitchellh

This comment has been minimized.

Copy link
Member Author

commented Mar 17, 2017

Just noticed you mentioned remote config. I think adding the ability to specify configurations on the CLI is something we can add back. Those aren't interpolations though. :)

@ozbillwang

This comment has been minimized.

Copy link

commented Mar 17, 2017

Thanks, @mitchellh

Since I can't make this work

terraform {
  backend "s3" {
   bucket = “terraform-xxxxxx"
   key = "${var.environment}/terraform.tfstate"
   encrypt = "true"
   kms_key_id = “xxx"
   region = “xxx"
 }
}

Then I prefer to feed the environment variable to terraform init directly via CLI. Otherwise, I have to prepare a backend configuration file every time.

@mitchellh

This comment has been minimized.

Copy link
Member Author

commented Mar 17, 2017

Right, I think we'll bring back the CLI based config too, but you can also use partial configurations, documented here: https://www.terraform.io/docs/backends/config.html (still requires a file but makes it more parameterizable)

@ozbillwang

This comment has been minimized.

Copy link

commented Mar 17, 2017

Thanks, @mitchellh

Yes, that's what I did currently.

In main.tf:

terraform {
  backend "s3" {
    encrypt = "true"
    region = "xxx"
  }
}

Then prepare a backend.conf file as

bucket = "terraform-xxxxxx"
key = "dev1/terraform.tfstate"
kms_key_id = "xxx"

This backend.conf has to be generated every time in a wrap script before I run terraform init -backend-config=backend.conf

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
7 participants
You can’t perform that action at this time.