Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow to interpolate ${var.*} inside backend configuration #17288

Open
alekbarszczewski opened this issue Feb 7, 2018 · 11 comments
Open

Allow to interpolate ${var.*} inside backend configuration #17288

alekbarszczewski opened this issue Feb 7, 2018 · 11 comments

Comments

@alekbarszczewski
Copy link

@alekbarszczewski alekbarszczewski commented Feb 7, 2018

Terraform Version

Terraform v0.11.3
+ provider.aws v1.8.0

Use case

For now I am using local (file) state.
I have stage variable that defaults to "dev", local project variable that is a constant "my-project" and local prefix variable that is equal to "${local.project}-${var.stage}". I prepend this prefix to all resources, for example I have bucket my-project-dev-files, dynamo table my-project-dev-users and so on. This way I can keep separate projects and stages (environments like dev/prod/etc) in one AWS account. When deploying my project from travis for example I just pass stage variable via env to terraform (var is based on repository branch). So develop branch uses stage "dev", master uses "prod" and so on.

Now I want to use remote state via S3 backend. However I have problem because terraform does not allow interpolation when configuring backend. I would like to use two separate S3 buckets for dev and prod stages for example. I know that I can pass some config via terraform argument -backend-config="KEY=VALUE", but it requires me to pass this additionally to stage var:
TF_VAR_stage=dev terraform apply -backend-config="bucket=my-project-dev". It would be nice to be able to still interpolate at least ${var.*} inside backend.

Or maybe my approach is not the right one?

Terraform Configuration Files

terraform {
  backend "s3" {
    bucket = "${local.prefix}-state"
    key    = "state.tfstate"
  }
}

data "terraform_remote_state" "network" {
  backend = "s3"
  config {
    bucket = "${local.prefix}-state"
    key    = "state.tfstate"
  }
}

provider "aws" {
  region  = "us-east-1"
}

data "aws_region" "current" {
  current = true
}

variable "stage" {
  type = "string"
  default = "dev"
}

locals {
  project = "my-project"
  prefix = "${local.project}-${var.stage}"
}
@apparentlymart

This comment has been minimized.

Copy link
Member

@apparentlymart apparentlymart commented Feb 7, 2018

Hi @alekbarszczewski! Thanks for this question / feature request.

Backends are specified the way they are because any operation Terraform does starts by first accessing the backend, before doing any other work. For example, the terraform workspace list command requires the backend to be able to inspect remote data to determine which workspaces exist, which for the S3 backend is currently implemented by prefix-matching keys within the given S3 bucket.

Our current recommendation is to treat Terraform -- and thus the Terraform states -- as something "outside" the environments they manage, rather than as part of the environment. In practice, that means that e.g. there is an S3 bucket for your Terraform states that is separate from the environments managed by Terraform. Some organizations think of this as a "special" environment called e.g. admin or tools, using this as a home for any pan-environment management tools, e.g. also including a CI system that is used to run Terraform.

We've documented an example pattern for Terraform across multiple AWS accounts, which is more separation than you want to do here but can be simplified down to an equivalent idea with multiple resource sets under a single AWS account.


In future we could add a feature to the S3 backend in particular so that it's able to alternatively prefix-match (or, perhaps, suffix-match) the bucket name itself, allowing you to have multiple S3 buckets in your account like mycompany-terraform-production, mycompany-terraform-staging, and then list all of the buckets in your account to find which workspaces exist.

So far it's seemed more common for organizations to use a separate AWS account per environment, which led to us creating the document above, but if multiple buckets in a single organization is also a common pattern it certainly seems technically possible to add it, at the expense of requiring some more liberal IAM policies to enable the bucket names to be enumerated. I'd be interested to hear if you and others would find such a feature useful, and if so what sort of bucket naming conventions you'd want to support. (You already shared this, but I'd like to see what patterns others follow too, in order to inform the requirements for the feature.)

@alekbarszczewski

This comment has been minimized.

Copy link
Author

@alekbarszczewski alekbarszczewski commented Feb 8, 2018

@apparentlymart Thanks for detailed response. I somehow missed the fact that in remote state stored in S3 each state (per workspace) is prefixed with <workspace>:. So I ended up having single state bucket for both staging/prod workspaces and it works perfectly. I just prefix all resources (their unique names in aws) with my-project-${terraform.workspace} instead of using variable my-project-${var.stage}. For me this is enough when keeping multiple environments (workspaces) on single AWS account.

But I agree that keeping each environment on separate AWS account is the best approach since it separates everything and you don't have to prefix each resource name.


I ended up using following .travis.yml file for continious integration / tests of my project:

language: node_js
node_js:
  - '8'
env:
  global:
    - AWS_ACCESS_KEY_ID=***
    - AWS_AWS_DEFAULT_REGION=us-east-1
    - secure: "***"
    - TEMP_WORKSPACE="travis-build-$TRAVIS_BUILD_ID"
install:
  - yarn
cache:
  directories:
    - node_modules
before_script:
  - echo "PREPARING TEMP WORKSPACE $TEMP_WORKSPACE"
  - ./bin/terraform init aws-stack/ &&
    ./bin/terraform workspace new "$TEMP_WORKSPACE" aws-stack/ &&
    make package &&
  - ./bin/terraform apply --auto-approve aws-stack/
script:
  - ./bin/terraform output aws-stack/
after_script:
  - ./bin/terraform destroy -force aws-stack/ &&
    ./bin/terraform workspace select default aws-stack/ &&
    ./bin/terraform workspace delete "$TEMP_WORKSPACE" aws-stack/

In before_script I create new workspace based on travis build id and deploy stack.
In script I will run my integration tests, based on terraform output.
In after_script (in case of both success/failure of script) I cleanup everything: destroy stack and delete workspace.


Unfortunately I ran into another issue: #17300

@alekbarszczewski

This comment has been minimized.

Copy link
Author

@alekbarszczewski alekbarszczewski commented Mar 13, 2018

@apparentlymart Hey, after rethinking this issue is a blocker for me. The idea of two AWS accounts per environment (dev/prod) is great, but right now, without interpolation of S3 bucket in backend it's not possible to separate dev and prod environments completely. There always must be a shared S3 bucket for remote state for both environments.

What we want now is to make environments completely separate from each other for security reasons/ I think there is no point of not allowing them to be separate. Both envs does not have to know about each other, share state or anything. The only thing that is blocking this is remote state s3 bucket that can't be interpolated in config and has to be shared because bucket names are globally unique.

I read docs about using three AWS accounts and role delegation but this is not what we want. For security reasons we want to make both environments completely separate.

Terraform is great and we are using it for our second project in production, however this one is really blocker for us. Right now we are considering generating backend.tf file dynamically when deploying to one of the environments, but it's not a way to go...

By any chance will you improve/fix this in the nearest future? Either interpolating backend bucket name or at least some dynamic prefix for bucket or something like that?

@arocki7

This comment has been minimized.

Copy link

@arocki7 arocki7 commented Sep 12, 2018

We are having the same issue here.

@UncleTawnos

This comment has been minimized.

Copy link

@UncleTawnos UncleTawnos commented Nov 15, 2018

simply for simplicity of configuration I'd LOVE to see it hapening

@apparentlymart

This comment has been minimized.

Copy link
Member

@apparentlymart apparentlymart commented Nov 15, 2018

The current way to achieve this entirely separate configuration per environment is to not use "workspaces" at all and instead use multiple separate backend configurations.

To do this, the backend block in the main configuration would be empty:

terraform {
  backend "s3" {
  }
}

Then create another file in the directory for each environment. These can either be hand written and checked in or generated by your automation wrapper, depending on what makes sense for your use-case. For example, here's a production.tfbackend file based on the original comment:

  bucket = "example-production-state"
  key    = "state.tfstate"

Switching between these environments is then done by reconfiguring the backend itself, rather than switching workspaces:

terraform init -reconfigure -backend-config=production.tfbackend

I understand that right now this means that you also need to pass a similar or identical value into a variable, which is inconvenient.

The backend and workspace features will get an overhaul in a future release to better permit separate configuration per workspace. The configuration can never support interpolation for the reasons I stated before, but a per-workspace configuration capability would allow the total separation between environments that you are looking for. The current focus of the Terraform team at HashiCorp is on the 0.12 release (which does not include backend changes) but we intend to address this in a later release.

@pixelicous

This comment has been minimized.

Copy link

@pixelicous pixelicous commented Feb 6, 2019

@apparentlymart
I am not entirely sure i am answering the same thing, but just to give my two cents here for anyone going through this post, it is possible to use workspaces in such/similar fashion.
We have a folder structure that resembles provider>account>environment>appmodule
We use workspaces as our differentiator for running the same module folder but on a different region. In my opinion the resources you would like to provision per region most likely would be the same, but it aint so true per environment, anyhow.. we can then use terraform.workspace as our location anywhere as well.

Any module ("appmodule" folder above) have a backend configuration but just the type is set, the wrapper always runs init first and the backend variables are passed to terraform at runtime (aka partial configuration). After init whenever plan/apply runs and we use remote state with interpolation it look as follows:

data "terraform_remote_state" "my_remote_state" {
  backend = "azurerm"
  workspace = "${terraform.workspace}"
    config {
    storage_account_name = "xxxxxx"
    container_name       = "${var.environment}"
    key                  = "${var.environment}-network.tfstate"
    access_key = "${var.access_key}"
  }
}
scottx611x added a commit to refinery-platform/isa-tab-exporter that referenced this issue Feb 11, 2019
@omar

This comment has been minimized.

Copy link

@omar omar commented Feb 12, 2019

The solution proposed by apparentlymart requires our build system to output files which is something I'd prefer to avoid to keep inline with how other build systems are generally configured (i.e. through environment variables). Additionally, I'd like to maintain parity between how our CI/CD systems run and how developers run things locally eliminating the need for command line switches when running specific commands.

If the backends supported pulling their values from environment variables in the same way they pull from the command line argument, interpolation could be avoided. For example:

  • backend.tf:
terraform {
  backend "s3" {
   # Keep this empty
  }
}
  • Environment Variables:
export TF_VAR_stage="production"
export TF_WORKSPACE="production"
export TF_BACKEND_S3_BUCKET="production-state"
export TF_BACKEND_S3_KEY="state.tfstate"

CI/CD systems and developers just need to configure the proper environment variables when running and everything will run the same. No need for extra command line switches when running Terraform commands.

If there's enough support for this idea, I could see myself contributing it. But I want to get a TF team member's take on this in case I'm missing some important context.

@solsglasses

This comment has been minimized.

Copy link

@solsglasses solsglasses commented Feb 13, 2019

Good to see a fresh issue created on this limitation. @omar's suggestion is, obviously, better than having a wrapper script that creates state.tf files prior to terraform being run. Show support to his idea by reacting with 👍 on #17288 (comment)

@omar

This comment has been minimized.

Copy link

@omar omar commented Feb 22, 2019

I've opened a PR, see #20428. Feedback welcome.

@manniche

This comment has been minimized.

Copy link

@manniche manniche commented Apr 11, 2019

@alekbarszczewski The PR #20428 has been closed with information about how to accomplish the objective that was the reason for opening this issue. Should this issue be closed?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
8 participants
You can’t perform that action at this time.