Support separate backend configurations for each workspace

[2rs2ts]

Terraform Version

Terraform v0.10.8

Terraform Configuration Files

terraform {
  backend "s3" {}
}

Debug Output

N/A

Crash Output

N/A

Expected Behavior

I should be able to run any terraform command with TF_WORKSPACE=... to separate the operations between different states, as originally requested in #14447, and implemented for the new workspaces terminology in #14952. I would expect that, therefore, in order for this to work terraform would need to configure the backends for each workspace separately, so that multiple states can be manipulated in parallel. Switching workspaces should not cause any messages about the backend being reconfigured.

Actual Behavior

terraform init does not care about workspace; it always tries to reconfigure the current backend when I switch to a new workspace and then initialize that one for a new state.

Steps to Reproduce

Please list the full steps required to reproduce the issue, for example:

1. TF_WORKSPACE=foo terraform init -backend-config=bucket=mybucket -backend-config=region=us-east-1 -backend-config=key=foo.tfstate
2. TF_WORKSPACE=bar terraform init -backend-config=bucket=mybucket -backend-config=region=us-east-1 -backend-config=key=bar.tfstate

Terraform will say the backend configuration has changed and will ask to copy state from s3.

Important Factoids

Our team uses one tfstate per AWS region per environment. So for instance our qa environment has state files for us-west-2, us-east-1, us-east-2, and so on respectively, and our production environment also has its own state files for us-west-2, us-east-1, us-east-2 and so on respectively. It is extremely common for us to be performing terraform operations against multiple regions per environment at once, and sometimes we even do it for multiple environments at once (similar to the use case in #14447)

References

• Allow per-command Environment selection (eg, with an environment variable) #14447 gave me false hope that this would work :( we've been stuck on 0.8 because the new terraform init breaks our use case, but I thought perhaps the TF_WORKSPACE env var would be the secret sauce...

[jbardin]

Hi @2rs2ts,

Thanks for opening the issue. A workspace is something that exists within a backend, so having a separate backend per workspace doesn't really fit the current model.

However, I would like to eventually have a more convenient way to switch between backend configurations. If we get something like that in place, I think it would be easier to switch backend+workspace combinations.

[2rs2ts]

@jbardin I would expect that when I switch workspaces, I am basically getting a completely different environment, much like python's virtualenv. Or maybe think of it in this way: a workspace is like a completely separate clone of your repo.

When I switch workspaces I should be able to apply changes without worrying about writing to the same backend. I mean, what you're describing sounds like all it takes is a little mishap and you apply development configuration to a customer-facing environment. It's kind of hard to see the value of workspaces now that you're telling me this.

Without having a ton of internal knowledge it sounds like all you have to do is invert the order of workspaces and backends. (Probably easier said than done.) Backends should exist within a workspace. When you switch workspaces you shouldn't have to re-init unless you're actually trying to change backend configuration or do other things init does. And when you use TF_WORKSPACE with your commands you should be able to run completely isolated operations on multiple workspaces concurrently.

Just one way to implement it would be to store all the backend configuration in subfolders where each subfolder is a separate workspace. So on a fresh clone and terraform init you'll get set up in the default subfolder, where all the backend configuration for the default workspace will live; and if you create a new workspace foo you get a new subfolder called foo and once you terraform init while in that workspace you'll now be able to switch between default and foo without having to re-run terraform init. A file in the .terraform directory could store which workspace is the currently active one, and when using TF_WORKSPACE, the currently active one is ignored in favor of using the env var. I hope I am describing this effectively. You may have an even better idea too.

[jbardin]

Hi @2rs2ts,

I appreciate the feedback. Workspaces only exist within a backend, or to state it another way, until a backend is loaded (including the default "local" backend) there are no workspaces. There are no plans to invert this relationship only in the cli, because this would conflict with how the backends operate, and would break integrations with other software.

As for this specific use case, I do want to have a method for easily switching between backend configurations. So in essence, we are looking into accomplishing what you want here, it's just not going to be done with workspaces alone.

[2rs2ts]

@jbardin Is there an ETA for this method for switching between backend configurations? Perhaps a milestone target?

[2rs2ts]

Also, do you know how other people work around this issue (i.e. the lack of support for concurrent operations)? Seems to me like a pretty common issue to have so I'm surprised I haven't seen any issues filed with similar complaints. Perhaps there's a workaround I haven't thought of.

[jbardin]

@2rs2ts, Sorry, we don't have a set timeline for that enhancement.

Most users and organization use a single backend per config, so there's usually nothing to work around. If the backend does need to be change, the -reconfigure flag can be used as a work around to update the config without any migration. I'm not sure what you mean by lack of support for concurrent operations, as workspaces themselves represent entirely separate states, and even within workspaces the most commonly used backends support state locking.

[2rs2ts]

I'm not sure what you mean by lack of support for concurrent operations, as workspaces themselves represent entirely separate states, and even within workspaces the most commonly used backends support state locking.

I meant concurrently working on different workspaces.

It's fine if you don't have a timeline. We'll come up with a workaround. Hopefully in the future terraform will support working on different workspaces concurrently.

Thanks for all the info!

[robbruce]

Just been working around the same issue, for ourselves we would like to use the same backend type, but a different bucket for the remote state; separate buckets for dev, test prod for example.

Our workaround has been to use git branches of the same name and then to have separate clones of our git repo at the relevant branches.

[apparentlymart]

Although it doesn't address the specific use-case described here, we do have a guide on the usage pattern the current implementation was designed for, which may be useful as a starting point for those who don't have existing practices in place or different regulatory requirements.

We do still plan to address this eventually by some means, but the team's current focus is on the configuration language. We plan to switch focus to other areas later in the year and will hopefully be able to implement something in this area at that point. We need to spend a little time prototyping first since, as noted before, the current architecture is not able to accommodate this requirement. More details to some once we've completed that investigation and have a more specific plan.

[deitch]

Just came across this. I had originally interpreted workspaces as completely separate environments, including backend, rather than within a backend. I struggled to make sense of it, since I do different AWS account for different environments. So if my single backend is an S3 bucket on account A, and workspace dev is in account A but workspace prod is in account B, it gets interesting. I can work around it by putting profile or keys or credential files in the backend, but you can see how there is a bit of an impedance mismatch in how I tried to do it.

My workaround ended up being having separate directories, each of which just has the backend and includes a shared directory as a module for everything else. It is a bit redundant on the variable and output definitions, but best I could figure out.

terraform/
 |
 |-prod/
     |
     |-main.tf (has one backend setting; references a module in ../shared)
 |-dev/
     |
     |-main.tf (has a different backend setting; references a module in ../shared)
 |-shared/
     | 
     |- site.tf (etc.)

[christhomas]

This appears to be a terrible design decision. Especially after reading dozens of similar comments. You appear to have done the exact opposite to what a workspace is supposed to be. An isolated environment separate from each other. The fact that dozens of people are telling you the same thing should inspire you to fix this issue in a more understandable way.

And if this breaks tools, then just change the tools, they were broken in the first place. So now is the time to fix them.

Here is a simple use case which breaks the current idea of workspaces.

1. Work with localstack on your dev machine
2. Work with AWS for staging and production
3. Create a workspace for both localstack, staging, and production
4. Switch workspace to staging, init, then apply the configuration, it's using s3 for remote state
5. Switch workspace to dev, init, then apply the configuration, it's using local state (since obviously I dont want to share my machines state with anybody else)
6. It's broken, the s3 backend is shared between localstack and staging and now it's complaining that localstack isn't using the s3 backend and now my entire setup is broken and I have to delete all the states and start over (cause it's gotten corrupted).

This is a really common scenario. Other people are just getting around your problems by using workarounds. But it would be nice if the tools operated in a better way which allows easier isolation between workspaces

Maybe a workspace flag on new called --share-backend=yes|no would easily resolve this problem

[christhomas]

So I was sufficiently motivated to solve this problem. I want that if I create a separate workspace, I want to not share backends.

#!/usr/bin/env bash
# usage: chris-terraform ... (any terraform command you want)

role=XYZ
credentials=(`aws sts assume-role --role-arn "${role}" --role-session-name terraform --query '[Credentials.AccessKeyId,Credentials.SecretAccessKey,Credentials.SessionToken]' --output text`)

AWS_ACCESS_KEY_ID=${credentials[0]}
AWS_SECRET_ACCESS_KEY=${credentials[1]}
AWS_SESSION_TOKEN=${credentials[2]}
AWS_SECURITY_TOKEN=${credentials[2]}

env_file=${PWD}/.terraform/environment

TF_ENV=()
[ ! -z "${TF_LOG}" ] && TF_ENV[0]="--env TF_LOG=${TF_LOG}"
[ ! -z "${TF_WORKSPACE}" ] && TF_ENV[1]="--env TF_WORKSPACE=${TF_WORKSPACE}"
[ -f "${env_file}" ] && TF_ENV[2]="--env TF_DATA_DIR=.terraform/$(cat ${env_file})"

# if using workspace command, then remove the TF_DATA_DIR env var
[ "$1" == "workspace" ] && TF_ENV[2]=

docker run ${INTERACTIVE} ${TF_ENV[@]} \
    --env AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID} \
    --env AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY} \
    --env AWS_SESSION_TOKEN=${AWS_SESSION_TOKEN} \
    --env AWS_SECURITY_TOKEN=${AWS_SECURITY_TOKEN} \
    --network backbone \
    -v ${PWD}:/app:consistent \
    -w /app \
    hashicorp/terraform:light $@

Usage:

chris-terraform workspace new localstack
chris-terraform init terraform/localstack <- this directory contains all the terraform files for localstack)

chris-terraform workspace new staging
chris-terraform init terraform/staging <- again, for staging

for localstack, I have a provider with custom endpoints for localstack, for staging, I have an s3 backend that I use to share state with my team.

Problem solved!

I could go further, I suppose I don't want to always override the workspace command like this, I could introduce a 'isolated-workspace' meta command which when found in the args, will apply the above code, then reinsert a 'workspace' command after it and when workspace is found on it's own will do the default terraform behaviour.

But for me, for this situation, I'm happy.

[Gerrit-K]

@christhomas Great to hear that you found a solution. I'm a bit hesitant to adapt it, but I completely agree with you on your first post. I cannot understand how they didn't see this scenario coming: sharing staging and production states while keeping dev states local is one of the most natural usages for environments.

[christhomas]

I've been using it for months and if you are consistent in working with workspaces like this, then it actually works out really nicely. No problems encountered so far.

[rmrbest]

@christhomas Thanks to share your solution. I would like to know how to works your solution. Can you share your entire code? I'm a bit lost...

[christhomas]

What would you like to know? That is the full solution, apart from the top lines which obtain the credentials from your IAM role. You could replace that with just using environment variables, or passing them into the script using the script parameters.

What part don't you understand?

[christhomas]

If anybody wants to ask questions, please ask here: https://gist.github.com/christhomas/ea90cc55502a3f804f0b6a8e59d05e60

[mgrotheer]

Has there been any updates/progress on this issue?

[eugene-chernyshenko]

It's very awful that this task hasn't get any updates :(

[CaptainStealthy]

Found this in the midst of a Google search. It's fairly easy to use separate backends in a CI/CD scenario by using environment variables. (or at least, that's how I think the Terraform extension for Azure DevOps does it, anyway)

But switching workspaces in this way locally is a bit of a pain :(

[EliiseS]

Would really like a solution to being able to locally deploy and remote deploy using the same terraform files. The folder structure is really not great.

[jbaker-dstl]

I am also interested in this - we have the use case where we deploy to completely separate AWS accounts, and would like to store our state in an S3 bucket within each account. I hoped Workspaces would enable this, but at the moment it looks like we're going to have to continue to change the back end block manually each time unless I've misunderstood :-(.

[robbie1douglas]

For what it's worth I am trying to do something similar albeit simpler. Really all I need is one workspace that deploys into our sandbox account and all the other workspaces to deploy into our production account. I don't mind if the S3 backend is only in one account for all the workspaces. The following appears to work (haven't had a chance to test thoroughly).

# backend.tf
terraform {
  backend "s3" {
    profile        = "default"
    bucket         = "my-terraform-state"
    key            = "terraform-multi-account-test/terraform.state"
    region         = "eu-west-1"
    encrypt        = true
    dynamodb_table = "my-terraform-state-lock"
  }
}

and

# provider.tf
variable "workspace_accounts" {
  type = map(string)
  default = {
    "sandbox" = "my-sandbox-keys"
    "dev"     = "default"
    "prod"    = "default"
  }
}

provider "aws" {
  shared_credentials_file = "$HOME/.aws/credentials"
  profile                 = var.workspace_accounts[terraform.workspace]
  region                  = "eu-west-1"
}

Notice the profile = "default" always uses the S3 backend in my default AWS account, but the provider will switch the AWS profile used based on the workspace.

[corticalstack]

Are we any closer to having workspaces that are treated as completely separate environments with different backends?

[ppar]

I've ran into the same situation as many others above.

It's possible to get the desired behaviour using the $TF_DATA_DIR, -backend-config= and -var-file= options. I've documented the pattern I'm using here:

https://gist.github.com/ppar/c4353e812f64f082dc7de8df7f1d6fdd

My approach doesn't use Workspaces, but AFAICS it addresses the expectation that people here are having from Workspaces.

[omar-bizreh]

I solved it by doing something like this since workspace is part of the backend. However; I'm not sure if this is considered bad practice

Script: tf_init.sh

#!/bin/bash
terraform init -backend-config="backend_$1.hcl" $2

and then I just call it: ./tf_init.sh dev/prod -reconfigure

I have two backend.hcl files

• backend_dev.hcl
• backend_prod.hcl

Content as follows:

bucket = BUCKET_NAME
profile = dev/prod-profile

Running terraform plan shows correct results

[toxeek]

Is there any updates on this issue?

[toxeek]

I suspect then that what is told here isn't true or possible? https://dev.to/aws-builders/mastering-terraform-how-to-manage-multiple-environments-with-dynamic-s3-backends-1p9