[Feature Request] Supports workspaces and var files

[takanakahiko]

Thank you for creating a great tool.

This tool allows you to specify directories using dir, from_dir, and to_dir.
But that doesn't work if you want to specify a workspace.

For example, this would be the case if you want to terraform workspace select hogehuga for each directory.

$ terraform init -reconfigure
$ terraform workspace select workspace1 # I want to add this operation
$ terraform state pull > tmp.tfstate

# switch backend to local ...

$ terraform init -reconfigure
$ terraform state mv -state=tmp.tfstate aws_security_group.foo aws_security_group.foo1
$ terraform plan -state=tmp.tfstate -detailed-exitcode

# switch backend to remote ...

$ terraform init -reconfigure
$ terraform workspace select workspace1 # I want to add this operation too
$ terraform state push tmp.tfstate

It would be nice to have the option to specify workspace and var_file for each type, but what do you think?

example

migration "state" "test" {
  dir = "dir1"
  workspace = "workspace1"
  var_file = "./variable. tfvars"
  actions = [
    "rm aws_security_group.baz",
  ]
}

migration "multi_state" "mv_dir1_dir2" {
  from_dir = "dir1"
  from_workspace = "workspace1"
  to_dir = "dir2"
  to_workspace = "workspace2"
  actions = [
    "mv aws_security_group.foo aws_security_group.foo2",
    "mv aws_security_group.bar aws_security_group.bar2",
  ]
}

[takanakahiko]

It's okay for me to create a pull request by me and check it to decide whether or not to adopt a new feature.

[minamijoyo]

@takanakahiko Thank you for the proposal!

Adding workspace and variable file support is ok because they are common use-case for Terraform, but, I don't use them in my work, so I'm not sure what is the best design for refactoring tfstate with the workspace model.

If my understanding is correct, using workspace has a single configuration and multiple states. Workspaces are usually related to environments such as prod, test and dev, a configuration change will be applied in a workflow for each environment.

When we want to rename a resource in configuration, we should move the resource in all states. Wouldn't it be better to apply one migration to all workspaces in each workflow? The workspace model enforces us exactly equal configurations across workspaces. I don't think it makes sense to apply migration only to some workspaces. I think a migration should correspond to a configuration, and switching workspaces should be implemented by changing the execution environment not to declare it in the migration.

If so, we can automatically select workspace in each workflow with an environment variable TF_WORKSPACE and switch variable files with TF_CLI_ARGS_name.
https://learn.hashicorp.com/tutorials/terraform/automate-terraform#multi-environment-deployment
https://www.terraform.io/docs/commands/environment-variables.html#tf_cli_args-and-tf_cli_args_name

What do you think of it?

[takanakahiko]

@minamijoyo Thank you for your reply!

For example, what I want to do is support for the following use cases:
I want to define ec2, which was managed in different directories called us and tokyo, in commonized.
In that case, I thought it was necessary to transfer resources between workspaces.

• us/ (workspace: us)
  • ec2.tf
• tokyo/ (workspace: tokyo)
  • ec2.tf
• commonized/
  • ec2.tf
  • variables-us.tfvars (workspaces: commonized-us )
  • variables-tokyo.tfvars (workspaces: commonized-tokyo )

# us/ec2.tf

provider "aws" {
  region = "us-west-2"
}

resource "aws_instance" "web-us" {
  ...
  tags = {
    Name = "HelloUS"
  }
}

# tokyo/ec2.tf

provider "aws" {
  region = "ap-northeast-1"
}

resource "aws_instance" "web-tokyo" {
  ...
  tags = {
    Name = "HelloTokyo"
  }
}

# commonized/ec2.tf

provider "aws" {
  region = "${var.region}"
}

resource "aws_instance" "web" {
  ...
  tags = {
    Name = "${var.name}"
  }
}

# commonized/variables-us.tfvars
region = "us-west-2"
name = "HelloUS"

# commonized/variables-tokyo.tfvars
region = "ap-northeast-1"
name = "HelloTokyo"

The need to apply the migration to only some workspaces is in the following areas:
In the process of refactoring, it is a unified process that the naming is not unified.

In this example, it is the case of the following migration

• "aws_instance.web-us" of us -> "aws_instance.web" of commonized-us
• "aws_instance.web-tokyo" of tokyo -> "aws_instance.web" of commonized-tokyo

The migration file looks like this.

migration "multi_state" "mv_us" {
  from_dir = "us"
  from_workspace = "us"
  to_dir = "commonized"
  to_workspace = "commonized-us"
  to_varfile = "./variables-us.tfvars"
  actions = [
    "mv aws_instance.web-us aws_instance.web",`
  ]
}

migration "multi_state" "mv_tokyo" {
  from_dir = "tokyo"
  from_workspace = "tokyo"
  to_dir = "commonized"
  to_workspace = "commonized-tokyo"
  to_varfile = "./variables-tokyo.tfvars"
  actions = [
    "mv aws_instance.web-tokyo aws_instance.web",`
  ]
}

I think using TF_WORKSPACE or TF_CLI_ARGS_name is a very smart way to do it.

But at this point, I don't know how difficult it is to implement.
After confirming the implementation of tfmigrate, we propose the implementation.

[minamijoyo]

@takanakahiko Thank you for sharing your use-case! It looks like merging multiple root modules (directories) into a single root module (directory) with workspaces. I understood in such case it's useful moving a state across workspaces.

When I wrote the above comment, I was looking for the best solution for supporting workspace. However, I read lots of issues around the workspace feature and reached a conclusion that there is no single solution to meet all use-cases. I can imagine someone may want to merging and splitting workspaces, at the same time others may want to renaming resource in all workspaces.

We have two options, but I think they actually doesn't conflict with each other.

(1) one migration file per one workspace:
Explicitly define workspace name (and variable file) in a migration file. This is exactly what you proposed.

• pros: Declarative and flexible. It allows us a fine-grain control.
• cons: It requires duplicated copy migration files if we want to apply the migration to all workspaces. It may be boring, but not so difficult.

(Note for me): I'm working on a migration history management feature in #2, which will allow us to apply all unapplied migrations. In your use-case, it doesn't a matter. However, If deployment pipelines and stages are separated with each workspace, we cannot apply all at once. A possible workaround is split migration file directories and history files per workspace. This means a path of setting file for tfmigrate can be passed from a command line argument.

(2) one migration file per one root module
Implicitly apply a migration file to the current workspace. This is what I commented the above.

• pros: Simple and consistent. Enforce exactly the same migration to all workspaces. It will probably easily support for history management because all we need to do is splitting a history file per workspace.
• cons: Some operations require different arguments, for example importing resource requires identifiers which may be different values across workspaces. If we want to support it a consistent way, we probably need to introduce a local variable and an expression in a migration file. It's not impossible, but it will add complexity.

I assumed using TF_WORKSPACE and TF_CLI_ARGS_name would work with the current implementation as it was, but it was wrong. When switching a remote backend to local, Terraform also requires the same workspace for local. Apparently we don't have it, it causes an initialization error. We probably need to create a temporary workspace in local. This problem also affects (1), so when (1) is implemented, I think we will get (2) for free (except importing.)

To sum up the above, I think your proposal, that is, adding workspace and varfile attributes to migration state type and from_workspace/to_workspace and from_varfile/to_varfile attributes to migration multi_state type in the migration file format, looks reasonable and acceptable, but I'm not sure how to implement it, we will need to find a way to switching backend to local with workspace.

[minamijoyo]

@takanakahiko I studied how to integrate workspace with tfmigrate and found that there are some limitations we should consider:

• terraform workspace select command returns an error if a given workspace doesn't exist.
• When switching backend to local, terraform init -reconfigure automatically selects the current workspace, but it returns an error if the workspace doesn't exist in local. However, we cannot run terraform workspace new before terraform init. It's a chicken and egg problem.
• When TF_WORKSPACE is exported, we cannot select other workspace, the environment variable takes precedence over an argument. The current implementation of tfmigrate can set per directory, not per command execution. This means we should not use TF_WORKSPACE.
• The local backend always has default workspace. So we should select default workspace before switching backend to local.
• A name of the currently selected workspace can be referred as ${terraform.workspace} interpolation in configurations, so we cannot always use default workspace for a temporary workspace, we should also use the exactly same workspace name in local, so we need to create a temporary workspace and delete it when rollback.
• Using workspace with the local backend creates terraform.tfstate.d directory, we should also delete it when rollback.
• The currently selected workspace is internally recorded in .terraform/environment file, it depends on which computer and when runs tfmigrate command. This means we cannot implicitly assume the initial workspace to default or the workspace attribute in a migration file. We should save an initial workspace before explicitly selecting the workspace and restore it even if tfmigrate fails.

With the above in mind, an implementation probably looks like the following:

1. Save an initial workspace
2. Select a given target workspace
3. Pull the current state from remote
4. Select default workspace
5. Switch backend to local
6. Create a temporary workspace in local with the same name
7. Compute a new state with some state operations
8. Check plan with a given variable file
9. Select default workspace
10. Delete the temporary workspace
11. Switch backend to remote
12. Select the target workspace
13. Push the new state to remote
14. Restore the initial workspace

To do this, we probably need to implement the following:

1. Add terraform workspace show, new, and delete commands to tfexec.TerraformCLI
2. Extend tfexec.terraformCLI.OverrideBackendToLocal() and tfmigrate.setupWorkDir() to support workspace switching
3. Add workspace attribute to tfmigrate.StateMigrator, fromWorkspace and toWorkspace to tfmigrate.MultiStateMigrator, and also add them to config.StateMigratorConfig and config.MultiStateMigratorConfig
4. Pass the workspace to tfmigrate.setupWorkDir().
5. Add varfile attribute to tfmigrate.StateMigrator, fromVarfile and toVarfile to tfmigrate.MultiStateMigrator, and also add them to config.StateMigratorConfig and config.MultiStateMigratorConfig
6. Pass the varfile to tfexec.TerraformCLI.Plan()

I think it will require other minor changes, but the grand design is not so wrong.

[takanakahiko]

Thank you for your reply!
I'm too busy to take time, so I'll reply if I can afford it. 🙇

[Gowiem]

Hey @minamijoyo, any plan to support workspaces in the foreseeable future? I see that as being the missing piece to this otherwise awesome tool.

[minamijoyo]

Hi, @Gowiem. Thank you for your feedback!

I understand the workspace feature should be supported, but honestly it's not high-priority for me. I'm working on multiple projects in my spare time and other things for now. So I cannot say any ETA, sorry 🙏

For anyone who want it,
please upvote 👍 this feature request to prioritize or send a pull request by yourself, which is always appreciate ❤️

Note: The internal structure has changed with refactoring since my last comment, but I think the implementation is still technically possible.

Thanks!

[Gowiem]

@minamijoyo appreciate getting back and totally get other projects taking over. Hopefully someone will pick this up one day!

[whiskeysierra]

I'm only interested in support for var-file. Is there a way to pass env vars to terraform right now? I tried setting TF_CLI_ARGS_import as well as TF_CLI_ARGS, but none of them is being picked up.

I haven't tried it yet, but maybe TFMIGRATE_EXEC_PATH would work? Alternatively, I could probably just pass my variables via *.auto.tfvars.json (see https://www.terraform.io/docs/language/values/variables.html#variable-definitions-tfvars-files).

[minamijoyo]

@whiskeysierra Yes, you can pass Terraform variables to terraform CLI with auto tfvar files you mentioned.

Or, to pass environment variables (e.g. TF_CLI_ARGS_import ) to terraform CLI, just exporting them probably works. If you want to customize them per directory, use direnv and set TFMIGRATE_EXEC_PATH to direnv exec . terraform.

If you still have trouble, could you open a new issue with the minimal reproduction case to clarify what you are trying?

[whiskeysierra]

Or, to pass environment variables (e.g. TF_CLI_ARGS_import ) to terraform CLI, just exporting them probably works.

I tried that, but it didn't seem to work.

[whiskeysierra]

I tried that, but it didn't seem to work.

Correction, it does work.

[kertzi]

@takanakahiko Thank you for the proposal!

Adding workspace and variable file support is ok because they are common use-case for Terraform, but, I don't use them in my work, so I'm not sure what is the best design for refactoring tfstate with the workspace model.

If my understanding is correct, using workspace has a single configuration and multiple states. Workspaces are usually related to environments such as prod, test and dev, a configuration change will be applied in a workflow for each environment.

When we want to rename a resource in configuration, we should move the resource in all states. Wouldn't it be better to apply one migration to all workspaces in each workflow? The workspace model enforces us exactly equal configurations across workspaces. I don't think it makes sense to apply migration only to some workspaces. I think a migration should correspond to a configuration, and switching workspaces should be implemented by changing the execution environment not to declare it in the migration.

If so, we can automatically select workspace in each workflow with an environment variable TF_WORKSPACE and switch variable files with TF_CLI_ARGS_name.
https://learn.hashicorp.com/tutorials/terraform/automate-terraform#multi-environment-deployment
https://www.terraform.io/docs/commands/environment-variables.html#tf_cli_args-and-tf_cli_args_name

What do you think of it?

I have exactly this use case and I think this is how many of users use them. Of course there might be other cases when temporary workspace is created for testing stuff but I think then there is no need for run migrations.

We use same configuration for multiple states.
Example:
production/

• eu-central-1 have own state file in s3
• eu-north-1 have own state file in s3

staging/

• eu-central-1 have own state file in s3
• eu-north-1 have own state file in s3
  and so on...

[kertzi]

I'm totally new to Go but I investigated this a bit.
One way to solve workspace problem is to use -migrate-state flag in init commands. It should migrate selected workspace to local and vice versa.
Only problem is that terraform will ask confirmation and -input=false doesn't work correctly.
So command must be prefixed like: echo "yes" | terraform .. when terraform will get input it needs.

So final init command should look something like this:

echo "yes" | terraform init -no-color -migrate-state

I didn't find a way how to implement this to code right away because this my first experience with Go language.

[minamijoyo]

Hi, all. The tfmigrate v0.2.6 has been released and it added an initial support of workspace for multi_state block (#31). It doesn't fully cover all features described above, but an awesome progress. Thanks to @dhaven.

[minamijoyo]

Hi, all. The tfmigrate v0.3.0 now supports workspace for single state block (#61). Thanks to @bhavanki.