New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

terraform init seems to not support basic terraform best practice #18632

Open
gtmtech opened this Issue Aug 8, 2018 · 6 comments

Comments

Projects
None yet
5 participants
@gtmtech

gtmtech commented Aug 8, 2018

Terraform all versions.

It's terraform best practice to store environment config in tfvars files . e.g:

    environments/
        |-- dev.tfvars      # store dev configuration
        |-- prod.tfvars    # store stage configuration
    resources/
        |-- main.tf           # resources etc.

(Nice clean separation of environment config from other config from terraform resources - good!)

And to use e.g. terraform plan -var-file environments/dev.tfvars

But (in terms of AWS at least) its also best practice to separate dev, prod in terms of aws accounts (for one, you like to have business/enterprise support on prod which is charged as a %age of complete account cost, so having everything in the same account is very expensive for support; for another its good for security boundaries anyway as some security fundamentals can only be applied on an account basis). This yields a natural fit of storing state in a different S3 bucket for each environment too - eg terraform-dev in dev, terraform-prod in prod. (If you dont do this, then you have far more cross-account assume-roles to do, because the state has to live in one account, whilst the terraforming is done in another account, but even if you do store in the same bucket the following still applies...)

Up to now, all fine and nice and clean, but terraform init doesn't cope with this at all. Why? because terraform init is rigid and has to use .terraform locally to store its local version of a single state.

Therefore a backend that looks like this:

terraform {
  backend "s3" {
#   The bucket cannot be specified here as it changes between environments
    dynamodb_table = "terraform_lock"
    encrypt        = true
    key            = "main.tfstate"
    region         = "eu-west-1"
  }
}

Should work in conjunction with terraform init -no-input -backend-config=bucket=terraform-dev Likewise it should work when "switching" with terraform init -no-input -backend-config=bucket=terraform-prod

But it's made needlessly complicated because firstly the second terraform init needs to see both buckets if you dont add -reconfigure (For some reason, I guess it wants to support state migration?):

Prior to changing backends, Terraform inspects the source and destination
states to determine what kind of migration steps need to be taken, if any.
Terraform failed to load the states. The data in both the source and the
destination remain unmodified. Please resolve the above error and try again.

and secondly, with -reconfigure, it introduces a huge delay anyway as downloading modules takes a while, and syncing/reading the state each time from a different place takes a while, and so changing the environments makes for slow and frustrating terraform experience.

Stored state should be in different files or buckets for each environment, otherwise anyone read/writing dev state can also read/write prod state and you have no graduated security in your team (which is fine in some places but not all). But constantly having to redo the terraform init every time you switch from dev to prod is a complete pain. Workspaces are also not the answer to this problem (once again storing stuff in the same place?)

Something as simple as allowing different places for the .terraform directory would completely fix this. ( #3503 ) I could then init twice - one for dev, one for prod, store them both and be a happy terraformer just by means of an environment variable- Not only this, but it brings so many advantages because it allows you to run multiple terraforms concurrently against different environments on the same source tree (terraform init completely blocks this having a single place).

This seems like such a trivial enhancement, can this be fixed? Or can others elaborate on what they do? Maybe I'm missing something obvious. My requirements are:

  • Store state remotely in different places for each environment for security reasons
  • Not have to run terraform init every time I switch environments as it takes ages

with a bonus (not requirement) that:

  • I can run terraform in dev and prod concurrently off the same working tree

Right now I work around this with symlinks and copying the whole working directory to a tmp location. It really sucks. But environments are a fundamental part of terraform so what am I missing?

Thanks!

@apparentlymart you might be interested in this?

@myungmobile

This comment has been minimized.

Show comment
Hide comment
@myungmobile

myungmobile commented Aug 8, 2018

+1

@apparentlymart

This comment has been minimized.

Show comment
Hide comment
@apparentlymart

apparentlymart Aug 11, 2018

Contributor

Hi @gtmtech! Thanks for writing this up.

In terms of customizing the location of the .terraform directory to start: this isn't directly possible, but you can get the same effect by using multiple working directories:

(assuming the current working directory is the one containing the root module config files)
$ mkdir prod
$ cd prod
$ terraform init .. -backend-config=bucket=terraform-dev
...
$ terraform apply ..
...

The above, while admittedly not intuitive, will allow you a separate .terraform directories for a single config within Terraform's current functionality.

I expect we could accept a PR to make the .terraform directory location configurable via an environment variable; I can't think of a reason right now why that would make any future development harder. With that said, I have some broader feedback on the approach you've sketched out here...


I'd actually recommend a different layout as a "best practice":

    environments/
        |-- dev/ # dev configuration
            |-- dev.tf
        |-- prod/ # prod configuration
            |-- prod.tf
    resources/ # shared module for elements common to all environments
        |-- main.tf

Then environments/prod/prod.tf might look like this:

terraform {
  backend "s3" {
    bucket         = "terraform-prod"
    dynamodb_table = "terraform_lock"
    encrypt        = true
    key            = "main.tfstate"
    region         = "eu-west-1"
  }
}

module "main" {
  source = "../resources"

  # per-environment settings here, as you had in the .tfvars files in your example
}

By having a separate root module for each environment, you can gather together all of the necessary information in a single place that is separate for each one:

  • The complete backend configuration (no need for terraform init arguments)
  • The variable values
  • Once you run terraform init in that dir, also the .terraform directory.

To switch between environments, you just switch directories, and there's no risk at all of accidentally tangling up any of the above and applying the wrong thing to the wrong target.

This also allows for more elaborate differences between environments in situations where it's warranted. For example, the access control settings might be configured entirely differently for each environment even though the main infrastructure is the same, because you generally want a more restrictive configuration for production than you would for a development environment. To model that, you can put the access control resources directly inside environments/dev and environments/prod, while still sharing the resources module across them both.

Workspaces are not a good fit for modelling environments, because they only address part of the problem: separating the state. This is one reason why they were renamed from "environments" to "workspaces" in an earlier release, since the initial name gave the impression that they were analogous to what most teams mean when they say "environment": an entirely isolated set of infrastructure.

I'm aware that we have some leftover old documentation that still uses example workspace names like "dev" and "prod" which give the impression that this is the intended use of workspaces. We intend to fix this, and have started by revising the When To Use Workspaces section to be a lot more explicit about what workspaces are suited for and what they are less suited for.

The purpose that workspaces were designed to serve -- temporary copies of an infrastructure during development -- is something that we now know most teams don't do in practice, since it's often too expensive to give each developer a separate stack and more reasonable to simply have a shared, long-lived "staging" or "dev" environment to rehearse changes. A shared long-lived environment has its own problems, such as that different developers will often conflict with one another, and so we intend to revisit this in a future major release to find a better workflow that works well for safely developing, testing, and deploying changes.

In the mean time, I think most teams should not use workspaces, and should instead use the module-per-environment pattern I described above. We intend to describe this in more detail in the documentation once we get past our current focus for the 0.12 release of improving the configuration language. (Indeed, many of the improvements in 0.12 are aimed at making it easier to write configurable modules, which will hopefully reduce some of the friction that the above suggested pattern has today due to issues like not being able to pass complex data structures into and out of modules.)

Thanks again for sharing this! We definitely do intend to revisit this in a later release and find a more intuitive workflow, since we're aware that there is lots of friction right now. I expect that we will lean towards removing remaining friction from the above pattern, whatever that might mean, because it gives access to the full power of the configuration language when defining an environment rather than cobbling together variables files and command line arguments.

Contributor

apparentlymart commented Aug 11, 2018

Hi @gtmtech! Thanks for writing this up.

In terms of customizing the location of the .terraform directory to start: this isn't directly possible, but you can get the same effect by using multiple working directories:

(assuming the current working directory is the one containing the root module config files)
$ mkdir prod
$ cd prod
$ terraform init .. -backend-config=bucket=terraform-dev
...
$ terraform apply ..
...

The above, while admittedly not intuitive, will allow you a separate .terraform directories for a single config within Terraform's current functionality.

I expect we could accept a PR to make the .terraform directory location configurable via an environment variable; I can't think of a reason right now why that would make any future development harder. With that said, I have some broader feedback on the approach you've sketched out here...


I'd actually recommend a different layout as a "best practice":

    environments/
        |-- dev/ # dev configuration
            |-- dev.tf
        |-- prod/ # prod configuration
            |-- prod.tf
    resources/ # shared module for elements common to all environments
        |-- main.tf

Then environments/prod/prod.tf might look like this:

terraform {
  backend "s3" {
    bucket         = "terraform-prod"
    dynamodb_table = "terraform_lock"
    encrypt        = true
    key            = "main.tfstate"
    region         = "eu-west-1"
  }
}

module "main" {
  source = "../resources"

  # per-environment settings here, as you had in the .tfvars files in your example
}

By having a separate root module for each environment, you can gather together all of the necessary information in a single place that is separate for each one:

  • The complete backend configuration (no need for terraform init arguments)
  • The variable values
  • Once you run terraform init in that dir, also the .terraform directory.

To switch between environments, you just switch directories, and there's no risk at all of accidentally tangling up any of the above and applying the wrong thing to the wrong target.

This also allows for more elaborate differences between environments in situations where it's warranted. For example, the access control settings might be configured entirely differently for each environment even though the main infrastructure is the same, because you generally want a more restrictive configuration for production than you would for a development environment. To model that, you can put the access control resources directly inside environments/dev and environments/prod, while still sharing the resources module across them both.

Workspaces are not a good fit for modelling environments, because they only address part of the problem: separating the state. This is one reason why they were renamed from "environments" to "workspaces" in an earlier release, since the initial name gave the impression that they were analogous to what most teams mean when they say "environment": an entirely isolated set of infrastructure.

I'm aware that we have some leftover old documentation that still uses example workspace names like "dev" and "prod" which give the impression that this is the intended use of workspaces. We intend to fix this, and have started by revising the When To Use Workspaces section to be a lot more explicit about what workspaces are suited for and what they are less suited for.

The purpose that workspaces were designed to serve -- temporary copies of an infrastructure during development -- is something that we now know most teams don't do in practice, since it's often too expensive to give each developer a separate stack and more reasonable to simply have a shared, long-lived "staging" or "dev" environment to rehearse changes. A shared long-lived environment has its own problems, such as that different developers will often conflict with one another, and so we intend to revisit this in a future major release to find a better workflow that works well for safely developing, testing, and deploying changes.

In the mean time, I think most teams should not use workspaces, and should instead use the module-per-environment pattern I described above. We intend to describe this in more detail in the documentation once we get past our current focus for the 0.12 release of improving the configuration language. (Indeed, many of the improvements in 0.12 are aimed at making it easier to write configurable modules, which will hopefully reduce some of the friction that the above suggested pattern has today due to issues like not being able to pass complex data structures into and out of modules.)

Thanks again for sharing this! We definitely do intend to revisit this in a later release and find a more intuitive workflow, since we're aware that there is lots of friction right now. I expect that we will lean towards removing remaining friction from the above pattern, whatever that might mean, because it gives access to the full power of the configuration language when defining an environment rather than cobbling together variables files and command line arguments.

@gtmtech

This comment has been minimized.

Show comment
Hide comment
@gtmtech

gtmtech Aug 14, 2018

@apparentlymart Thanks for taking the time for a detailed explanation! I have seen the directory per environment approach used above, and appreciate the comments you make about it.

In my experience this approach leads to teams having 2 places to put resources - in the (common) resources folder, or in the (environment specific) folder. Over time teams can put more and more in the environment specific folders (as things may initially just be for one environment) and they end up with a difficult-to-compare set of resources and possibly also multiple copy+pastes across different environments with the inevitable question (why isnt production anything like staging) question being asked later down the line.

It's for this reason I prefer instead to mandate a single resources area, and use feature-flags (a variable map "flags") and copious use of count on all resources to toggle features on and off in environments, and use tfvars lists and maps to wire in other config. In this way the differences between two environments are just comparing two tfvars files which I find simpler for team to get a handle on, than having to compare trees of differently configured resources and submodules, and with a single area for resources, teams have to make sure they work across all environments, which means they dont end up putting any environment specific resources in the resource definitions themselves. This tends to lead to cleaner, DRYer code.

However I also appreciate that your approach solves the immediate problem I'm having of separate state files (although it also requires me to cd quite a lot).

I think on balance if you're up for a PR, I might see if I can do one, to use an optional env var for the location of the .terraform directory, and then we can use both approaches in different situations where they make sense for the end users?

gtmtech commented Aug 14, 2018

@apparentlymart Thanks for taking the time for a detailed explanation! I have seen the directory per environment approach used above, and appreciate the comments you make about it.

In my experience this approach leads to teams having 2 places to put resources - in the (common) resources folder, or in the (environment specific) folder. Over time teams can put more and more in the environment specific folders (as things may initially just be for one environment) and they end up with a difficult-to-compare set of resources and possibly also multiple copy+pastes across different environments with the inevitable question (why isnt production anything like staging) question being asked later down the line.

It's for this reason I prefer instead to mandate a single resources area, and use feature-flags (a variable map "flags") and copious use of count on all resources to toggle features on and off in environments, and use tfvars lists and maps to wire in other config. In this way the differences between two environments are just comparing two tfvars files which I find simpler for team to get a handle on, than having to compare trees of differently configured resources and submodules, and with a single area for resources, teams have to make sure they work across all environments, which means they dont end up putting any environment specific resources in the resource definitions themselves. This tends to lead to cleaner, DRYer code.

However I also appreciate that your approach solves the immediate problem I'm having of separate state files (although it also requires me to cd quite a lot).

I think on balance if you're up for a PR, I might see if I can do one, to use an optional env var for the location of the .terraform directory, and then we can use both approaches in different situations where they make sense for the end users?

@apparentlymart

This comment has been minimized.

Show comment
Hide comment
@apparentlymart

apparentlymart Aug 14, 2018

Contributor

Thanks for that extra context, @gtmtech!

It is true that the module-per-environment approach does rely on humans to police themselves and each other (complying with a policy on which resources -- if any -- belong in the environment-specific module) whereas using variables implicitly forces that through the limitations of a .tfvars file. Our usual attitude is that Terraform should encourage a best-practice but give you room to step out of it when it's not appropriate, and so we generally lean towards the "trust the humans" angle when designing features.

However, given that moving the .terraform directory doesn't seem like it would create any particular harm (either directly for end-users or making future development more difficult) I'm happy to be flexible and make room for this alternative approach, even though I don't feel ready to call it a "best practice" yet. (Good experience reports may change my mind! 😀)

I think it is probably best to wait until the long-running 0.12 development branch is merged into master before starting a PR on this, since I expect they'll cover similar ground and we'd end up having to resolve merge conflicts down the road otherwise. However, once that is true (still gonna be at least a few weeks away though, I expect 😖) we'd be happy to review a PR for this.

Thanks again for writing this up, and for the interest in working on a PR!

Contributor

apparentlymart commented Aug 14, 2018

Thanks for that extra context, @gtmtech!

It is true that the module-per-environment approach does rely on humans to police themselves and each other (complying with a policy on which resources -- if any -- belong in the environment-specific module) whereas using variables implicitly forces that through the limitations of a .tfvars file. Our usual attitude is that Terraform should encourage a best-practice but give you room to step out of it when it's not appropriate, and so we generally lean towards the "trust the humans" angle when designing features.

However, given that moving the .terraform directory doesn't seem like it would create any particular harm (either directly for end-users or making future development more difficult) I'm happy to be flexible and make room for this alternative approach, even though I don't feel ready to call it a "best practice" yet. (Good experience reports may change my mind! 😀)

I think it is probably best to wait until the long-running 0.12 development branch is merged into master before starting a PR on this, since I expect they'll cover similar ground and we'd end up having to resolve merge conflicts down the road otherwise. However, once that is true (still gonna be at least a few weeks away though, I expect 😖) we'd be happy to review a PR for this.

Thanks again for writing this up, and for the interest in working on a PR!

@davidgoate

This comment has been minimized.

Show comment
Hide comment
@davidgoate

davidgoate Aug 26, 2018

@apparentlymart A couple of questions about the recommended structure above:

    environments/
        |-- dev/ # dev configuration
            |-- dev.tf
        |-- prod/ # prod configuration
            |-- prod.tf
    resources/ # shared module for elements common to all environments
        |-- main.tf

using example dev.tf

module "main" {
  source = "../resources"

  # per-environment settings here, as you had in the .tfvars files in your example
}
  1. Is the first source attribute meant to be ../../resources instead?
  2. How do outputs work in this structure? I used to have an outputs.tf in the root directory and now I see no output when running

terraform output
The state file either has no outputs defined, or all the defined
outputs are empty. Please define an output in your configuration
with the output keyword and run terraform refresh for it to
become available. If you are using interpolation, please verify
the interpolated value is not empty. You can use the
terraform console command to assist.

My old structure was:

        |-- file1.tf
        |-- file2.tf
        |-- variables.tf
        |-- outputs.tf

Now I have:

    environments/
        |-- dev/ 
            |-- dev.tf
        |-- prod/ 
            |-- prod.tf
    resources/ # Everything that used to be at the root directory level
         |-- file1.tf
         |-- file2.tf
         |-- variables.tf
         |-- outputs.tf

My dev.tf looks like this:

terraform {
  backend "s3" {
    ...
  }
  version = "~> 0.11.8"
}

provider "aws" {
...
}

module "main" {
  source = "../../resources"
}

When I run from the dev directory terraform apply or terraform output I see no outputs, is there a way to "import" the outputs.tf so that all output are printed?

davidgoate commented Aug 26, 2018

@apparentlymart A couple of questions about the recommended structure above:

    environments/
        |-- dev/ # dev configuration
            |-- dev.tf
        |-- prod/ # prod configuration
            |-- prod.tf
    resources/ # shared module for elements common to all environments
        |-- main.tf

using example dev.tf

module "main" {
  source = "../resources"

  # per-environment settings here, as you had in the .tfvars files in your example
}
  1. Is the first source attribute meant to be ../../resources instead?
  2. How do outputs work in this structure? I used to have an outputs.tf in the root directory and now I see no output when running

terraform output
The state file either has no outputs defined, or all the defined
outputs are empty. Please define an output in your configuration
with the output keyword and run terraform refresh for it to
become available. If you are using interpolation, please verify
the interpolated value is not empty. You can use the
terraform console command to assist.

My old structure was:

        |-- file1.tf
        |-- file2.tf
        |-- variables.tf
        |-- outputs.tf

Now I have:

    environments/
        |-- dev/ 
            |-- dev.tf
        |-- prod/ 
            |-- prod.tf
    resources/ # Everything that used to be at the root directory level
         |-- file1.tf
         |-- file2.tf
         |-- variables.tf
         |-- outputs.tf

My dev.tf looks like this:

terraform {
  backend "s3" {
    ...
  }
  version = "~> 0.11.8"
}

provider "aws" {
...
}

module "main" {
  source = "../../resources"
}

When I run from the dev directory terraform apply or terraform output I see no outputs, is there a way to "import" the outputs.tf so that all output are printed?

@davidgoate

This comment has been minimized.

Show comment
Hide comment
@davidgoate

davidgoate Aug 26, 2018

@apparentlymart I think I figured out a way to get the outputs in the structure, from within the dev directory I now run terraform output --module=main

davidgoate commented Aug 26, 2018

@apparentlymart I think I figured out a way to get the outputs in the structure, from within the dev directory I now run terraform output --module=main

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment