Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

terraform get: can't use variable in module source parameter? #1439

Closed
amaczuga opened this issue Apr 9, 2015 · 124 comments
Closed

terraform get: can't use variable in module source parameter? #1439

amaczuga opened this issue Apr 9, 2015 · 124 comments

Comments

@amaczuga
Copy link

@amaczuga amaczuga commented Apr 9, 2015

I'm trying to avoid hard-coding module sources; the simplest approach would be:

variable "foo_module_source" {
  default = "github.com/thisisme/terraform-foo-module"
}

module "foo" {
  source = "${var.foo_module_source}"
}

The result I get while attempting to run terraform get -update is

Error loading Terraform: Error downloading modules: error downloading module 'file:///home/thisisme/terraform-env/${var.foo_module_source}': source path error: stat /home/thisisme/terraform-env/${var.foo_module_source}: no such file or directory
@radeksimko
Copy link
Member

@radeksimko radeksimko commented Apr 9, 2015

I'm trying to avoid hard-coding module sources

Is there any particular reason behind that? Do you expect some modules to have the same interface, so you can swap these?

@mitchellh
Copy link
Member

@mitchellh mitchellh commented Apr 9, 2015

This is as intended. We should add validation that this isn't allowed.

The reason is simply that it breaks our compile -> semantic check -> execute loop. i.e. imagine if your C code could arbitrarily download new C files during compile/execution. Would be weird.

@amaczuga
Copy link
Author

@amaczuga amaczuga commented Apr 9, 2015

Do you expect some modules to have the same interface

yes, that is exactly my point - for the flexible running plans against various versions/forks of identically interfaced modules, without refactoring base terraform code

@amaczuga
Copy link
Author

@amaczuga amaczuga commented Apr 9, 2015

This is as intended.

Er. Forgive me - I'm lost here, due to labels - that is - marked bug, yet your comment suggest a wontfix

@radeksimko
Copy link
Member

@radeksimko radeksimko commented Apr 9, 2015

marked bug, yet your comment suggest a wontfix

The fix is to add the validation so you get something a bit more clear rather than "error downloading module" I guess.

@clstokes
Copy link
Contributor

@clstokes clstokes commented Apr 16, 2015

FWIW, this is something I wanted to do as well and found wasn't supported. In my case, I wanted to avoid duplicating git::ssh://git@github.com/... across tens or hundreds of files and do something like source = "${var.module_path}//modules/common-vpc".

@clstokes
Copy link
Contributor

@clstokes clstokes commented Apr 16, 2015

Also to set the branch/tag via a variable would be helpful...

@radeksimko
Copy link
Member

@radeksimko radeksimko commented Apr 16, 2015

Also to set the branch/tag via a variable would be helpful...

See #1145

@clstokes
Copy link
Contributor

@clstokes clstokes commented Apr 16, 2015

@radeksimko I'm familiar with ref as added in a recent version, but I'm suggesting something like source = "github.com/clstokes/terraform-modules//modules/common-vpc?ref=${var.module_branch}".

Are variables allowed at all in modules sources?

@mitchellh
Copy link
Member

@mitchellh mitchellh commented Apr 18, 2015

@clstokes They're not yet

@ketzacoatl
Copy link
Contributor

@ketzacoatl ketzacoatl commented May 12, 2015

+1 on this. I want admins and automated-ci to be able to specify the local path, allow flexibility to pull from git or filesystem, etc, but this is not possible without allowing interpolation in the source param.

@mitchellh
Copy link
Member

@mitchellh mitchellh commented May 14, 2015

This is not a bad idea but it is very hard to do with the current architecture of how modules work with Terraform. It also shifts a lot of potential errors away from a compile-time error to a runtime error, which we've wanted to avoid. I'm going to keep this tagged with "thinking"

@ketzacoatl
Copy link
Contributor

@ketzacoatl ketzacoatl commented May 16, 2015

@mitchellh, how are compile-tile and runtime differentiated in Terraform? Are you referring to tf plan vs tf apply? If this is the case, I would like to share my experience as a user has never built confidence in tf apply succeeding if tf plan succeeds.

Said another way, TF as it is right now gives me a lot of compile time and runtime errors. For example, you can easily tell TF to create an SSH key that seems fine with tf plan but errors out with tf apply. Either way, my vote for unblocking this capability (understanding it isn't simple, given current architecture) stems from wanting the ability (as a user) to choose whether or not a variable in the module source is a good decision for my code. Thanks for listening :)

@pikeas
Copy link
Contributor

@pikeas pikeas commented Jun 10, 2015

+1, I understand why this may be architecturally tricky to get right, but it would be great to have on the admin/DRY side of things.

@kaelumania
Copy link

@kaelumania kaelumania commented Jun 17, 2015

+1 I also think that the gained flexibility would outweigh the disadvantages.

@kokovoj
Copy link

@kokovoj kokovoj commented Sep 1, 2015

👍

Use-case for this would be allowing for the flexibility to store module source in a variable for :

a. module source pointing at a corporate source control behind a corporate VPN

variable "your_project_source" {
  default = "https://your_src_system/your_project//terraform"
}

OR
b. use a local path on the dev box (after that src was already checked out locally, so don't need to be on the corporate VPN)

variable "your_project_source" {
  default = "/Users/joeshmoe/projects/your_project/terraform"
}

(and overriding one or the other in terraform.tfvars) and then

module "your_project" {
  source = "${var.your_project_source}"
  ...
}

@apparentlymart
Copy link
Member

@apparentlymart apparentlymart commented Sep 2, 2015

One very specific complexity with this is that currently modules need to be pre-fetched using terraform get prior to terraform plan, and currently that command does not take any arguments that would allow you to set variables. By the time plan is running, Terraform is just thinking about the module name and paying no attention to the module source, since the module is assumed to already be retrieved into the .terraform subdirectory.

Perhaps in some cases this could be worked around by breaking a configuration into two separate runs, with an initial run creating a remote state that can be consumed by the second run. Since terraform_remote_state is just a regular resource its configuration arguments can be interpolated, even by things that aren't known until apply time, as long as a dependency cycle doesn't result.

This is of course not as convenient as creating everything in one step using directly-referenced modules, but maybe it's a reasonable workaround for some situations in the mean time.

@apparentlymart
Copy link
Member

@apparentlymart apparentlymart commented Sep 2, 2015

@kokovoj 's use-case, of switching to a different version in a development environment, got me thinking about how that gets solved in other languages.

When I have a problem like that in e.g. Go, NodeJS or Python I don't use any runtime features to solve it, but rather I just ignore the location/version of the module given in the dependency list and just install whatever one I want, exploiting the fact that (just like in Terraform) the "get" step is separated from the "compile" and "run" steps, and so we can do manual steps in between to arrange for the versions we want.

Terraform obscures this ability a little by storing the local modules in a directory named after the MD5 hash of the module name under the .terraform directory, so it's harder to recognize which one is which by eye... but you can, if you locate the right one, install it from a different source or modify it in-place. (I've done this several times while debugging, in fact.)

So with all of this said, perhaps Terraform could just be a little more transparent about where it looks for modules and embrace the idea that terraform get just installs the default module locations, but it's fine to manually install from other locations, or even to write your own separate tool to install from wherever you want. If we went this route, the only thing that would need to change in Terraform is to switch to a more user-friendly on-disk module representation and to commit not to change it in future versions of Terraform.

(It would also be nice to extend terraform get to be able to handle certain overrides itself, but that is made more complex by the fact that there can be nested modules that have their own dependencies, and so such syntax would probably end up quite complicated if it had to happen entirely on the command line.)

@lorengordon
Copy link
Contributor

@lorengordon lorengordon commented Jun 19, 2018

Personally I'd love to see interpolation for the entire source parameter. The chosen direction to implement support for just the version is very limiting. I'd rather like to pull all my source definitions to the top of a configuration, in a locals definition, so I don't have to go hunting through every file to find/update the string. I expect it would make modules much more maintainable overall.

@pixelicous
Copy link

@pixelicous pixelicous commented Jun 28, 2018

@lorengordon I agree.. this is nonsense.. that and the fact that everytime you pull a whole repository instead of a leaf

@luisdavim
Copy link

@luisdavim luisdavim commented Jun 28, 2018

We use this http://bensnape.com/2016/01/14/terraform-design-patterns-the-terrafile/ I think it would be reasonable to have something like that natively.

@jjshoe
Copy link

@jjshoe jjshoe commented Jul 2, 2018

@mitchellh - It would be great if hashicorp could re-look at this. Though it's been closed, and split into two cases, which don't address all the reasons for this, it's more commented then any current open issue.

@akvadrako
Copy link

@akvadrako akvadrako commented Jul 9, 2018

The best workaround I have found is by using putting something like this in override.tf

module "core" {
    source = "/Users/dev/terraform-modules/core"
}
...

Not ideal, but seems to work.

@MichaelDeCorte
Copy link

@MichaelDeCorte MichaelDeCorte commented Jul 10, 2018

@mitchellh agreement with @jjshoe the original issue of allowing interpolation for the source parameter has not been addressed. This issue should be opened, or a new one forked off. But it should not be closed.

@MichaelDeCorte
Copy link

@MichaelDeCorte MichaelDeCorte commented Jul 10, 2018

@akvadrako I'm not following your workaround. Can you elaborate?

@akvadrako
Copy link

@akvadrako akvadrako commented Jul 10, 2018

@MichaelDeCorte It's just that it's possible to override the module source parameters with an external file. My use case is module development, where I want to replace several references to git repos with local checkouts.

@MichaelDeCorte
Copy link

@MichaelDeCorte MichaelDeCorte commented Jul 10, 2018

@mitchellh elaborating an example to allow the for absolute paths relative to TF-Home. Assume the below directory / file structure. This is a common pattern where repo1 is a shared repository that is downloaded locally via a script as a workaround for the source interpolation issue.

TF-Home /

  • app1.tf
  • appSubDir1
    • app2.tf
  • repo1 /
    • foo1.tf
    • foo2.tf

Assume that app1, app2 and foo1.tf all depend on foo2. The source parameter would be:
app1: repo1/foo2.tf
app2: ../repo1/foo2.tf
foo1: foo2.tf

I hope it's clear that its not great.

@pixelicous
Copy link

@pixelicous pixelicous commented Jul 29, 2018

@akvadrako
Guys the best method to get around it is to wrap your terraform in a script.

Our powershell wrapper does so many things to over come terraform restrictions, we cant use terraform without, basically we did something like the guys in terragrunt did, plus many more addons on it, i cant understand how somebody can even use terraform as is out of the box without some interpolation in those missing places..

anyhow, i really hope hashicorp will decide to change some parts of the product, because it is really constricting, some of those things should have been thought of much before

@luisdavim
Copy link

@luisdavim luisdavim commented Jul 29, 2018

Yeah, we've been using the Terrafile approach (see my comment above) it works pretty well but it forces us to use a wrapper script, I think that the Terrafile pattern should be supported by Terraform.
This could easily be added to the get phase.

@nikolay
Copy link

@nikolay nikolay commented Sep 4, 2018

The rationale to disallow this so that intelligent people can't download random modules is the same as not having a division operator as somebody may decide to divide by zero one day.

@tuusberg
Copy link

@tuusberg tuusberg commented Oct 20, 2018

+1. Changing module versions manually is error prone.

@midacts
Copy link

@midacts midacts commented Oct 21, 2018

@richardgavel
Copy link

@richardgavel richardgavel commented Nov 14, 2018

There is a similar issue in not being able to use interpolation syntax when providing configuration for back ends (say S3 bucket/region). You get around that by using terraform init -backend-config so that value is known at the beginning of the lifecycle. Couldn't something be done similarly (provide the value as some kind of command line param)?

@discreet
Copy link

@discreet discreet commented Nov 28, 2018

This is something I've been wanting for a while and have been thinking a lot about. The use case I have is I wrote a bunch of terraform code to deploy a kubernetes cluster. In my code I have a variables module which lives in a git repo and contains all my input variables based on region and environment. This allows me to use the same exact code to deploy my kubernetes cluster to multiple AWS account and into multiple regions and environments with only changing two inputs to terraform apply. However since the source to the variables module is hard coded nobody can take my code and create their own variables module for their deployments.

@josephcaxton
Copy link

@josephcaxton josephcaxton commented Dec 5, 2018

I need to be able to pass variable. Is Hashcorp looking to resolve this issue?

@GarinKartes
Copy link

@GarinKartes GarinKartes commented Dec 5, 2018

@major0
Copy link

@major0 major0 commented Dec 7, 2018

While I can understand the reasons for not supporting general var/local inclusion .. I feel that many (all?) of the above use cases could be resolved by adding${path.root} to the list of allowed local module source prefixes.

@damon-atkins
Copy link

@damon-atkins damon-atkins commented Jan 13, 2019

Or even something like source yaml_lookup://../lookupfile.yaml which contains module name and source pairs. Or some sort of cli option --source_overrides=something.yaml The value is saved in the state, and warns if anything is different to the last run.

@jmturwy
Copy link

@jmturwy jmturwy commented Jun 26, 2019

Another example as to why this is beneficial:

`####################### Global value #######################
locals {
orgname = "acmeCorp"
}

######################
module "iam" {
source = "./iam/customer/${local.orgname}"
org-name = "${local.orgname}"
providers = {
aws = "aws.customer-${local.orgname}"
}

}
######################
module "s3-bucket" {
source = "./s3/customer/${local.orgname}"
org-name = "${local.orgname}"
env = "production"
providers = {
aws = "customer-${local.orgname}"
}
}
#######################
module "vpc" {
source = "./vpc/customer/${local.orgname}"
org-name = "${local.orgname}"
env = "production"
region = "us-westt-1"
cidr-octs = "10.7"
peer-account = "xxxxxxxxxxxxxx"
peer-vpc = "vpc-xxxxxxxxxxxxxxxxx"
peer-cidr = "192.10.0.0/16"
}`

this would be called acmecorp.tf, we would just copy this module and renamed it to loonytoons.tf and change the local var to loonytoons thus saving a lot of copy pasta

@non7top
Copy link

@non7top non7top commented Jul 2, 2019

Adding to a comment by richardgavel from Nov 14, 2018

Backend configuration is stored in .terraform/terraform.tfstate, so store module sources in there and require re-init if those change, i.e something like module.cluster1.app -> source="github.com/example/example"

Having such feature is particularly useful if you want to test new module version which is located in some feature branch in another (shared) repo, you then have to edit all paths to module manually and re-init anyways.

@BattleBrisket
Copy link

@BattleBrisket BattleBrisket commented Jul 17, 2019

+1 for this. I have a git-based module to configure team permissions, and I have ~80 teams. That's a lot of wet, brittle code that won't stand up to any significant change in the repository structure.

If we cannot have the source set as a variable, could we specify some module-specific config values that would load at runtime? If I could store the git URL and a ref tag somewhere in tfvars, for example, that would meet my needs.

@anderson-dan-w
Copy link

@anderson-dan-w anderson-dan-w commented Jul 18, 2019

A use I see easily popping up (in that literally my first project that I'm working on terraform with), I want to have multiple modules that I pull from, but I will always want those to use same branch, within a project:

prod_git_tag = "v.0.0.1"
staging_git_tag = "v.0.5.7"

and then

module "vpc" {
  source = "${git_repo_path}?ref=${git_tag}"
}

module "kubernetes-cluster" {
  source = "${git_repo_path}?ref=${git_tag}"
}
 # etc...

which seems pretty reasonable to me - when I pass in git_tag=prod_git_tag, now they all reference the same git_tag and can be updated with one line, rather than in all the various places.

@ghost
Copy link

@ghost ghost commented Jul 24, 2019

I'm going to lock this issue because it has been closed for 30 days . This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@hashicorp hashicorp locked and limited conversation to collaborators Jul 24, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging a pull request may close this issue.