terraform get: can't use variable in module source parameter? #1439

Closed
amaczuga opened this Issue Apr 9, 2015 · 108 comments

Comments

Projects
None yet
@amaczuga

amaczuga commented Apr 9, 2015

I'm trying to avoid hard-coding module sources; the simplest approach would be:

variable "foo_module_source" {
  default = "github.com/thisisme/terraform-foo-module"
}

module "foo" {
  source = "${var.foo_module_source}"
}

The result I get while attempting to run terraform get -update is

Error loading Terraform: Error downloading modules: error downloading module 'file:///home/thisisme/terraform-env/${var.foo_module_source}': source path error: stat /home/thisisme/terraform-env/${var.foo_module_source}: no such file or directory
@radeksimko

This comment has been minimized.

Show comment
Hide comment
@radeksimko

radeksimko Apr 9, 2015

Member

I'm trying to avoid hard-coding module sources

Is there any particular reason behind that? Do you expect some modules to have the same interface, so you can swap these?

Member

radeksimko commented Apr 9, 2015

I'm trying to avoid hard-coding module sources

Is there any particular reason behind that? Do you expect some modules to have the same interface, so you can swap these?

@mitchellh

This comment has been minimized.

Show comment
Hide comment
@mitchellh

mitchellh Apr 9, 2015

Member

This is as intended. We should add validation that this isn't allowed.

The reason is simply that it breaks our compile -> semantic check -> execute loop. i.e. imagine if your C code could arbitrarily download new C files during compile/execution. Would be weird.

Member

mitchellh commented Apr 9, 2015

This is as intended. We should add validation that this isn't allowed.

The reason is simply that it breaks our compile -> semantic check -> execute loop. i.e. imagine if your C code could arbitrarily download new C files during compile/execution. Would be weird.

@mitchellh mitchellh added bug core labels Apr 9, 2015

@amaczuga

This comment has been minimized.

Show comment
Hide comment
@amaczuga

amaczuga Apr 9, 2015

Do you expect some modules to have the same interface

yes, that is exactly my point - for the flexible running plans against various versions/forks of identically interfaced modules, without refactoring base terraform code

amaczuga commented Apr 9, 2015

Do you expect some modules to have the same interface

yes, that is exactly my point - for the flexible running plans against various versions/forks of identically interfaced modules, without refactoring base terraform code

@amaczuga

This comment has been minimized.

Show comment
Hide comment
@amaczuga

amaczuga Apr 9, 2015

This is as intended.

Er. Forgive me - I'm lost here, due to labels - that is - marked bug, yet your comment suggest a wontfix

amaczuga commented Apr 9, 2015

This is as intended.

Er. Forgive me - I'm lost here, due to labels - that is - marked bug, yet your comment suggest a wontfix

@radeksimko

This comment has been minimized.

Show comment
Hide comment
@radeksimko

radeksimko Apr 9, 2015

Member

marked bug, yet your comment suggest a wontfix

The fix is to add the validation so you get something a bit more clear rather than "error downloading module" I guess.

Member

radeksimko commented Apr 9, 2015

marked bug, yet your comment suggest a wontfix

The fix is to add the validation so you get something a bit more clear rather than "error downloading module" I guess.

@clstokes

This comment has been minimized.

Show comment
Hide comment
@clstokes

clstokes Apr 16, 2015

Contributor

FWIW, this is something I wanted to do as well and found wasn't supported. In my case, I wanted to avoid duplicating git::ssh://git@github.com/... across tens or hundreds of files and do something like source = "${var.module_path}//modules/common-vpc".

Contributor

clstokes commented Apr 16, 2015

FWIW, this is something I wanted to do as well and found wasn't supported. In my case, I wanted to avoid duplicating git::ssh://git@github.com/... across tens or hundreds of files and do something like source = "${var.module_path}//modules/common-vpc".

@clstokes

This comment has been minimized.

Show comment
Hide comment
@clstokes

clstokes Apr 16, 2015

Contributor

Also to set the branch/tag via a variable would be helpful...

Contributor

clstokes commented Apr 16, 2015

Also to set the branch/tag via a variable would be helpful...

@radeksimko

This comment has been minimized.

Show comment
Hide comment
@radeksimko

radeksimko Apr 16, 2015

Member

Also to set the branch/tag via a variable would be helpful...

See #1145

Member

radeksimko commented Apr 16, 2015

Also to set the branch/tag via a variable would be helpful...

See #1145

@clstokes

This comment has been minimized.

Show comment
Hide comment
@clstokes

clstokes Apr 16, 2015

Contributor

@radeksimko I'm familiar with ref as added in a recent version, but I'm suggesting something like source = "github.com/clstokes/terraform-modules//modules/common-vpc?ref=${var.module_branch}".

Are variables allowed at all in modules sources?

Contributor

clstokes commented Apr 16, 2015

@radeksimko I'm familiar with ref as added in a recent version, but I'm suggesting something like source = "github.com/clstokes/terraform-modules//modules/common-vpc?ref=${var.module_branch}".

Are variables allowed at all in modules sources?

@mitchellh

This comment has been minimized.

Show comment
Hide comment
@mitchellh

mitchellh Apr 18, 2015

Member

@clstokes They're not yet

Member

mitchellh commented Apr 18, 2015

@clstokes They're not yet

@ketzacoatl

This comment has been minimized.

Show comment
Hide comment
@ketzacoatl

ketzacoatl May 12, 2015

Contributor

+1 on this. I want admins and automated-ci to be able to specify the local path, allow flexibility to pull from git or filesystem, etc, but this is not possible without allowing interpolation in the source param.

Contributor

ketzacoatl commented May 12, 2015

+1 on this. I want admins and automated-ci to be able to specify the local path, allow flexibility to pull from git or filesystem, etc, but this is not possible without allowing interpolation in the source param.

@mitchellh

This comment has been minimized.

Show comment
Hide comment
@mitchellh

mitchellh May 14, 2015

Member

This is not a bad idea but it is very hard to do with the current architecture of how modules work with Terraform. It also shifts a lot of potential errors away from a compile-time error to a runtime error, which we've wanted to avoid. I'm going to keep this tagged with "thinking"

Member

mitchellh commented May 14, 2015

This is not a bad idea but it is very hard to do with the current architecture of how modules work with Terraform. It also shifts a lot of potential errors away from a compile-time error to a runtime error, which we've wanted to avoid. I'm going to keep this tagged with "thinking"

@mitchellh mitchellh added enhancement thinking and removed bug labels May 14, 2015

@ketzacoatl

This comment has been minimized.

Show comment
Hide comment
@ketzacoatl

ketzacoatl May 16, 2015

Contributor

@mitchellh, how are compile-tile and runtime differentiated in Terraform? Are you referring to tf plan vs tf apply? If this is the case, I would like to share my experience as a user has never built confidence in tf apply succeeding if tf plan succeeds.

Said another way, TF as it is right now gives me a lot of compile time and runtime errors. For example, you can easily tell TF to create an SSH key that seems fine with tf plan but errors out with tf apply. Either way, my vote for unblocking this capability (understanding it isn't simple, given current architecture) stems from wanting the ability (as a user) to choose whether or not a variable in the module source is a good decision for my code. Thanks for listening :)

Contributor

ketzacoatl commented May 16, 2015

@mitchellh, how are compile-tile and runtime differentiated in Terraform? Are you referring to tf plan vs tf apply? If this is the case, I would like to share my experience as a user has never built confidence in tf apply succeeding if tf plan succeeds.

Said another way, TF as it is right now gives me a lot of compile time and runtime errors. For example, you can easily tell TF to create an SSH key that seems fine with tf plan but errors out with tf apply. Either way, my vote for unblocking this capability (understanding it isn't simple, given current architecture) stems from wanting the ability (as a user) to choose whether or not a variable in the module source is a good decision for my code. Thanks for listening :)

@pikeas

This comment has been minimized.

Show comment
Hide comment
@pikeas

pikeas Jun 10, 2015

Contributor

+1, I understand why this may be architecturally tricky to get right, but it would be great to have on the admin/DRY side of things.

Contributor

pikeas commented Jun 10, 2015

+1, I understand why this may be architecturally tricky to get right, but it would be great to have on the admin/DRY side of things.

@kaelumania

This comment has been minimized.

Show comment
Hide comment
@kaelumania

kaelumania Jun 17, 2015

Contributor

+1 I also think that the gained flexibility would outweigh the disadvantages.

Contributor

kaelumania commented Jun 17, 2015

+1 I also think that the gained flexibility would outweigh the disadvantages.

@kokovoj

This comment has been minimized.

Show comment
Hide comment
@kokovoj

kokovoj Sep 1, 2015

👍

Use-case for this would be allowing for the flexibility to store module source in a variable for :

a. module source pointing at a corporate source control behind a corporate VPN

variable "your_project_source" {
  default = "https://your_src_system/your_project//terraform"
}

OR
b. use a local path on the dev box (after that src was already checked out locally, so don't need to be on the corporate VPN)

variable "your_project_source" {
  default = "/Users/joeshmoe/projects/your_project/terraform"
}

(and overriding one or the other in terraform.tfvars) and then

module "your_project" {
  source = "${var.your_project_source}"
  ...
}

kokovoj commented Sep 1, 2015

👍

Use-case for this would be allowing for the flexibility to store module source in a variable for :

a. module source pointing at a corporate source control behind a corporate VPN

variable "your_project_source" {
  default = "https://your_src_system/your_project//terraform"
}

OR
b. use a local path on the dev box (after that src was already checked out locally, so don't need to be on the corporate VPN)

variable "your_project_source" {
  default = "/Users/joeshmoe/projects/your_project/terraform"
}

(and overriding one or the other in terraform.tfvars) and then

module "your_project" {
  source = "${var.your_project_source}"
  ...
}
@apparentlymart

This comment has been minimized.

Show comment
Hide comment
@apparentlymart

apparentlymart Sep 2, 2015

Contributor

One very specific complexity with this is that currently modules need to be pre-fetched using terraform get prior to terraform plan, and currently that command does not take any arguments that would allow you to set variables. By the time plan is running, Terraform is just thinking about the module name and paying no attention to the module source, since the module is assumed to already be retrieved into the .terraform subdirectory.

Perhaps in some cases this could be worked around by breaking a configuration into two separate runs, with an initial run creating a remote state that can be consumed by the second run. Since terraform_remote_state is just a regular resource its configuration arguments can be interpolated, even by things that aren't known until apply time, as long as a dependency cycle doesn't result.

This is of course not as convenient as creating everything in one step using directly-referenced modules, but maybe it's a reasonable workaround for some situations in the mean time.

Contributor

apparentlymart commented Sep 2, 2015

One very specific complexity with this is that currently modules need to be pre-fetched using terraform get prior to terraform plan, and currently that command does not take any arguments that would allow you to set variables. By the time plan is running, Terraform is just thinking about the module name and paying no attention to the module source, since the module is assumed to already be retrieved into the .terraform subdirectory.

Perhaps in some cases this could be worked around by breaking a configuration into two separate runs, with an initial run creating a remote state that can be consumed by the second run. Since terraform_remote_state is just a regular resource its configuration arguments can be interpolated, even by things that aren't known until apply time, as long as a dependency cycle doesn't result.

This is of course not as convenient as creating everything in one step using directly-referenced modules, but maybe it's a reasonable workaround for some situations in the mean time.

@apparentlymart

This comment has been minimized.

Show comment
Hide comment
@apparentlymart

apparentlymart Sep 2, 2015

Contributor

@kokovoj 's use-case, of switching to a different version in a development environment, got me thinking about how that gets solved in other languages.

When I have a problem like that in e.g. Go, NodeJS or Python I don't use any runtime features to solve it, but rather I just ignore the location/version of the module given in the dependency list and just install whatever one I want, exploiting the fact that (just like in Terraform) the "get" step is separated from the "compile" and "run" steps, and so we can do manual steps in between to arrange for the versions we want.

Terraform obscures this ability a little by storing the local modules in a directory named after the MD5 hash of the module name under the .terraform directory, so it's harder to recognize which one is which by eye... but you can, if you locate the right one, install it from a different source or modify it in-place. (I've done this several times while debugging, in fact.)

So with all of this said, perhaps Terraform could just be a little more transparent about where it looks for modules and embrace the idea that terraform get just installs the default module locations, but it's fine to manually install from other locations, or even to write your own separate tool to install from wherever you want. If we went this route, the only thing that would need to change in Terraform is to switch to a more user-friendly on-disk module representation and to commit not to change it in future versions of Terraform.

(It would also be nice to extend terraform get to be able to handle certain overrides itself, but that is made more complex by the fact that there can be nested modules that have their own dependencies, and so such syntax would probably end up quite complicated if it had to happen entirely on the command line.)

Contributor

apparentlymart commented Sep 2, 2015

@kokovoj 's use-case, of switching to a different version in a development environment, got me thinking about how that gets solved in other languages.

When I have a problem like that in e.g. Go, NodeJS or Python I don't use any runtime features to solve it, but rather I just ignore the location/version of the module given in the dependency list and just install whatever one I want, exploiting the fact that (just like in Terraform) the "get" step is separated from the "compile" and "run" steps, and so we can do manual steps in between to arrange for the versions we want.

Terraform obscures this ability a little by storing the local modules in a directory named after the MD5 hash of the module name under the .terraform directory, so it's harder to recognize which one is which by eye... but you can, if you locate the right one, install it from a different source or modify it in-place. (I've done this several times while debugging, in fact.)

So with all of this said, perhaps Terraform could just be a little more transparent about where it looks for modules and embrace the idea that terraform get just installs the default module locations, but it's fine to manually install from other locations, or even to write your own separate tool to install from wherever you want. If we went this route, the only thing that would need to change in Terraform is to switch to a more user-friendly on-disk module representation and to commit not to change it in future versions of Terraform.

(It would also be nice to extend terraform get to be able to handle certain overrides itself, but that is made more complex by the fact that there can be nested modules that have their own dependencies, and so such syntax would probably end up quite complicated if it had to happen entirely on the command line.)

@blalor

This comment has been minimized.

Show comment
Hide comment
@blalor

blalor Dec 24, 2015

Contributor

This is definitely something I'd like to see implemented. I'd be using it to solve the issue of where to pull a module's source when running in local dev or CI environments.

Contributor

blalor commented Dec 24, 2015

This is definitely something I'd like to see implemented. I'd be using it to solve the issue of where to pull a module's source when running in local dev or CI environments.

@orclev

This comment has been minimized.

Show comment
Hide comment
@orclev

orclev Feb 1, 2016

I'm also looking for a good solution to this. It seems like at the very least terraform get would need to be changed to support providing variables and to look in the conventional places for the variable settings.

Is the hash that modules get stored under a hash of the source attribute? If so it seems like so long as you hashed it after performing the interpolation everything would still be kosher. You would have had to run terraform get again if you modify any of the variables, so it isn't possible to self-modify included modules during a particular run of apply because the module won't exist yet. If you terraform apply with a different set of variables than you supplied for terraform get then it likewise would fail since the module wouldn't exist yet under the appropriate hash.

orclev commented Feb 1, 2016

I'm also looking for a good solution to this. It seems like at the very least terraform get would need to be changed to support providing variables and to look in the conventional places for the variable settings.

Is the hash that modules get stored under a hash of the source attribute? If so it seems like so long as you hashed it after performing the interpolation everything would still be kosher. You would have had to run terraform get again if you modify any of the variables, so it isn't possible to self-modify included modules during a particular run of apply because the module won't exist yet. If you terraform apply with a different set of variables than you supplied for terraform get then it likewise would fail since the module wouldn't exist yet under the appropriate hash.

@mmell

This comment has been minimized.

Show comment
Hide comment
@mmell

mmell Feb 2, 2016

Contributor

👍 for @clstokes use case: #1439 (comment)

Contributor

mmell commented Feb 2, 2016

👍 for @clstokes use case: #1439 (comment)

@YokiWorks

This comment has been minimized.

Show comment
Hide comment

👍 for @kokovoj

@jonapich

This comment has been minimized.

Show comment
Hide comment
@jonapich

jonapich Feb 18, 2016

What threw me off here is the ${path.module} interpolation. I have a hierarchy like this:

|- ./dev  <--- this is my cwd
|- ./modules
|- ./modules/specific_module
|- ./modules/generic_module

I am using nested modules, so I thought I would use source = "${path.module}/../another_module" when loading the nested one and ran into the "source cannot use interpolation" error.

Turns out the interpolation is not necessary in this case! (wat?)

  • From ./dev I can module "mod" { source = "../modules/specific_module" } as expected.
  • From ./modules/specific_module/mod.tf, I can create a module "mod" { source = "../generic_module" } and it works.

It is very surprising given the following documentation in the "How to create a module" "Paths and Embedded Files" section:

(...) since paths in Terraform are generally relative to the working directory that Terraform was executed from

In the above, we use ${path.module} to get a module-relative path. This is usually what you'll want in any case.

The section that follows, "Nested Modules", doesn't mention that loading a module from a module uses the module's path as the cwd as an exception.

What threw me off here is the ${path.module} interpolation. I have a hierarchy like this:

|- ./dev  <--- this is my cwd
|- ./modules
|- ./modules/specific_module
|- ./modules/generic_module

I am using nested modules, so I thought I would use source = "${path.module}/../another_module" when loading the nested one and ran into the "source cannot use interpolation" error.

Turns out the interpolation is not necessary in this case! (wat?)

  • From ./dev I can module "mod" { source = "../modules/specific_module" } as expected.
  • From ./modules/specific_module/mod.tf, I can create a module "mod" { source = "../generic_module" } and it works.

It is very surprising given the following documentation in the "How to create a module" "Paths and Embedded Files" section:

(...) since paths in Terraform are generally relative to the working directory that Terraform was executed from

In the above, we use ${path.module} to get a module-relative path. This is usually what you'll want in any case.

The section that follows, "Nested Modules", doesn't mention that loading a module from a module uses the module's path as the cwd as an exception.

@franklinwise

This comment has been minimized.

Show comment
Hide comment
@franklinwise

franklinwise Feb 29, 2016

+1 - It would be very helpful to be able to use a variable for the source, particularly targeting different branches.

+1 - It would be very helpful to be able to use a variable for the source, particularly targeting different branches.

@johnrengelman

This comment has been minimized.

Show comment
Hide comment
@johnrengelman

johnrengelman Mar 1, 2016

Contributor

Another user case is providing credentials for accessing a private repository over HTTPS. Each user could set their own username & password then.

Contributor

johnrengelman commented Mar 1, 2016

Another user case is providing credentials for accessing a private repository over HTTPS. Each user could set their own username & password then.

@robcoward

This comment has been minimized.

Show comment
Hide comment
@robcoward

robcoward Mar 16, 2016

Contributor

As @johnrengelman mentioned, the use of private sources requiring user credentials is just not realistic if we are forced to hard-code the credentials. We have to be allowed to pass in credentials to private github repos etc via some mechanism, variable interpolation or whatever, especially if you are trying to use Atlas with Terraform, pulling config from a GitHub repo in the first place.

Contributor

robcoward commented Mar 16, 2016

As @johnrengelman mentioned, the use of private sources requiring user credentials is just not realistic if we are forced to hard-code the credentials. We have to be allowed to pass in credentials to private github repos etc via some mechanism, variable interpolation or whatever, especially if you are trying to use Atlas with Terraform, pulling config from a GitHub repo in the first place.

@ketzacoatl

This comment has been minimized.

Show comment
Hide comment
@ketzacoatl

ketzacoatl Mar 16, 2016

Contributor

Standard/best-practices for accessing private repos on git is (in my experience) done with SSH and pub/private keys, not HTTPS with username/passwords.

Contributor

ketzacoatl commented Mar 16, 2016

Standard/best-practices for accessing private repos on git is (in my experience) done with SSH and pub/private keys, not HTTPS with username/passwords.

@suneeta-mall

This comment has been minimized.

Show comment
Hide comment
@suneeta-mall

suneeta-mall Jul 7, 2017

+1. This is currently blocking us too and using sed to workaround this! Would be nice to have this feature.

+1. This is currently blocking us too and using sed to workaround this! Would be nice to have this feature.

@RobertFr0st

This comment has been minimized.

Show comment
Hide comment
@RobertFr0st

RobertFr0st Jul 17, 2017

  • 1, would like to see!
  • 1, would like to see!
@mukund1989

This comment has been minimized.

Show comment
Hide comment
@mukund1989

mukund1989 Jul 20, 2017

pretty please!

pretty please!

@MCoulombe727

This comment has been minimized.

Show comment
Hide comment
@MCoulombe727

MCoulombe727 Jul 21, 2017

Not being able to interpolate module's source location is a huge deal breaker for me. Any decent sized project that wants reusable parts would need something more flexible than hardcoding the files' locations.

Not being able to interpolate module's source location is a huge deal breaker for me. Any decent sized project that wants reusable parts would need something more flexible than hardcoding the files' locations.

@apparentlymart

This comment has been minimized.

Show comment
Hide comment
@apparentlymart

apparentlymart Jul 21, 2017

Contributor

Hi everyone! Thanks again for the great discussion here, and thanks to everyone for confirming and clarifying their different use-cases.

I've captured the two big ones we discussed before in their own issues, so we can have a more focused discussion about each of them separately:

  • #15613: Changing module version numbers without editing config
  • #15614: Passing credentials for module sources without putting them in config

The first of these has some notes about a possible design building on some internal discussions we'd had on this subject. The second doesn't currently have a design proposal attached, but we can discuss some more the requirements and see if we can get to one.


Regarding @MichaelDeCorte's use-case of module sources that aren't relative to the source module, it is intentional that modules are specified relative to one another when they appear in the local filesystem, since we believe that reduces coupling by avoiding the modules needing to "know" where their root is. At this time I don't expect we will make changes to support absolute paths here, since that would complicate the model and add more different situations to document and test.

However, if you believe you have a compelling use-case that would warrant the additional complexity here, please feel free to open a fresh issue for this with some more details on the use-case. It's easier to have discussions about these things with concrete examples to work from, since that way we can potentially find alternative implementations that solve some or all of the same problems with fewer drawbacks.


Given that there are now some more specific issues covering the main use-cases here, and this is progressing from general "thinking" to more specific design work, I'm going to close this issue and encourage further discussion in those issues. If anyone has an additional use-case not covered already in this comment then please do open a new issue referencing this one -- ideally following a similar format to the ones I opened above, focusing on a problem statement rather than a candidate solution -- and we'll then be able to have a more detailed discussion about each separate use-case and find the best approach for each.

Thanks again for all the discussion here! People sharing their use-cases is always very helpful to feel out the shape of a problem and explore solutions for it.

Contributor

apparentlymart commented Jul 21, 2017

Hi everyone! Thanks again for the great discussion here, and thanks to everyone for confirming and clarifying their different use-cases.

I've captured the two big ones we discussed before in their own issues, so we can have a more focused discussion about each of them separately:

  • #15613: Changing module version numbers without editing config
  • #15614: Passing credentials for module sources without putting them in config

The first of these has some notes about a possible design building on some internal discussions we'd had on this subject. The second doesn't currently have a design proposal attached, but we can discuss some more the requirements and see if we can get to one.


Regarding @MichaelDeCorte's use-case of module sources that aren't relative to the source module, it is intentional that modules are specified relative to one another when they appear in the local filesystem, since we believe that reduces coupling by avoiding the modules needing to "know" where their root is. At this time I don't expect we will make changes to support absolute paths here, since that would complicate the model and add more different situations to document and test.

However, if you believe you have a compelling use-case that would warrant the additional complexity here, please feel free to open a fresh issue for this with some more details on the use-case. It's easier to have discussions about these things with concrete examples to work from, since that way we can potentially find alternative implementations that solve some or all of the same problems with fewer drawbacks.


Given that there are now some more specific issues covering the main use-cases here, and this is progressing from general "thinking" to more specific design work, I'm going to close this issue and encourage further discussion in those issues. If anyone has an additional use-case not covered already in this comment then please do open a new issue referencing this one -- ideally following a similar format to the ones I opened above, focusing on a problem statement rather than a candidate solution -- and we'll then be able to have a more detailed discussion about each separate use-case and find the best approach for each.

Thanks again for all the discussion here! People sharing their use-cases is always very helpful to feel out the shape of a problem and explore solutions for it.

@jjshoe

This comment has been minimized.

Show comment
Hide comment
@jjshoe

jjshoe Aug 3, 2017

@apparentlymart how about when I want to use one method to find a module, and if not found, fall back to another?

Think local development, before I push my changes up to git.

edit: #1439 (comment)

jjshoe commented Aug 3, 2017

@apparentlymart how about when I want to use one method to find a module, and if not found, fall back to another?

Think local development, before I push my changes up to git.

edit: #1439 (comment)

@boostrack

This comment has been minimized.

Show comment
Hide comment
@boostrack

boostrack Aug 6, 2017

If you need Terraform to be able to fetch modules from private GitHub repos on a remote machine (like Terraform Enterprise or a CI server), you'll need to provide Terraform with credentials that can be used to authenticate as a user with read access to the private repo.
First, create a machine user on GitHub with read access to the private repo in question, then embed this user's credentials into the source parameter:
module "private-infra" {
  source = "git::https://MACHINE-USER:MACHINE-PASS@github.com/org/privatemodules//modules/foo"
}

Note: Terraform does not yet support interpolations in the source field, so the machine username and password will have to be embedded directly into the source string. You can track GH-1439 to learn when this limitation is addressed.

SOURCE GH-1439

https://www.terraform.io/docs/modules/sources.html

@apparentlymart maybe the docs should not reference this "closed" issue

boostrack commented Aug 6, 2017

If you need Terraform to be able to fetch modules from private GitHub repos on a remote machine (like Terraform Enterprise or a CI server), you'll need to provide Terraform with credentials that can be used to authenticate as a user with read access to the private repo.
First, create a machine user on GitHub with read access to the private repo in question, then embed this user's credentials into the source parameter:
module "private-infra" {
  source = "git::https://MACHINE-USER:MACHINE-PASS@github.com/org/privatemodules//modules/foo"
}

Note: Terraform does not yet support interpolations in the source field, so the machine username and password will have to be embedded directly into the source string. You can track GH-1439 to learn when this limitation is addressed.

SOURCE GH-1439

https://www.terraform.io/docs/modules/sources.html

@apparentlymart maybe the docs should not reference this "closed" issue

@simplytunde

This comment has been minimized.

Show comment
Hide comment
@simplytunde

simplytunde Aug 24, 2017

Yes, the issue is already closed but the Doc says to keep tracking the same issue. Any suggestion on what to put instead?

Yes, the issue is already closed but the Doc says to keep tracking the same issue. Any suggestion on what to put instead?

@boostrack

This comment has been minimized.

Show comment
Hide comment
@boostrack

boostrack Aug 24, 2017

maybe just remove ?

Note: Terraform does not yet support interpolations in the source field, so the machine username and password will have to be embedded directly into the source string. You can track GH-1439 to learn when this limitation is addressed.

SOURCE GH-1439

The worst than not docs is no accurate docs. maybe just that interpolations in the source field are not supported

boostrack commented Aug 24, 2017

maybe just remove ?

Note: Terraform does not yet support interpolations in the source field, so the machine username and password will have to be embedded directly into the source string. You can track GH-1439 to learn when this limitation is addressed.

SOURCE GH-1439

The worst than not docs is no accurate docs. maybe just that interpolations in the source field are not supported

@apparentlymart

This comment has been minimized.

Show comment
Hide comment
@apparentlymart

apparentlymart Aug 24, 2017

Contributor

Hmm thanks for pointing that out, @boostrack. We don't normally make direct references to github from the website so I didn't even think to check for that. For the moment I'm just going to remove that link altogether since we don't have any good means to ensure that this sort of problem won't happen again, and github issues are too transient to be referenced in the long-lived documentation.

Contributor

apparentlymart commented Aug 24, 2017

Hmm thanks for pointing that out, @boostrack. We don't normally make direct references to github from the website so I didn't even think to check for that. For the moment I'm just going to remove that link altogether since we don't have any good means to ensure that this sort of problem won't happen again, and github issues are too transient to be referenced in the long-lived documentation.

@alevinetx

This comment has been minimized.

Show comment
Hide comment
@alevinetx

alevinetx Oct 4, 2017

The note for lack of source interpolation is buried under the Private GitHub Repos section. Because this section doesn't apply to me, I missed this note the many times I was looking at the page.

May I suggest you call this restriction out at the top of the page in the generic summary?

The note for lack of source interpolation is buried under the Private GitHub Repos section. Because this section doesn't apply to me, I missed this note the many times I was looking at the page.

May I suggest you call this restriction out at the top of the page in the generic summary?

@TinajaLabs

This comment has been minimized.

Show comment
Hide comment
@TinajaLabs

TinajaLabs Oct 18, 2017

Perhaps this is the wrong place to bring this up, but how is one to build a system where one can create abstract modules that call to either an AWS deployment or an Openstack deployment.

I need to be able to deploy the same infrastructure into either environment and I'm not sure how to structure it. This is the beauty of "providers", right?

I should be able to build custom modules that have references to instance_image (ami for aws, image_id for openstack) and instance_type (instance_type for aws, flavor_id for openstack).

It seems one should be able to build a set of custom modules where the source is either a set of aws modules, or openstack modules. Hence the ability to turn this on a root level variable for source definition.

Is everyone else using a completely separate code base for each type of infrastructure deployment? How ultimately tedious and error prone that would be.

Perhaps this is the wrong place to bring this up, but how is one to build a system where one can create abstract modules that call to either an AWS deployment or an Openstack deployment.

I need to be able to deploy the same infrastructure into either environment and I'm not sure how to structure it. This is the beauty of "providers", right?

I should be able to build custom modules that have references to instance_image (ami for aws, image_id for openstack) and instance_type (instance_type for aws, flavor_id for openstack).

It seems one should be able to build a set of custom modules where the source is either a set of aws modules, or openstack modules. Hence the ability to turn this on a root level variable for source definition.

Is everyone else using a completely separate code base for each type of infrastructure deployment? How ultimately tedious and error prone that would be.

@ju2wheels

This comment has been minimized.

Show comment
Hide comment
@ju2wheels

ju2wheels Oct 18, 2017

@TinajaLabs Thats a discussion for the Gitter channel: gitter.im/hashicorp-terraform/Lobby

@TinajaLabs Thats a discussion for the Gitter channel: gitter.im/hashicorp-terraform/Lobby

@dmikalova

This comment has been minimized.

Show comment
Hide comment
@dmikalova

dmikalova Nov 14, 2017

Can a warning be added that interpolations aren't allowed since there should be a validation?
Error loading modules: module name: invalid source string: ${path.root}/modules/name isn't very obvious at first.

Can a warning be added that interpolations aren't allowed since there should be a validation?
Error loading modules: module name: invalid source string: ${path.root}/modules/name isn't very obvious at first.

@luisdavim

This comment has been minimized.

Show comment
Hide comment
@luisdavim

luisdavim Jan 17, 2018

I would like to at least be able to use variables to specify module versions

I would like to at least be able to use variables to specify module versions

@nik-shornikov

This comment has been minimized.

Show comment
Hide comment
@nik-shornikov

nik-shornikov Jan 17, 2018

lack of interpolation is pretty stealthed-out in the doc and the error you get for trying is still opaque

nik-shornikov commented Jan 17, 2018

lack of interpolation is pretty stealthed-out in the doc and the error you get for trying is still opaque

@pixelicous

This comment has been minimized.

Show comment
Hide comment
@pixelicous

pixelicous Feb 1, 2018

Please implement this!! no more running away to use cases and finding workarounds, interpolation should have much greater support >.<

Please implement this!! no more running away to use cases and finding workarounds, interpolation should have much greater support >.<

@ketzacoatl

This comment has been minimized.

Show comment
Hide comment
@ketzacoatl

ketzacoatl Feb 1, 2018

Contributor

There are new issues to capture further discussion on more specific nuances of this topic.

Contributor

ketzacoatl commented Feb 1, 2018

There are new issues to capture further discussion on more specific nuances of this topic.

@pll

This comment has been minimized.

Show comment
Hide comment
@pll

pll Feb 1, 2018

@ketzacoatl Can you please link to those new issues so we can follow those discussions?

Thanks.

pll commented Feb 1, 2018

@ketzacoatl Can you please link to those new issues so we can follow those discussions?

Thanks.

@pixelicous

This comment has been minimized.

Show comment
Hide comment
@pixelicous

pixelicous Feb 1, 2018

@ketzacoatl i would love a reference to where this specific issue is solved, i need to interpolate the source of a module, is it possible to reference a different directory name based on a passed variable.. The variables should be graphed in a way that this would be attainable

pixelicous commented Feb 1, 2018

@ketzacoatl i would love a reference to where this specific issue is solved, i need to interpolate the source of a module, is it possible to reference a different directory name based on a passed variable.. The variables should be graphed in a way that this would be attainable

@luisdavim

This comment has been minimized.

Show comment
Hide comment
@luisdavim

luisdavim Feb 1, 2018

We're maintaining a terraform wrapper to overcome some of these issues. We implement the Terrafile pattern. It would be a great improvement if terraform would support the Terrafile approach natively.

We're maintaining a terraform wrapper to overcome some of these issues. We implement the Terrafile pattern. It would be a great improvement if terraform would support the Terrafile approach natively.

@coryjamesfisher

This comment has been minimized.

Show comment
Hide comment
@coryjamesfisher

coryjamesfisher Feb 10, 2018

I come from the software development side of the house and abstraction to an interface seems like a good move.

My ideal is to create an interface for a module representing one logical component of our infrastructure. Then each module worries about the provider specific details.

So for example I would create 2 modules:
module/aws/webserver
module/openstack/webserver
Finally a main.tf file with:
variable "cloud_provider" { default = "aws", description = "Pass aws or openstack" }
...
source = module/${vars.cloud_provider}/web

I'm looking for interpolation to work in the module source path.
If there is another way to do this I'd be happy to hear it. I have one idea of wrapping the Terraform script with a templating/replacement engine of my own driven by a shell script. But it'd be nice to have a solution in Terraform itself.

I come from the software development side of the house and abstraction to an interface seems like a good move.

My ideal is to create an interface for a module representing one logical component of our infrastructure. Then each module worries about the provider specific details.

So for example I would create 2 modules:
module/aws/webserver
module/openstack/webserver
Finally a main.tf file with:
variable "cloud_provider" { default = "aws", description = "Pass aws or openstack" }
...
source = module/${vars.cloud_provider}/web

I'm looking for interpolation to work in the module source path.
If there is another way to do this I'd be happy to hear it. I have one idea of wrapping the Terraform script with a templating/replacement engine of my own driven by a shell script. But it'd be nice to have a solution in Terraform itself.

@pixelicous

This comment has been minimized.

Show comment
Hide comment
@pixelicous

pixelicous Feb 11, 2018

@coryjamesfisher Same issue as you here.. couldn't find a workaround besides using third party tools to handle this problem, which is really ugly.

I seriously do not understand the problem, there should be an option to init with the variable or based upon folder structure.. most of the workarounds and issues are caused due to this.. funny thing now when i started using workspaces, you cant even use the workspace name as interpolation for a default of a variable.. very ugly.. need to go through a local..

pixelicous commented Feb 11, 2018

@coryjamesfisher Same issue as you here.. couldn't find a workaround besides using third party tools to handle this problem, which is really ugly.

I seriously do not understand the problem, there should be an option to init with the variable or based upon folder structure.. most of the workarounds and issues are caused due to this.. funny thing now when i started using workspaces, you cant even use the workspace name as interpolation for a default of a variable.. very ugly.. need to go through a local..

@eliblight

This comment has been minimized.

Show comment
Hide comment
@eliblight

eliblight Mar 6, 2018

+1 Here is my use case (would love to hear about a work around)

I have multiple projects/deployments, each needs to be deployed on multiple regions/zones.

following github.com/kung-foo/multiregion-terraform/blob/master/main.tf, I adopted the module per region approch.
For a single project this looks like this:

  • main.tf
module "us-east4" {
  source = "per-region"
  region = "us-east4"
  zones = ["us-east4-a", "us-east4-b"]
}

module "us-west1" {
  source = "per-region"
  region = "us-west1"
  zones = ["us-west1-a", "us-west1-b", "us-west1-c"]
}
...
  • per-region/main.tf
provider "google" {
  region      = "${var.region}"
  project     = "a-single-project"
}

resource "google_compute_instance" "gci" {
  count = "${length(var.zones)}"
  zone = "${var.zones[count.index]}"
  ...
  • per-region/variables.tf
variable "region" {}

variable "zones" {
  type = "list"
}

BUT if I have multiple project, each and eveyone of them will need the main regions and zones file.

So I wanted to have the following workaround

  • project-a/main.tf
module "deployment" {
  source = "../topology"
  dir = "../project-a/per-region"
}
  • project-a/per-region/main.tf (same as before)
  • project-a/per-region/variables.tf (same as before)
  • topology/main.tf
module "us-east4" {
  source = "${var.dir}"
  region = "us-east4"
  zones = ["us-east4-a", "us-east4-b"]
}

module "us-west1" {
  source = "${var.dir}"
  region = "us-west1"
  zones = ["us-west1-a", "us-west1-b", "us-west1-c"]
}
  • topology/variables.tf
variable "dir" {}

each project follows the same pattern

  • project-X/main.tf - side stepping to topology and back
  • project-X/per-region - specific deployment of each project per region

Alas this does not work.

Any workarounds / suggestions?

+1 Here is my use case (would love to hear about a work around)

I have multiple projects/deployments, each needs to be deployed on multiple regions/zones.

following github.com/kung-foo/multiregion-terraform/blob/master/main.tf, I adopted the module per region approch.
For a single project this looks like this:

  • main.tf
module "us-east4" {
  source = "per-region"
  region = "us-east4"
  zones = ["us-east4-a", "us-east4-b"]
}

module "us-west1" {
  source = "per-region"
  region = "us-west1"
  zones = ["us-west1-a", "us-west1-b", "us-west1-c"]
}
...
  • per-region/main.tf
provider "google" {
  region      = "${var.region}"
  project     = "a-single-project"
}

resource "google_compute_instance" "gci" {
  count = "${length(var.zones)}"
  zone = "${var.zones[count.index]}"
  ...
  • per-region/variables.tf
variable "region" {}

variable "zones" {
  type = "list"
}

BUT if I have multiple project, each and eveyone of them will need the main regions and zones file.

So I wanted to have the following workaround

  • project-a/main.tf
module "deployment" {
  source = "../topology"
  dir = "../project-a/per-region"
}
  • project-a/per-region/main.tf (same as before)
  • project-a/per-region/variables.tf (same as before)
  • topology/main.tf
module "us-east4" {
  source = "${var.dir}"
  region = "us-east4"
  zones = ["us-east4-a", "us-east4-b"]
}

module "us-west1" {
  source = "${var.dir}"
  region = "us-west1"
  zones = ["us-west1-a", "us-west1-b", "us-west1-c"]
}
  • topology/variables.tf
variable "dir" {}

each project follows the same pattern

  • project-X/main.tf - side stepping to topology and back
  • project-X/per-region - specific deployment of each project per region

Alas this does not work.

Any workarounds / suggestions?

@Binternet

This comment has been minimized.

Show comment
Hide comment
@Binternet

Binternet Mar 11, 2018

+1 for that, I am organizing my AWS infrastructure to modules and trying to define each module path under /services folder. instead of hard-coding /services path in each module source parameter, it should be set as a variable.

+1 for that, I am organizing my AWS infrastructure to modules and trying to define each module path under /services folder. instead of hard-coding /services path in each module source parameter, it should be set as a variable.

@Midacts

This comment has been minimized.

Show comment
Hide comment
@Midacts

Midacts Apr 24, 2018

Has this been implemented yet or on the books to be implemented?

Midacts commented Apr 24, 2018

Has this been implemented yet or on the books to be implemented?

@nbering

This comment has been minimized.

Show comment
Hide comment
@nbering

nbering Apr 24, 2018

To recap, here's the closing comment from @apparentlymart.

#1439 (comment)

When closed, he split this issue into #15613 and #15614.

Also in #1439 (comment) @mitchellh, indicated that this is one of the places where variables cannot be used (by design) because of the execution order of various steps. I can't find anything to indicate that this was considered for change.

nbering commented Apr 24, 2018

To recap, here's the closing comment from @apparentlymart.

#1439 (comment)

When closed, he split this issue into #15613 and #15614.

Also in #1439 (comment) @mitchellh, indicated that this is one of the places where variables cannot be used (by design) because of the execution order of various steps. I can't find anything to indicate that this was considered for change.

@lorengordon

This comment has been minimized.

Show comment
Hide comment
@lorengordon

lorengordon Jun 19, 2018

Contributor

Personally I'd love to see interpolation for the entire source parameter. The chosen direction to implement support for just the version is very limiting. I'd rather like to pull all my source definitions to the top of a configuration, in a locals definition, so I don't have to go hunting through every file to find/update the string. I expect it would make modules much more maintainable overall.

Contributor

lorengordon commented Jun 19, 2018

Personally I'd love to see interpolation for the entire source parameter. The chosen direction to implement support for just the version is very limiting. I'd rather like to pull all my source definitions to the top of a configuration, in a locals definition, so I don't have to go hunting through every file to find/update the string. I expect it would make modules much more maintainable overall.

@pixelicous

This comment has been minimized.

Show comment
Hide comment
@pixelicous

pixelicous Jun 28, 2018

@lorengordon I agree.. this is nonsense.. that and the fact that everytime you pull a whole repository instead of a leaf

@lorengordon I agree.. this is nonsense.. that and the fact that everytime you pull a whole repository instead of a leaf

@luisdavim

This comment has been minimized.

Show comment
Hide comment
@luisdavim

luisdavim Jun 28, 2018

We use this http://bensnape.com/2016/01/14/terraform-design-patterns-the-terrafile/ I think it would be reasonable to have something like that natively.

We use this http://bensnape.com/2016/01/14/terraform-design-patterns-the-terrafile/ I think it would be reasonable to have something like that natively.

@jjshoe

This comment has been minimized.

Show comment
Hide comment
@jjshoe

jjshoe Jul 2, 2018

@mitchellh - It would be great if hashicorp could re-look at this. Though it's been closed, and split into two cases, which don't address all the reasons for this, it's more commented then any current open issue.

jjshoe commented Jul 2, 2018

@mitchellh - It would be great if hashicorp could re-look at this. Though it's been closed, and split into two cases, which don't address all the reasons for this, it's more commented then any current open issue.

@akvadrako

This comment has been minimized.

Show comment
Hide comment
@akvadrako

akvadrako Jul 9, 2018

The best workaround I have found is by using putting something like this in override.tf

module "core" {
    source = "/Users/dev/terraform-modules/core"
}
...

Not ideal, but seems to work.

The best workaround I have found is by using putting something like this in override.tf

module "core" {
    source = "/Users/dev/terraform-modules/core"
}
...

Not ideal, but seems to work.

@MichaelDeCorte

This comment has been minimized.

Show comment
Hide comment
@MichaelDeCorte

MichaelDeCorte Jul 10, 2018

@mitchellh agreement with @jjshoe the original issue of allowing interpolation for the source parameter has not been addressed. This issue should be opened, or a new one forked off. But it should not be closed.

@mitchellh agreement with @jjshoe the original issue of allowing interpolation for the source parameter has not been addressed. This issue should be opened, or a new one forked off. But it should not be closed.

@MichaelDeCorte

This comment has been minimized.

Show comment
Hide comment
@MichaelDeCorte

MichaelDeCorte Jul 10, 2018

@akvadrako I'm not following your workaround. Can you elaborate?

@akvadrako I'm not following your workaround. Can you elaborate?

@akvadrako

This comment has been minimized.

Show comment
Hide comment
@akvadrako

akvadrako Jul 10, 2018

@MichaelDeCorte It's just that it's possible to override the module source parameters with an external file. My use case is module development, where I want to replace several references to git repos with local checkouts.

akvadrako commented Jul 10, 2018

@MichaelDeCorte It's just that it's possible to override the module source parameters with an external file. My use case is module development, where I want to replace several references to git repos with local checkouts.

@MichaelDeCorte

This comment has been minimized.

Show comment
Hide comment
@MichaelDeCorte

MichaelDeCorte Jul 10, 2018

@mitchellh elaborating an example to allow the for absolute paths relative to TF-Home. Assume the below directory / file structure. This is a common pattern where repo1 is a shared repository that is downloaded locally via a script as a workaround for the source interpolation issue.

TF-Home /

  • app1.tf
  • appSubDir1
    • app2.tf
  • repo1 /
    • foo1.tf
    • foo2.tf

Assume that app1, app2 and foo1.tf all depend on foo2. The source parameter would be:
app1: repo1/foo2.tf
app2: ../repo1/foo2.tf
foo1: foo2.tf

I hope it's clear that its not great.

MichaelDeCorte commented Jul 10, 2018

@mitchellh elaborating an example to allow the for absolute paths relative to TF-Home. Assume the below directory / file structure. This is a common pattern where repo1 is a shared repository that is downloaded locally via a script as a workaround for the source interpolation issue.

TF-Home /

  • app1.tf
  • appSubDir1
    • app2.tf
  • repo1 /
    • foo1.tf
    • foo2.tf

Assume that app1, app2 and foo1.tf all depend on foo2. The source parameter would be:
app1: repo1/foo2.tf
app2: ../repo1/foo2.tf
foo1: foo2.tf

I hope it's clear that its not great.

@Yanson Yanson referenced this issue in GoogleCloudPlatform/terraform-google-nat-gateway Jul 18, 2018

Closed

Use non-broken version of managed-instance-group and allow override #67

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment