New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow modules to access global variables #5480

Open
joshuaspence opened this Issue Mar 6, 2016 · 23 comments

Comments

Projects
None yet
@joshuaspence
Contributor

joshuaspence commented Mar 6, 2016

I have a module that creates an EC2 instance with a bunch of common properties. Whenever I use this function, I end up passing a bunch of common variables such as ${var.region} (which is used by our bootstrap script injected into user data).

module "instance" {
  source = "..."
  region = "${var.region}"
}

It would be useful if this variable could be set once rather than every time that the module is used... I think that this could be implemented in one of two ways:

  1. Allow default values to be set. In puppet, default values for a file resource can be set with a File meta-resource. Perhaps a similar thing could happen in Terraform.
Module {
  source = "..."
  region = "${var.region}"
}
  1. Allow the module to access variables globally with an alternative syntax. Perhaps ${global.region} instead of ${var.region}.
@phinze

This comment has been minimized.

Member

phinze commented Mar 7, 2016

Hi @joshuaspence - thanks for the feedback!

We've discussed global variables in the past:

In general we're against the particular solution of Global Variables, since it makes the input -> resources -> output flow of Modules less explicit, and explicitness is a core design goal.

That being said, we're definitely aware of the heavy repetition required in Terraform configs today, and it's something we'd like to address without sacrificing clarity (which is not a simple combination to achieve!).

The puppet-style default attribute syntax is one interesting idea. A related discussion can be found here: #1478

It'd be great to get these efforts consolidated in a canonical issue - if you feel like #1478 fits the bill, let's merge this down with that conversation. If not, let's figure out how to get this issue expressing the need clearly. 👍

@cahlbin

This comment has been minimized.

cahlbin commented Apr 13, 2016

I skimmed through the related issues and found most of them suggest, well truly global access which I agree should not be allowed. I would like to suggest a terraform function that would be available in the module scope, that would allow to propagate a specific set of variables in a more consise syntax.

Example:

# Currently one have to propagate each variable:
module "instance" {
  source = "..."
  var1 = "${var.var1}"
  var2 = "${var.var2}"
}

, could instead with propagate_vars look something like:

module "instance" {
  source = "..."
  propagate_vars("var1", "var2", ...)
}

That could possibly also be implemented so that one can easily propagate map-typed variables, as they are only references by var name so terraform.

@zanona

This comment has been minimized.

zanona commented Aug 25, 2016

I believe we could have something like providers which would be really helpful to be applied to all instances of a module, for example:

provider "cool_module" {
  global_module_var = "foo"
  reference_module_var = "${aws_instance.web.arn}"
}

module "instance_1" {
  source = "./cool_module"
}
module "instance_2" {
  source = "./cool_module"
}

Perhaps there's already a way to achieve that currently?

@garo

This comment has been minimized.

garo commented Sep 7, 2016

#1478 would solve this kind of problems partially, but I'm not sure it would be useful in a nested module dependency.

Consider an example:

  • A module which defines a single instance. Uses resource "aws_instance" along with a data "template_file" for the instance user_data.
  • A module which defines a single VPC. Includes the single instance module several times. For example variables like domain and key_name need to be passed to every module call.
  • A top level which calls the VPC module a few times. Uses a few variables to define high level data like domain and key name for the entire VPC.

In altogether I find that the HCL restricts the expressibility a bit too much as it's just a one-to-one conversion from HCL to json (and back if required). Consider adding preparser/template/iteration/control/loop features so that a simple HCL script could generate complex json , which would then be consumed with Terraform.

@reubenavery

This comment has been minimized.

reubenavery commented Nov 25, 2016

+1, and I would strongly argue for following the patterns from Puppet in variable scoping. The spaghetti code resulting from a highly modularized terraform project quickly becomes a nightmare.

@sheerun

This comment has been minimized.

Contributor

sheerun commented Mar 7, 2017

Global variables is something kubernetes introduced for charts to simplify configuration:
https://github.com/kubernetes/helm/blob/master/docs/charts.md#global-values

@keirhbadger

This comment has been minimized.

keirhbadger commented Apr 7, 2017

Just figured out a way of doing this

If you create just an output.tf file in a directory and have value="global string" rather than value="${variable name}" you can run terraform apply and it will populate the terraform.tfstate file with your global string

Now you can access this in your std terraform tf files using remote states

@rolandkool

This comment has been minimized.

rolandkool commented Jul 25, 2017

@keirhbadger can you share some code example how you did this? I've tried to accomplish the same using remote state in consul, but for some reason having a terraform module that just outputs some variables doesn't result in a remote state containing those variables. So when I want to read them in a module, they are empty.
Thanks!

@apparentlymart apparentlymart added config and removed core labels Jul 25, 2017

@keirbadger

This comment has been minimized.

keirbadger commented Jul 25, 2017

So this is an output.tf file in a global_vars directory

output "environment"        { value = "prod" }										
output "region"             { value = "us-east-2" }

Run terraform apply in that dir and the terraform.tfstate file will contain these lines

            "outputs": {
                "environment": {
                    "sensitive": false,
                    "type": "string",
                    "value": "prod"
                },
                "region": {
                    "sensitive": false,
                    "type": "string",
                    "value": "us-east-2"
                }
            },

You can now access this is a diff Terraform playbook with something like

data "terraform_remote_state" "global" {
  backend = "local"
  config {
      path = "../global_vars/terraform.tfstate"
  }
}

And
Name = "priv-b-${data.terraform_remote_state.global.environment}"

@apparentlymart

This comment has been minimized.

Contributor

apparentlymart commented Jul 25, 2017

Hi all,

Thanks for the great discussion here and sorry we've been quiet for a while.

Recently we've started some early planning for a set of improvements to the configuration language for an upcoming Terraform release. One of the features we're looking at is the ability to pass complex objects into modules, which (amongst other uses) can be a compromise to reduce the boilerplate of "global settings" without hurting the explicitness goal discussed earlier.

In combination with the feature added in #15449 (which is paused for the moment while we get 0.10.0 released) this may allow configuration like the following:

# This is a DRAFT and not yet implemented; final syntax/behavior may differ

locals {
  config = {
    region = "${var.region}"
    # ...and any other settings you want to make available to modules...
  }
}

module "example" {
  source = "./example"

  # pass the whole config object in a single variable
  config = "${local.config}"
}

Passing around this sort of composite structure will inevitably increase coupling between caller and callee, so we'd still recommend that users do this sort of thing with care, but this at least opens up more options for concisely passing data between modules.

The other pattern we hope to allow with this feature is to invert dependencies by making it easier to pass around resources created elsewhere. Although this doesn't apply to the region use-case that motivated this issue, it could make it more convenient to write small, self-contained modules that compose together, which is tricky today:

# This is a DRAFT and not yet implemented; final syntax/behavior may differ

resource "aws_vpc" "example" {
  # ... vpc settings ...
}

module "security_groups" {
  source = "./security_groups"

  # pass the entire VPC object to the module, giving it access to the id, CIDR block, etc
  vpc = "${aws_vpc.example}"
}

As noted above, this is all just early design sketch right now and the details are likely to shift as we prototype and learn more, but the general hope is to foster new patterns that can reduce the amount of boilerplate while remaining consistent with the goals of keeping modules loosely coupled with each other and keeping module interactions explicit.

@rolandkool

This comment has been minimized.

rolandkool commented Jul 26, 2017

@keirbadger thanks for your example. I was doing the same and I now think I'm hitting some kind of bug with the kubernetes namespace resource. The labels don't seem to like the remote_state variables I pull in. Will raise a separate issue for that.

@braderhart

This comment has been minimized.

braderhart commented Oct 3, 2017

Really looking forward to this feature and sad to see it still here. Can you kindly point me in the direction of what needs to be done to define some sort of include or module functionality like this? I thought of using templates but realize that they can only be assigned per parameter values, and not for the entire block. An include functionality would be nice for situations like these as well.

@petewilcock

This comment has been minimized.

petewilcock commented Apr 5, 2018

I have a somewhat hacky workaround for this, but it's the best I've got for now:

I specify the variable I want to be global in my root module. In this example, I'm specifying the security group for an aws instance. I then set that variable to be an output of the root module:

output "aws_security_group" {
  value = "${var.aws_security_group}"
}

I then specify a data source for the root module's tfstate file:

data "terraform_remote_state" "root_module" {
  backend = "local"
  config {
    path = "../../../../terraform.tfstate"
  }
}

and then reference this in my modules like so:

aws_instance "instance" {
  region = "us-east-1"
  security_groups = ["${data.terraform_remote_state.root_module.aws_security_group}"]
}

Aside from feeling a bit janky, this relies on the root module being updated with variable changes (an apply that does nothing will achieve this), and that updated file being available immediately to the modules you want the change to ripple across.

+1 for a better way.

@braderhart

This comment has been minimized.

braderhart commented Apr 5, 2018

@petewilcock Best workaround I've found is to use Ansible instead.

@accolver

This comment has been minimized.

accolver commented Apr 5, 2018

I got around it by defining my global variables in my root directory and them symlinking them in my module.

ln -s ../../variables.tf ./global_variables.tf

This symlink lives along side my scoped variables.tf file in my module(s).

@braderhart

This comment has been minimized.

braderhart commented Apr 5, 2018

@accolver How does that work for Windows users editing source files in Git?

@Jakexx360

This comment has been minimized.

Jakexx360 commented Apr 9, 2018

@accolver Your solution worked, but with absolute paths. Wish I didn't have to add a Makefile to the project in order to symlink all of my modules

@Alexhha

This comment has been minimized.

Alexhha commented Jun 20, 2018

@apparentlymart

Have I correctly understood that your example from #5480 (comment) still haven't been implemented?

@o6uoq

This comment has been minimized.

o6uoq commented Oct 3, 2018

+1

1 similar comment
@humanchair

This comment has been minimized.

humanchair commented Nov 1, 2018

+1

@tdmalone

This comment has been minimized.

tdmalone commented Nov 1, 2018

@o6uoq @humanchair Please use GitHub reactions rather than commenting with a +1. Comments add noise and additional notifications for those who are watching this issue, and they don’t actually contribute to Hashicorp’s internal issue trackers which do look at the reactions. Thanks!

@hammadzz

This comment has been minimized.

hammadzz commented Nov 29, 2018

@apparentlymart any update on when this can be expected? I see locals are in but am dying to be able to pass them through to a module soon.

@ispulkit

This comment has been minimized.

ispulkit commented Dec 6, 2018

Waiting for updates. Dying to see this happening.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment