count and for_each for modules

[AnthonyWC]

Is it possible to dynamically select map variable, e.g?

Currently I am doing this:

vars.tf

locals {
  map1 = {
    name1 = "foo"
    name2 = "bar"
  }
}

main.tf

module "x1" {
  source = "../"
  parameter = "${local.map1["name1"]}"
}

module "x2" {
  source = "../"
  parameter = "${local.map1["name2"]}"
}

This works but it's repetitive/DRY to hardcode the key name.

Ideally I want to able to do something like this:

module "x1" {
  source = "../"
  parameter = "${local.map1[$var.select]}"
}

Where I can dynamically alter the key variable in the same file. I thought about using null_data_source:

data "null_data_source" "test" {
  inputs = {
    current = "${var.selector}"
  }
}

parameter  = "${local.map1[data.null_data_source.test.outputs["current"]]}"

But don't think I will able to inject different variable to selector value since I can't use locals within module block.

[apparentlymart]

Hi @AnthonyWC!

I'm not sure I understood correctly your question, but you can use any reference in the [ ... ] index brackets that you could use outside of them in the same context. For example, you could add a new local value for which name to select, like this:

locals {
  map1 = {
    name1 = "foo"
    name2 = "bar"
  }
  selector = "name1"
}

module "x1" {
  source = "../"
  parameter = "${local.map1[local.selector]}"
}

If I'm not answering the write question here, it'd help to have a more concrete, real-world example so that it's easier to understand what you're trying to achieve. It can be hard to figure out your intent with just generic names like "map1", "name1", etc.

[AnthonyWC]

Ok so originally I had something like this:

module "x1" {
  source = "../"
  tag_Name = "value1"
}

module "x2" {
  source = "../"
  tag_Name = "value2"
}

I want to reduce the number of copying, so I change it:

locals {
  tag_Name_map = {
    name1 = "value1"
    name2 = "value2"
  }
}

module "x1" {
  source = "../"
  tag_Name = "${local.tag_Name_map[name1]}"
}

module "x2" {
  source = "../"
  tag_Name = "${local.tag_Name_map[name2]}"
}

But I think I should able to go further with something like this:
(pseudo-code)

module "x1" {
  source = "../"
  tag_Name = "${local.tag_Name_map[$var.name]}"
}

module "x2" {
  source = "../"
  tag_Name = "${local.tag_Name_map[$var.name]}"
}

Where I can select the local map variable based on another variable on a per module basis. Not sure if locals is the right way to achieve it.

[apparentlymart]

I think I am still not understanding properly... in your second example you shown indexes name1 and name2, which you replaced both with $var.name in the final example. It seems like you're looking for something that would allow $var.name to be "name1" for module x1 and "name2" for module x2, but Terraform has no way to know that x1 maps to "name1" and x2 names to "name2".

I think maybe what you want to do is insert the module's own name in there? So in that case your tag map would look like this:

locals {
  tag_Name_map = {
    x1 = "value1"
    x2 = "value2"
  }
}

In that case, what you want to do here is not possible at this time. Terraform's configuration format prefers being explicit over implicit, and also being direct rather than indirect. While of course this is subjective, the design principles for the language actually consider your first example to be better because the value can be seen directly inside the module block, rather than needing to refer elsewhere. Unless that value is also going to be used in another location, factoring it out into a separate local map doesn't actually reduce the amount of copying... it just places the same information in a different place.

I think the closest thing to what you want here would be a future feature to address the request in #953. Other work has prevented us from prototyping that further since my last comment here, but we want to eventually support both count and the for_each argument discussed in #17179, which could allow you to write something like this:

locals {
  tag_Name_map = {
    name1 = "value1"
    name2 = "value2"
  }
}

# Not yet implemented and details may change before release
module "x" {
  # Create an instance of this module for each element in locals.tag_Name_map
  for_each = "${locals.tag_Name_map}"

  source = "../"
  tag_Name = "${each.value}" # use each value from the map
}

[AnthonyWC]

Thanks for the detailed response. I was thinking maybe it was possible to somehow set the value of $var.name differently within each module. Maybe with inline template if it was allowed to be set differently within each module (which doesn't look to work that way currently).

The other direction may be to look outside of Terraform and use something else to dynamically generate the terraform file, which separate out the templating logic and won't affect Terraform itself. Maybe something like a lightweight version of pongo2 for Terraform or some other project like this one (https://github.com/mjuenema/python-terrascript).

[apparentlymart]

I'm not personally familiar with python-terrascript, but indeed code generation is a reasonable approach if you are working on a system that has lots of repetitive, systematic elements that you can generate from some high-level representation. For example, I've seen this used to generate various aws_iam_* resources (or rather, modules containing them) based on a CSV file of users, because the tabular form of a CSV file can be more convenient in some situations.

The module function in that program does seem like it could help you achieve what you want here, though I have not tested it. I assume that it will generate a JSON-formatted Terraform configuration like this (which should be named something.tf.json rather than just something.tf as for the native syntax):

{
  "module": {
    "x1": {
      "source": "../",
      "tag_Name": "foo"
    },
    "x2": {
      "source": "../",
      "tag_Name": "bar"
    }
  }
}

Once something like that for_each mechanism I described before is implemented, it will be possible to do more operations like this within Terraform itself, but we support the JSON format because we expect there will always be situations where code generation is preferable.

[kuwas]

@apparentlymart

We're currently also doing something similar to @AnthonyWC, running a templating engine to preprocess input and generate Terraform from them.

I'm sure this comes up often, but are there any plans for Terraform to introduce a native way to do this? I dont meant as a part of HCL which will handle the actual runtime options. Instead, I was thinking of something similar to the way that Ansible processes templates to generate python before using it for execution, Terraform can generate HCL from templates.

[jakauppila]

@pselle Now with resource for_each implemented, I was curious what it might look like for module for_each? Is most of the heavy lifting done at this point or is it still quite a large effort?

[pselle]

@jakauppila Effort is hard to say, but modules and resources are handled in different areas of Terraform -- additionally, resources already have count whereas modules don't as yet (thus, count and for_each for modules are both a bit greenfield).

As described in the 0.12 preview last year, I expect some groundwork has already been made (necessary changes to the statefile, reserving those keywords ...).

I hope that gives some better picture of where the project is at with regards to modules and for_each, even if it's not the most hoped-for answer :)

[kuwas]

Any timelines on the implementation of this?