New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ability to iterate over a list/map and create resources #8573

Closed
garo opened this Issue Aug 31, 2016 · 15 comments

Comments

Projects
None yet
@garo
Copy link

garo commented Aug 31, 2016

Currently Terraform and HCL can do iteration using count and then the lookup() and element() functions. These work for simple cases, but they fail if there's a need to do a more complex iterations and nested definitions.

Consider this example what I would want to do:

variable "topics" {
  default = ["new_users", "deleted_users"]
}

variable "environments" {
  default = ["prod", "staging", "testing", "development]
}

for $topic in topics {
  # Define SNS topics which are shared between all environments
  resource "aws_sns_topic" "$topic.$env" { ... }

  for $env in environments {
    # Then for each topic define a queue for each env
    resource "aws_sqs_queue" "$topic.$env-processors" { ... }

    # And bind the created queue to its sns topic
    resource "aws_sns_topic_subscription" "$topic.$env-to-$topic.$env-processors" {
      topic_arn = "${aws_sns_topic.${topic.$env}.arn}"
      endpoint = "${aws_sqs_queue.{$topic.$env-processors}.arn}"
    }
  }
}

The especially problematic parts are the topic_arn and endpoint properties in this example, where I want to reference an ARN for another resource, which name is created based on iterating two different lists.

There has been similar suggestions like #4410 and #58

The best what I've come up so far is this:


variable "foo" {
    default = ["1", "2", "3"]
}

variable "bar" {
    default = ["a", "b"]
}

resource "aws_sns_topic" "test" {
    name = "${element(var.foo, count.index / length(var.bar))}-${element(var.bar, count.index)}"
    count = "${length(var.foo) * length(var.bar)}"
}

but this doesn't solve the aws_sns_topic_subscription problem, it's error prone to mistakes as it requires complex expression in the name and count fields.

@mengesb

This comment has been minimized.

Copy link
Contributor

mengesb commented Sep 2, 2016

@garo A more rich iterator strategy would certainly solve this problem (perhaps loop) but more so you're also looking for nested or dependent interpolation (variables containing interpolation) which certainly... I would love. It does, however, break the declarative nature of things and not all 'creative' solutions are able to function purely declaratively.

I do something similar to what you have with aws_sns_* or aws_sqs_*, but with subnets, networks, etc. I've chosen to go with maps, and utilize length(), element(), keys(), and values().

https://github.com/mengesb/tf_hachef/blob/master/main.tf#L40

In the above link you can see that I dynamically create the subnets based on the information held in the var.subnets map. I have a number of things keyed off this, however to the user, all they really need to do is create a map with suitable key-value pairs that aws consumes. I know this isn't really a solution as it were, but as close to a work-around as possible.

I, for one, would love to see a looping iterator and also a guard of sorts (like a not_if, only_if, etc..), however while not declarative I do have other plans with extremely complex math result sets to control count ... which I'd rather not do as it cannot be understood by many.

@garo

This comment has been minimized.

Copy link

garo commented Sep 5, 2016

Thanks @mengesb for the comments. I tend to think that using loops and iterations in this kind of setup doesn't break the declarative nature how Terraform currently works. As HCL is currently translated into JSON, this kind of logic could be implemented into HCL itself, so that it would expand the JSON it produces. So for example a code which would iterate over a list of two entries and declare two resources, the produced JSON would look like there would be no loop but just two declared resources.

There are other languages which have control and especially iteration structures, for example the C++ template metaprogramming which executes at compile time, but which are still declarative in nature as they only produce constant definitions and declarations.

Actually I think that the current limitations forces Terraform users to workaround with a really complex way of using count, element()and other functions, which could be made much more human readable and easier to debug with proper control structures instead.

@fgimian

This comment has been minimized.

Copy link

fgimian commented Oct 10, 2016

I would absolutely love to see loops (and conditionals) in HCL. It would improve Terraform significantly.

I think even our Hashicorp friends are craving for this when you see the effort they had to go to so they can iterate through items in their best-practices repo.

resource "aws_subnet" "public" {
  vpc_id            = "${var.vpc_id}"
  cidr_block        = "${element(split(",", var.cidrs), count.index)}"
  availability_zone = "${element(split(",", var.azs), count.index)}"
  count             = "${length(split(",", var.cidrs))}"

  tags      { Name = "${var.name}.${element(split(",", var.azs), count.index)}" }
  lifecycle { create_before_destroy = true }

  map_public_ip_on_launch = true
}

ewww 😄

Cheers
Fotis

@fgimian

This comment has been minimized.

Copy link

fgimian commented Oct 22, 2016

Just want to clarify that you don't actually have to do what was done above in the best-practices repo. Instead, use lists of the same length for everything and obtain the respective item in each list 😄

e.g.

resource "aws_subnet" "tools" {
  count = "${length(var.azs)}"
  vpc_id = "${var.vpc_id}"
  availability_zone = "${element(var.azs, count.index)}"
  cidr_block = "${element(var.tools_subnet_cidrs, count.index)}"
  tags {
    Name = "Tools ${element(var.az_labels, count.index)}"
  }
}
@davidk01

This comment has been minimized.

Copy link

davidk01 commented Dec 27, 2016

I've given up on trying to use the declarative iteration/lookup/interpolation constructs within terraform. Instead whenever I need to create some linked set of resources I do it with ERB. I'm a lot happier after I gave up on trying to do everything from within terraform. This is not a ding against terraform. All custom external DSLs have this problem. It would have been different if terraform was a library like Chef that allowed constructing the template structure with actual code but alas it is not. This actually opens up a possibility for someone to take up that torch. Build a real DSL in a real language that can generate the proper terraform template.

@Crapworks

This comment has been minimized.

Copy link

Crapworks commented Mar 1, 2017

Just as a heads up, I had the same problem myself and created a small wrapper that you can use to use Jinja2 templates in terraform. This script also makes all your variables available to Jinja. Just in case anyone is looking for a ready to use script :)

https://github.com/Crapworks/terratools/tree/master/terratemplate

@naikajah

This comment has been minimized.

Copy link

naikajah commented Mar 30, 2017

@garo This is a small implementation I wrote which iterates over a map.
The sample map looks like following:

default = {
    "eu-west-1a" = "10.xx.xx.xx/24,10.xx.xx.xx/24"
    "eu-west-1b" = "10.xx.xx.xx/24,10.xx.xx.xx/24"
    "eu-west-1c" = "10.xx.xx.xx/24,10.xx.xx.xx/24"
  }

And then using the combination of VPC's region, data source aws_availability_zones and some simple mathematic operations I was able to iterate over the map. Please have a look into the code below:

https://github.com/naikajah/AWS-Infrasetup/tree/master/subnet.tf

The code looks little complex but it does the trick

@ZeroPointEnergy

This comment has been minimized.

Copy link

ZeroPointEnergy commented Jul 4, 2017

@naikajah this is amazing and very helpful. Thought about something like this before but did not even try since I was burned so many times already by Terrafrom when I tried to do some clever workaround. I'm amazed this is actually working. Thanks for sharing.

@pll

This comment has been minimized.

Copy link

pll commented Sep 20, 2017

None of this helps with taking a basic Key/Value map to be used for tagging and applying it to something like an ASG, which uses a completely different tagging API.

It would be nice if there were a way to take a basic map iterate over that to create a decent map that could used for ASGs.

@shroo-paulli

This comment has been minimized.

Copy link

shroo-paulli commented Apr 11, 2018

There is a simpler solution: Terraform in JavaScript (or Python or Ruby). The concept is simple, instead of enhancing DSL to provide basic programming constructs (if, else, for, while, etc.), we can put DSL in JavaScript. In the web ecosystem, we have seen a few good use cases of that: React JSX  -  HTML in JavaScript, styled components  - CSS in JavaScript.

Please refer to my article to see an example usage: Add loops, conditions and logic in Terraform or CloudFormation using JavaScript

@pll

This comment has been minimized.

Copy link

pll commented Apr 11, 2018

It appears that all you're doing is templating the DSL itself. Is there something special about JS that makes JS a better choice than python and jinja2 or ruby and erb as language choice?

I think I would not want to go that direction in general. The point of a DSL is to remove the overhead of a programming language and to be able to concisely declare the desired end-state. By tightly coupling and intertwining the DSL and a programming template layer, you obscure what that desired end-state is supposed to be. And, perhaps, if your infrastructure is so complex that there is no way around doing things this way, that might be a sign that your design is too complex to begin with and should be broken down into smaller, simpler components that can be declaratively stated using the DSL.

@shroo-paulli

This comment has been minimized.

Copy link

shroo-paulli commented Apr 11, 2018

@pll No Paul, I am not saying it is better than jinja2 or ruby, just another alternative, but I do think it is better than the terraform -> count tricks (when it is complicated).

I have also demonstrated an advanced use case that you can never achieve using just the terraform DSL: find the ARN of an AWS Lambda function (provisioned by the serverless framework) then use it in the PreSignUp trigger of AWS Cognito (provisioned by terraform).

In addition, JavaScript will generate an output file that is your final terraform file that you can check-in to the source control. It will be easier to read than following (from the original issue comment):

resource "aws_sns_topic" "test" {
    name = "${element(var.foo, count.index / length(var.bar))}-${element(var.bar, count.index)}"
    count = "${length(var.foo) * length(var.bar)}"
}
@pll

This comment has been minimized.

Copy link

pll commented Apr 11, 2018

@shroo-paulli Hi Paul, okay, I wasn't sure if you were advocating for JS because of something inherent to JS when it comes to templating that makes it a better choice. I don't know JS, so I wasn't sure is there was something I was missing.

I'm not sure I fully understand your use-case wrt to finding the ARN. Are you saying the Lambda was created/generated by another Lambda and is not created by terraform to begin with? If so, I have not had to deal with that scenario yet, so I can see the difficulty there.

I do agree, there are many things terraform is not suited for. And in those cases, falling back to the AWS API in a real language is often the only choice. I try very hard to keep those things to a minimum, but they are inevitable!

@engineforce

This comment has been minimized.

Copy link

engineforce commented Apr 11, 2018

@pll Yes I am using serverless framework to provision AWS Lambda (because it is specialized in serverless deployment and development). Never mind, it is just and example to show that we can leverage the template layer (JavaScript or Python or Ruby) on top of terraform DSL when dealing with complex thing, instead of fallback all the way to AWS API.

@apparentlymart apparentlymart added config and removed core labels Aug 1, 2018

@apparentlymart

This comment has been minimized.

Copy link
Contributor

apparentlymart commented Aug 1, 2018

Hi all! I just spotted this issue again after losing track of it due to it using an outdated labeling scheme. In the mean time, the same need was described in #17179 which contains some discussion about how we're planning to address this.

Although this issue is older, because #17179 contains more recent discussion I'm going to close this one just to consolidate the discussion over there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment