Unable to create aws_waf_ipset from a list of IPs

[jmulloy-simplisafe]

I would like to use terraform to load an AWS WAF whitelist and load the same IP whitelist into consul so our API apps can pick it up to enforce access by IP. I can't find any way to load a list of IPs into an aws_waf_ipset resource. This problem seems to have been solved for security groups by adding a separate aws_security_group_rule resource that can be iterated via count. Perhaps this would be the easiest solution for this problem.

I've also been trying to do the reverse, getting the IPs out of a manually configured aws_waf_ipset resource and turning it into a list that I could load into consul, but I can't get that to work either.

Also the documentation for this resource doesn't make it clear that repeating the stanza is how you add multiple IPs to the resource.

Terraform Version

0.7.8

Affected Resource(s)

• aws_waf_ipset

Terraform Configuration Files

resource "aws_waf_ipset" "foobar_whitelist" {
  name = "tf-whitelist-test"
  ip_set_descriptors {
    type = "IPV4"
    value = "10.10.10.10/32"
  }
  ip_set_descriptors {
    type = "IPV4"
    value = "20.20.20.20/32"
  }
}

[sharmaansh21]

@jmulloy-simplisafe yes adding multiple stanza will add multiple ips

[rmccubbin]

Just to note you can also use an array as follows:

resource "aws_waf_ipset" "foobar_whitelist" {
  name = "tf-whitelist-test"
  ip_set_descriptors = [ {
    type = "IPV4"
    value = "10.10.10.10/32"
  },
  {
    type = "IPV4"
    value = "20.20.20.20/32"
  } ]
}

However the original problem that we can't go from a list of IPs to a list of ip_set_descriptors still seems to remain. For example I have > 1000 ips which I would like to blacklist. I can read those in from a file using "${split("\n", file("ips.txt"))}" but I am struggling to find a way to turn that list into the ip_set_descriptors. I looked at formatList and templating but it seems like those would just create a list of strings e.g. ["{ value=\"10.10.10.10/32\"...}"] rather than an actual object. If ip_set_descriptors were its own resource we could use count operations or if there were some means of creating objects for a list that would also solve the problem.

[allanalvaro]

I'm looking forward to having this feature. I have a long list of IPs to add and I had to write an external python script using boto3 to create that ipset.

[mgladdish]

I ended up using https://github.com/cbroglie/terrastache to get around this deeply annoying limitation

[butla]

I resolved that by putting a list of maps in my .tfvars file, like this:

whitelisted_ips = [
    { value = "1.2.3.4/32", type="IPV4"},
    { value = "2.3.4.5/28", type="IPV4"},
]

And assigning that to ip_set_descriptors in the module:

variable "whitelisted_ips" { type = "list" }

resource "aws_waf_ipset" "whitelisted_ips" {
  name = "WhitelistedIps"
  ip_set_descriptors = "${var.whitelisted_ips}"
}

Having to put type="IPV4" on every single line is a bit offensive to me as a programmer, but it works :) And it's doesn't require as much engineering effort as starting to use templates to pre-process Terraform scripts.

@rmccubbin I guess #7034 will be our salvation :)

[leighmhart]

I use established lists in a repo based module which are used for security groups and other functions elsewhere which are in the format of:

cidr_blocks = [
  "X.X.X.X/Y",
  "X.X.X.X/Y",
]

which I would love to re-use and/or reference in WAF IP Sets. I can get Terraform to render the required format in JSON which looks fine in an output variable, but can't get it to work as a parameter to ip_set_descriptor in the aws_wafregional_ipset resource.

data "template_file" "template" {
  template = <<TXT
{
  "type": "$${type}",
  "value": "$${value}"
}
TXT
  count    = "${length(module.offices.cidr_blocks)}"
  vars {
    type = "IPV4"
    value = "${module.offices.cidr_blocks[count.index]}"
  }
}

The output looks good:

output "json" {
  value = "${data.template_file.template.*.rendered}"
}

renders to:

json = [
    {
  "type": "IPV4",
  "value": "X.X.X.X/Y"
}
,
    {
  "type": "IPV4",
  "value": "X.X.X.X/Y"
}
]

but when I try to use it:

resource "aws_wafregional_ipset" "test" {
  name = "test"

  ip_set_descriptor = ["${data.template_file.data_json.*.rendered}"]
}

I get

Error: aws_wafregional_ipset.test: "ip_set_descriptor.0.type": required field is not set
Error: aws_wafregional_ipset.test: "ip_set_descriptor.0.value": required field is not set

If I don't surround the ip_set_descriptor value with [] I get:

Error: aws_wafregional_ipset.test: ip_set_descriptor: should be a list

[leighmhart]

I ended up using template_file to create a CloudFormation template to create the WAF Regional IP Sets and then referenced them by ID in my TF hardcoded elsewhere. This was much simpler than duplicating my IP lists.

# in my offices module
output "cidr_blocks" {
  value = [
    "X.X.X.X/Y",
   "X.X.X.X/Y",
  ]
}

# ... in my ipsets-cf.tf - I load the offices module

data "template_file" "ipsets" {
  template = <<JSON
{
  "Type": "$${type}",
  "Value": "$${value}"
}
JSON
  count = "${length(module.offices.cidr_blocks)}"
  vars {
    type  = "IPV4"
    value = "${module.offices.cidr_blocks[count.index]}"
  }
}

data "template_file" "waf_ipset" {
  template = <<JSON
{
  "AWSTemplateFormatVersion": "2010-09-09",
  "Description": "AWS WAF Template to load WAF IP Sets",
  "Resources": {
    "WAFRegionalIPSet": {
      "Type": "AWS::WAFRegional::IPSet",
      "Properties": {
        "Name": "Offices",
        "IPSetDescriptors": [
          $${ipsets}
        ]
      }
    }
  }
}
JSON
  vars {
    ipsets = "${join(",", data.template_file.ipsets.*.rendered)}"
  }
}

resource "aws_cloudformation_stack" "waf-ipset" {
  name = "waf-ipset"
  template_body = "${data.template_file.waf_ipset.rendered}"
}

Hopefully this helps someone and saves them 4 hours of googling / fighting interpolation limitations.

[rdonkin]

@allandrick This blog post has code to create an ip-whitelist module that can be used to create an ipset - not looked at in detail but might be relevant: https://jonnyzzz.com/blog/2019/04/29/terraform-waf/

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.