Feature Request - Allow list/array in 'query' in 'external' data source

[vikas027]

Terraform Version

0.8.7

Affected Resource(s)

external data source

Terraform Configuration Files

data "external" "create_policy" {
  program = ["bash", "${path.module}/create_policy.sh"]

  query = {
    list_of_images = "${var.list_of_images}"
  }
}

Expected Behavior

List list_of_images should have been passed to the script.

Actual Behavior

Error refreshing state: 1 error(s) occurred:

* query: 1 error(s) decoding:

* '[list_of_images]' expected type 'string', got unconvertible type '[]interface {}'

---

Terraform Version

0.8.7

Affected Resource(s)

• external

Terraform Configuration Files

data "external" "create_policy" {
  program = ["bash", "${path.module}/create_policy.sh"]
}

create_policy.sh

#!/bin/bash
set -e
# This works
jq -n --arg foobaz "$FOOBAZ" '{"foobaz":$foobaz}'
# This throws an error
jq -n '{"Version":"2008-10-17","Statement":[{"Sid":"repo_policy","Effect":"Allow","Principal":{"AWS":["arn:aws:iam::${account_id}:root","arn:aws:iam::${account_id}:role/ecr_restricted_admin"]},"Action":"*"}]}'

Expected Behavior

Terraform shouldn't thrown an error when the the bash script is producing a valid json

$ bash modules/ecr/create_policy.sh  | jq .
{
  "Version": "2008-10-17",
  "Statement": [
    {
      "Sid": "repo_policy",
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "arn:aws:iam::${account_id}:root",
          "arn:aws:iam::${account_id}:role/ecr_restricted_admin"
        ]
      },
      "Action": "*"
    }
  ]
}

Actual Behavior

* data.external.create_policy: command "bash" produced invalid JSON: json: cannot unmarshal array into Go value of type string

Steps to Reproduce

Create the resource and the script files and run terraform plan

[apparentlymart]

Hi @vikas027! Sorry for this limitation.

Currently this data source is constrained by some assumptions made by Terraform's schema system, which doesn't allow for mixed types in a map of values.

This might change in future -- certainly more flexibility here is desirable if possible -- but for now the workaround is to pass in the list as a delimited string and then split it again in the external script:

  query = {
    list_of_images = "${join(" ", var.list_of_images)}"
  }

---

Incidentally, your filename create_policy.sh suggests that you are using this data source to perform an action. This isn't the intended purpose of a data source, so if you go down this path be sure to design the script such that it is able to detect when no action needs to be taken and exit quickly, or else the action you specify will be taken on every Terraform run. The other caveat is that as a data source it will, by default, happen during the plan phase, so Terraform's usual guarantees about not doing any actions during plan will not apply here.

In future they may be a resource "external" that provides a similar interface for the create, refresh, update and delete actions of managed resources, which would at that point be a more appropriate way to do what it seems you want to do here.

[vikas027]

Hey @apparentlymart ,

Thanks for the quick response. Yeah, I know what I am trying to do is not ideal but I could not an elegant way to run nested loops as asked and suggested here.

I am pretty sure this would be a common requirement. Any ideas/hints, how to do the same. Thanks.

[jflammia]

+1 for this issue. It would be nice for Terraform to actually support valid JSON output instead of comma delimited strings.

Here is what my external data source executes:
wget -qO- https://ip-ranges.amazonaws.com/ip-ranges.json | jq '.prefixes[] | select(.service=="CLOUDFRONT") | .ip_prefix' | sort -V -t '/' -k 1 | jq --slurp . | jq '{cloudfront_ips: .}'

It outputs:

{
  "cloudfront_ips": [
    "13.32.0.0/15",
    "13.54.63.128/26",
    "34.195.252.0/24",
    "35.162.63.192/26",
    "35.167.191.128/26",
    "52.15.127.128/26",
    "52.46.0.0/18",
    "52.52.191.128/26",
    "52.56.127.0/25",
    "52.57.254.0/24",
    "52.66.194.128/26",
    "52.78.247.128/26",
    "52.84.0.0/15",
    "52.199.127.192/26",
    "52.212.248.0/26",
    "52.220.191.0/26",
    "52.222.128.0/17",
    "54.182.0.0/16",
    "54.192.0.0/16",
    "54.230.0.0/16",
    "54.233.255.128/26",
    "54.239.128.0/18",
    "54.239.192.0/19",
    "54.240.128.0/18",
    "204.246.164.0/22",
    "204.246.168.0/22",
    "204.246.174.0/23",
    "204.246.176.0/20",
    "205.251.192.0/19",
    "205.251.249.0/24",
    "205.251.250.0/23",
    "205.251.252.0/23",
    "205.251.254.0/24",
    "216.137.32.0/19"
  ]
}

However, when I run terraform plan, I get:
... scripts/get_cloudfront_ips.sh" produced invalid JSON: json: cannot unmarshal array into Go value of type string

[debu99]

+1

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.