Unable to add multiple maps in terraform vars file.

[roshpr]

I'm unable to have multiple maps variable in terraform variables file. This was working until version 0.7.5 & started breaking from version 0.7.6. Not sure whether this was removed on purpose. Does not make sense if this provision was removed on purpose. I have many maps grouped into multiple variables.

Terraform Version

Terraform v0.7.7

Affected Resource(s)

variable file

Terraform Configuration Files

prod.tfvars file

private_intf = "eth0"

variable "bastion_amis" {
  type = "map"
  default = {
    us-west-1 = "ami-049d8641"
    us-west-2 = "ami-cb699cab"
    us-east-1 = "ami-2d39803a"
  }
}
variable "amis" {
  description = "Base AMI to launch the instances with"
  type = "map"
  default = {
    us-east-1 = "ami-2d39803a"
    us-west-1 = "ami-049d8641"
    us-west-2 = "ami-9abea4fb"
    us-east-1 = "ami-a6b8e7ce"
  }
}
variable "deployment" {
  type = "map"
  default = {
    name = "D"
    type = "dev"
    csp_region = "central"
  }
}
variable "aws" {
  type = "map"
  default = {
    profile = "cbm"
    access_key = ""
    secret_key = ""
    key_path = "roshpr.net-key.pem"
    key_name = "roshpr.net-key"
    region = "us-east-1"
  }
}

Debug Output

file=aws/deployment_input/terraform_prod_central_sample.tfvars aws/deployment/production/central/
2016/10/24 20:53:50 [INFO] Terraform version: 0.7.7  fa6a83ebdc323f2b415779786e102e69ddbf9a48
2016/10/24 20:53:50 [INFO] CLI args: []string{"/Users/roshpr/bin/terraform_0.7.7", "plan", "--state=aws/tfstates/us_east_1/production_central_terraform.tfstate", "--var-file=aws/deployment_input/terraform_prod_central_sample.tfvars", "aws/deployment/production/central/"}
2016/10/24 20:53:50 [DEBUG] Detected home directory from env var: /Users/roshpr
2016/10/24 20:53:50 [DEBUG] Detected home directory from env var: /Users/roshpr
2016/10/24 20:53:50 [DEBUG] Attempting to open CLI config file: /Users/roshpr/.terraformrc
2016/10/24 20:53:50 [DEBUG] File doesn't exist, but doesn't need to. Ignoring.
2016/10/24 20:53:50 [DEBUG] Detected home directory from env var: /Users/roshpr
2016/10/24 20:53:50 [DEBUG] plugin: waiting for all plugin processes to complete...
invalid value "aws/deployment_input/terraform_prod_central_sample.tfvars" for flag -var-file: multiple map declarations not supported for variables

Expected Behavior

Multiple maps should be allowed in the tfvars file

Actual Behavior

Error: invalid value "aws/deployment_input/terraform_prod_central_sample.tfvars" for flag -var-file: multiple map declarations not supported for variables

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

1. terraform apply

References

This issue is caused because of the following change
9fc50a7#diff-8eeb149e47145a08d13764b14a34d35e

[zdxn]

This is a huge issue for me currently.
Thanks for opening it.

[zdxn]

This is a huge issue for me currently.
Thanks for opening it.

[pporada-gl]

While not a solution, you can use Terraform to search for AMIs for you. This still leaves you with 2 maps in your vars file though which is broken.

data "aws_ami" "jenkins" {
    most_recent   = true
    owners        = ["self"]
    filter {
        name    = "tag:OS_Version"
        values  = ["Fedora"]
    }
    filter {
        name    = "tag:Release"
        values  = ["25"]
    }
    filter {
        name    = "architecture"
        values  = ["x86_64"]
    }
    filter {
        name    = "virtualization-type"
        values  = ["hvm"]
    }
    filter {
        name    = "name"
        values  = ["Docker*"]
    }
}

[pporada-gl]

While not a solution, you can use Terraform to search for AMIs for you. This still leaves you with 2 maps in your vars file though which is broken.

data "aws_ami" "jenkins" {
    most_recent   = true
    owners        = ["self"]
    filter {
        name    = "tag:OS_Version"
        values  = ["Fedora"]
    }
    filter {
        name    = "tag:Release"
        values  = ["25"]
    }
    filter {
        name    = "architecture"
        values  = ["x86_64"]
    }
    filter {
        name    = "virtualization-type"
        values  = ["hvm"]
    }
    filter {
        name    = "name"
        values  = ["Docker*"]
    }
}

[Zordrak]

Can we please get some comment from Hashicorp as to whether this is a bug that has somehow slopped testing, or whether this is a feature that we just don't understand and need to design around?

[Zordrak]

Can we please get some comment from Hashicorp as to whether this is a bug that has somehow slopped testing, or whether this is a feature that we just don't understand and need to design around?

[jacknagz]

This is also a big problem for me, I load a lot of variables from a JSON file

[jacknagz]

This is also a big problem for me, I load a lot of variables from a JSON file

[mattwilmott]

Same issue here, again not sure if its by design or ....

[mattwilmott]

Same issue here, again not sure if its by design or ....

[mikey-]

This also happening in v0.7.11 :c

[mikey-]

This also happening in v0.7.11 :c

[mattwilmott]

And Terraform v0.8.0-dev

[mattwilmott]

And Terraform v0.8.0-dev

[manterfield]

So, I'm not sure this error message is actually accurate. I had the same error and assumed as you did, that my file with multiple maps was causing it (cos, you know... that's what it said).

However, I also had a .tfvars file written in HCL. Once I changed that file to use the key = value syntax, but kept the other vars file with multiple maps... the error went away.

I'm not sure what the proper format for the tfvars file currently is. It looks like this information was removed from the docs with this commit.

[manterfield]

So, I'm not sure this error message is actually accurate. I had the same error and assumed as you did, that my file with multiple maps was causing it (cos, you know... that's what it said).

However, I also had a .tfvars file written in HCL. Once I changed that file to use the key = value syntax, but kept the other vars file with multiple maps... the error went away.

I'm not sure what the proper format for the tfvars file currently is. It looks like this information was removed from the docs with this commit.

[manterfield]

I've just had a look at the code for parsing tfvars files. It looks like it should still be in the key=value format, so your tfvars file posted above won't parse.

This issue should be kept open though, as this needs to be replaced in the docs and the error message itself seems to be erroneous.

[manterfield]

I've just had a look at the code for parsing tfvars files. It looks like it should still be in the key=value format, so your tfvars file posted above won't parse.

This issue should be kept open though, as this needs to be replaced in the docs and the error message itself seems to be erroneous.

[mitchellh]

@manterfield is right!

You can specify maps across multiple tfvar files with #8219 (fix queued). And you can specify multiple vars in a tfvars file just fine. But the syntax of your example here is wrong.

I agree this bug should be left open to improve parsing/validation of this file.

[mitchellh]

@manterfield is right!

You can specify maps across multiple tfvar files with #8219 (fix queued). And you can specify multiple vars in a tfvars file just fine. But the syntax of your example here is wrong.

I agree this bug should be left open to improve parsing/validation of this file.

[JulienChampseix]

@mitchellh have you a syntax example of tfvars (including map) we should use for avoid error

multiple map declarations not supported for variables

The example from @roshpr should also support, no ? What's your feeling about it ?

I'm using version 0.9.4 and having the same trouble of multiple map declarations as mentionned at the beginning of this issue.

[JulienChampseix]

@mitchellh have you a syntax example of tfvars (including map) we should use for avoid error

multiple map declarations not supported for variables

The example from @roshpr should also support, no ? What's your feeling about it ?

I'm using version 0.9.4 and having the same trouble of multiple map declarations as mentionned at the beginning of this issue.

[rixka]

version 0.8.4

I received this error too however it was due to a copy and paste issue which resulted in duplicate map declarations.

variable_map = { ... }
variable_map = { ... }

I feel that the "solution" to this is simply to update the error message so that people know what to look at the contents of their terraform.tfvars file rather than assume it is a bug.

[rixka]

version 0.8.4

I received this error too however it was due to a copy and paste issue which resulted in duplicate map declarations.

variable_map = { ... }
variable_map = { ... }

I feel that the "solution" to this is simply to update the error message so that people know what to look at the contents of their terraform.tfvars file rather than assume it is a bug.

[dbeckham]

This looks to be a specific issue for the JSON format. Maps appear to work correctly for the HCL format, but is very broken when using JSON.

Here's an example of the issue using two configuration files that have exactly the same data, one described in HCL format and one in JSON format.

Using Terraform 0.9.11

hcl.tfvars

test_map = {
    "key1" = {
        "index1" = "data2"
        "index2" = "data2"
    }
    "key2" = {
        "index1" = "data1"
    }
}

json.tfvars

{ "test_map": { "key1": {"index1": "data1", "index2": "data2"}, "key2": {"index1": "data1"} } }

main.tf

variable "test_map" {
    type = "map"
}

If you run terraform using the HCL file, you get the normal Terraform output telling you:

$ terraform plan -var-file=hcl.tfvars
No changes. Infrastructure is up-to-date.

If you run the same command using the JSON version, you get the error:

$ terraform plan -var-file=json.tfvars
invalid value "json.tfvars" for flag -var-file: multiple map declarations not supported for variables

[dbeckham]

This looks to be a specific issue for the JSON format. Maps appear to work correctly for the HCL format, but is very broken when using JSON.

Here's an example of the issue using two configuration files that have exactly the same data, one described in HCL format and one in JSON format.

Using Terraform 0.9.11

hcl.tfvars

test_map = {
    "key1" = {
        "index1" = "data2"
        "index2" = "data2"
    }
    "key2" = {
        "index1" = "data1"
    }
}

json.tfvars

{ "test_map": { "key1": {"index1": "data1", "index2": "data2"}, "key2": {"index1": "data1"} } }

main.tf

variable "test_map" {
    type = "map"
}

If you run terraform using the HCL file, you get the normal Terraform output telling you:

$ terraform plan -var-file=hcl.tfvars
No changes. Infrastructure is up-to-date.

If you run the same command using the JSON version, you get the error:

$ terraform plan -var-file=json.tfvars
invalid value "json.tfvars" for flag -var-file: multiple map declarations not supported for variables

[neybar]

Is there any update on this issue? I just ran into it again. terraform.tfvars (in HCL) work with an array of maps, but the same data in terraform.tfvars.json will produce the error message.

[neybar]

Is there any update on this issue? I just ran into it again. terraform.tfvars (in HCL) work with an array of maps, but the same data in terraform.tfvars.json will produce the error message.

[apparentlymart]

Hi all! Sorry for all the weirdness here.

Unfortunately right now there are some quirks in the mapping of JSON to HCL that arise from some ambiguities in how the JSON version of HCL is defined. (To be specific, HCL JSON allows blocks to either be single JSON objects or arrays of JSON objects, and it "normalizes" single JSON objects to be single-element arrays during parsing, which is the wrong assumption in this case.)

We're planning to make a number of improvements to the configuration language together, and addressing the ambiguities in the JSON parsing is a big part of that. The main focus of this work is the main configuration files (.tf files) but since tfvars files are also HCL files they will also benefit from these improvements. We're still in the planning phase of this work, since many of these improvements are breaking changes that we want to release together in a single release; we'll have more to share on this soon in other issues.

For now, unfortunately I think the main workaround here is to use the native HCL syntax, though I understand that's frustrating when you are generating these files programmatically.

[apparentlymart]

Hi all! Sorry for all the weirdness here.

Unfortunately right now there are some quirks in the mapping of JSON to HCL that arise from some ambiguities in how the JSON version of HCL is defined. (To be specific, HCL JSON allows blocks to either be single JSON objects or arrays of JSON objects, and it "normalizes" single JSON objects to be single-element arrays during parsing, which is the wrong assumption in this case.)

We're planning to make a number of improvements to the configuration language together, and addressing the ambiguities in the JSON parsing is a big part of that. The main focus of this work is the main configuration files (.tf files) but since tfvars files are also HCL files they will also benefit from these improvements. We're still in the planning phase of this work, since many of these improvements are breaking changes that we want to release together in a single release; we'll have more to share on this soon in other issues.

For now, unfortunately I think the main workaround here is to use the native HCL syntax, though I understand that's frustrating when you are generating these files programmatically.

[Emerson]

Should this be working if we're using the native HCL syntax? I'm still getting the error when using native HCL

[Emerson]

Should this be working if we're using the native HCL syntax? I'm still getting the error when using native HCL

[brycehemme]

I'm still getting this issue as well. It appears this is an issue with HCL and JSON both. Any timeline or even a plan, yet?

[brycehemme]

I'm still getting this issue as well. It appears this is an issue with HCL and JSON both. Any timeline or even a plan, yet?

[apparentlymart]

Hi all,

I think there are a few different problems here that all have the same symptom, which may be causing some confusion since the responses from Mitchell and I above are addressing different situations.

I will try here to talk about each of these situations separately.

tfvars in JSON format

As I noted above, the current handling of JSON has some flaws because Terraform uses HCL's low-level API rather than its high-level API, and so it misses out on some of the normalization steps done to support configuration in both formats.

I'm not sure exactly what is causing the problem with JSON .tfvars files, but I think it's the same issue that makes locals blocks in main be ignored or produce errors unless they are written exactly the right way. In situations where the attributes are free-form rather than fixed based on a schema Terraform must iterate over them, and this process seems to happen incorrectly for certain JSON constructs.

We are currently in the process of integrating an improved version of HCL that, amongst other things, takes a different approach to JSON processing that mean it is able to successfully parse more JSON forms and -- crucially -- to produce explicit error messages for JSON it is not able to interpret.

This is the current focus of the Terraform team at HashiCorp and will be included in a forthcoming major release.

variable blocks in .tfvars files

The original example in this issue was of a .tfvars file containing variable blocks, which causes a confusing message today because (as noted above) Terraform iterates over all of the top-level constructs in the file to produce a key/value map, and those variable blocks are currently interpreted as attributes named variable, which the loader then rejects (with that confusing error message) because there are multiple instances and they contain content that isn't valid for a variable value.

The new parser implementation mentioned above will also address this problem, since it explicitly understands the difference between an attribute and a nested block, and is thus able to determine that those variable blocks are invalid in this context and produce a better error message. Although we aren't planning to do this for the initial rollout of the new parser, it is now technically possible for us to recognize the presence of variable blocks and produce a specialized error message for that case, which we will probably do given that this has caused confusion for a number of different people.

This was already discussed in comments above but I just want to state it again to be explicit: a tfvars file is just a collection of key = value pairs, assigning values to variables. The variable declarations themselves (the variable blocks) belong in normal .tf files along with all other configuration. Although these two file formats use HCL syntax, they have a different purpose and structure.

A .tfvars file should look like this:

foo = "bar"
baz = "foo"

Each of the attribute names in a .tfvars file should have a corresponding variable block in one of your .tf files:

variable "foo" {
}

variable "bar" {

}

The .tf file declares that the variable exists, while the .tfvars file assigns a value to that variable.

Duplicate assignment to the same variable

It's possible to provide multiple -var and -var-file arguments when running Terraform and between those arguments there may be conflicting assignments to the same variable. This is allowed only if these assignments are given as separate arguments. Within a single .tfvars file all of the keys must be unique.

This is actually the situation that the error message in question is trying to talk about, but because there are currently several different situations that lead to the same message it's very confusing.

Since the HCL parser/decoder has more information about what Terraform is expecting to find in these files, this situation will soon be a separate error message than the others above. Again, this must wait until we've finished integrating the new parser. Since this is intentionally an error situation the error will remain in the new version but it should, at least, be a more helpful error message that gives the source location information for both definitions and explains more clearly what the problem is.

---

I hope all of the above is helpful context for what's going on here. The same set of work will lead to the resolution of all of these issues, along with many more tagged with the "config" label.

This was originally marked as a core regression before the problem was completely understood, but the change that motivated the original request was actually stricter validation of this file where before it was ignoring the invalid elements altogether. Therefore I'm going to retag this as a configuration bug, which will make it easier for us to find it again once we get closer on the work I described above so that we can post updates.

I have to ask for your patience while we get through the current set of work. It's a pretty substantial change that cuts across all of Terraform's subsystems, so we need to proceed carefully and cautiously. As noted above, it is the current priority for the Terraform team at HashiCorp and so it will be released as soon as we're sure it's ready.

[apparentlymart]

Hi all,

I think there are a few different problems here that all have the same symptom, which may be causing some confusion since the responses from Mitchell and I above are addressing different situations.

I will try here to talk about each of these situations separately.

tfvars in JSON format

As I noted above, the current handling of JSON has some flaws because Terraform uses HCL's low-level API rather than its high-level API, and so it misses out on some of the normalization steps done to support configuration in both formats.

I'm not sure exactly what is causing the problem with JSON .tfvars files, but I think it's the same issue that makes locals blocks in main be ignored or produce errors unless they are written exactly the right way. In situations where the attributes are free-form rather than fixed based on a schema Terraform must iterate over them, and this process seems to happen incorrectly for certain JSON constructs.

We are currently in the process of integrating an improved version of HCL that, amongst other things, takes a different approach to JSON processing that mean it is able to successfully parse more JSON forms and -- crucially -- to produce explicit error messages for JSON it is not able to interpret.

This is the current focus of the Terraform team at HashiCorp and will be included in a forthcoming major release.

variable blocks in .tfvars files

The original example in this issue was of a .tfvars file containing variable blocks, which causes a confusing message today because (as noted above) Terraform iterates over all of the top-level constructs in the file to produce a key/value map, and those variable blocks are currently interpreted as attributes named variable, which the loader then rejects (with that confusing error message) because there are multiple instances and they contain content that isn't valid for a variable value.

The new parser implementation mentioned above will also address this problem, since it explicitly understands the difference between an attribute and a nested block, and is thus able to determine that those variable blocks are invalid in this context and produce a better error message. Although we aren't planning to do this for the initial rollout of the new parser, it is now technically possible for us to recognize the presence of variable blocks and produce a specialized error message for that case, which we will probably do given that this has caused confusion for a number of different people.

This was already discussed in comments above but I just want to state it again to be explicit: a tfvars file is just a collection of key = value pairs, assigning values to variables. The variable declarations themselves (the variable blocks) belong in normal .tf files along with all other configuration. Although these two file formats use HCL syntax, they have a different purpose and structure.

A .tfvars file should look like this:

foo = "bar"
baz = "foo"

Each of the attribute names in a .tfvars file should have a corresponding variable block in one of your .tf files:

variable "foo" {
}

variable "bar" {

}

The .tf file declares that the variable exists, while the .tfvars file assigns a value to that variable.

Duplicate assignment to the same variable

It's possible to provide multiple -var and -var-file arguments when running Terraform and between those arguments there may be conflicting assignments to the same variable. This is allowed only if these assignments are given as separate arguments. Within a single .tfvars file all of the keys must be unique.

This is actually the situation that the error message in question is trying to talk about, but because there are currently several different situations that lead to the same message it's very confusing.

Since the HCL parser/decoder has more information about what Terraform is expecting to find in these files, this situation will soon be a separate error message than the others above. Again, this must wait until we've finished integrating the new parser. Since this is intentionally an error situation the error will remain in the new version but it should, at least, be a more helpful error message that gives the source location information for both definitions and explains more clearly what the problem is.

---

I hope all of the above is helpful context for what's going on here. The same set of work will lead to the resolution of all of these issues, along with many more tagged with the "config" label.

This was originally marked as a core regression before the problem was completely understood, but the change that motivated the original request was actually stricter validation of this file where before it was ignoring the invalid elements altogether. Therefore I'm going to retag this as a configuration bug, which will make it easier for us to find it again once we get closer on the work I described above so that we can post updates.

I have to ask for your patience while we get through the current set of work. It's a pretty substantial change that cuts across all of Terraform's subsystems, so we need to proceed carefully and cautiously. As noted above, it is the current priority for the Terraform team at HashiCorp and so it will be released as soon as we're sure it's ready.

[beoboo]

+1

[beoboo]

+1