Arrays where they shouldn't be

[jantman]

I'm using -reverse to convert an HCL file, specifically configuration for the HashiCorp Vault Secure Introduction client (a proprietary, Enterprise-only Vault add-on); HCL example here to JSON.

The input is:

environment "aws" {
}

vault {
  address = "https://vault.service.consul:8200"
  mount_path = "auth/aws"
}

serve "file" {
  path = "/ramdisk/vault-token"
}

However, using the current 0.0.6 version, the JSON output is:

  "environment": [
    {
      "aws": [
        {}
      ]
    }
  ],
  "serve": [
    {
      "file": [
        {
          "path": "/ramdisk/vault-token"
        }
      ]
    }
  ],
  "vault": [
    {
      "address": "https://vault.service.consul:8200",
      "mount_path": "auth/aws"
    }
  ]
}

This is detected as invalid by VSI, even though the HCL file works correctly. It appears that the problem is that there are extra arrays inserted. The working JSON from this HCL seems to be:

{
  "environment": {
    "aws": {}
  },
  "serve": {
    "file": {
      "path": "/ramdisk/vault-token"
    }
  },
  "vault": {
    "address": "https://vault.service.consul:8200",
    "mount_path": "auth/aws"
  }
}

[kvz]

Thanks for reporting!

We're mostly using the other way around ourselves. We switched to the official hcl repo in #2, and I think this behavior should be reported upstream there, since we're merely a simple wrapper around its printer functions (also see https://github.com/kvz/json2hcl/blob/master/main.go#L40) so I don't think this is something we can/should patch at our level. Sorry I can't be of more help!

[Acconut]

Personally, I do not believe that this is a bug or misbehavior of the official hcl package. I spent some time building a function to remove the unnecessary arrays but you will eventually hit some unfortunate situations when creating arrays by repeating the same block, e.g.:

    "basic-dynamodb-table" = {
      "attribute" = {
        "name" = "TopScore"
        "type" = "N"
      }

      "attribute" = {
        "name" = "UserId"
        "type" = "N"
      }
    }

Right now, this will interpreted as

          "basic-dynamodb-table": [
            {
              "attribute": [
                {
                  "name": "TopScore",
                  "type": "N"
                },
                {
                  "name": "UserId",
                  "type": "N"
                }
              ]

However, if we start to remove those arrays, it turns into this:

      "basic-dynamodb-table": {
        "attribute": {
          "name": [
            "TopScore",
            "UserId"
          ],
          "type": [
            "N",
            "N"
          ]
        },

Notice, that basic-dynamodb-table is not an array anymore but an object, which is great. However, the same took place for attribute which is probably not wanted. And the issue is, that the official hcl parser does not supply enough information to us, based on which we could decide whether we should keep it as an array or not. So right now, we are in a bit unfortunate situation where this is not easily achievable.

[jantman]

Hmm... thanks for the info, @Acconut. I'm going to open an issue upstream nonetheless, and see if they can shed some light (even if it just becomes a tally mark for clarifying the config format used by various tools).

[Acconut]

Thank you for bringing this issue up in Hashicorp's repository. I will monitor the discussion and see how we can improve the tool.

[erick-thompson]

I'm running into a similar issue, but with Terraform. Under the root node, there is a variables key with a value that the tool is turning into an array, when it should be a hash/dictionary. Is this the same core issue?

[Acconut]

@erick-thompson Most likely, yes, but it would be nice if you could supply an example for us to check it.

[erick-thompson]

Sure. I'll paste a shortened example here. If you need the full files, I can upload them. The HCL is

variable "container_name" {
	default = "insightprofileservice"
}

variable "container_port" {
	default = "8202"
}

module "insightprofileservice_service" {
  source = "<not shown here>"
  cluster_name = "${var.cluster_name}"
  environment = "${var.environment}"
  desired_count = "${var.desired_count}"
  task_policy_file = "ecs_policy.json"
}

When I run the tool with reverse, I get

{
  "module": [
    {
      "insightprofileservice_service": [
        {
          "cluster_name": "${var.cluster_name}",
          "desired_count": "${var.desired_count}",
          "environment": "${var.environment}",
          "source": "\u003cnot shown here\u003e",
          "task_policy_file": "ecs_policy.json"
        }
      ]
    }
  ],
  "variable": [
    {
      "container_name": [
        {
          "default": "insightprofileservice"
        }
      ]
    },
    {
      "container_port": [
        {
          "default": "8202"
        }
      ]
    }
  ]
}

The problem here is that "variable" is set to an array, but terraform expects it to be a hash/dictionary (this is true for all other resource types as well). So if I then convert the json (module omitted for clarity) to:

{
  "variable": {
      "container_name": {
          "default": "insightprofileservice"
      },
      "container_port": {
          "default": "8202"
      }
    }
}

Then terraform works fine. The problem is that, except for list variables, I don't believe that terraform ever wants arrays, but the output of this tool has arrays everywhere.

BTW, thanks for putting this tool together. HCL is great for humans to work with, but it isn't great to automate.

Thanks,
Erick

[Acconut]

@erick-thompson Unfortunately, this is the same problem as above and therefore we cannot do much about this. :|

[erick-thompson]

That is what I figured. Any point in filing an issue in the terraform repo?

[Acconut]

I don't think so. This is not a problem with Terraform but instead arises because HCL cannot be directly mapped to JSON (see hashicorp/hcl#162 (comment)).

[erick-thompson]

Thanks. I chimed in on the thread just in case.

BTW thanks for the awesome project. It's allowed me to figure out what variables are in play.

[Acconut]

Thank you for your kind words :)

[gellweiler]

This is the workaround I currently use, because I need the extra [] to be gone:

flattenDataStructure(&objs)
data, e = json.Marshal(objs)

// Workaround for: https://github.com/hashicorp/hcl/issues/237, https://github.com/kvz/json2hcl/issues/3
func flattenDataStructure(data *map[string]interface{}) {
	for k, v := range *data {
		vListOfMap, ok := v.([]map[string]interface{})
		if ok {
			if len(vListOfMap) == 1 {
				(*data)[k] = vListOfMap[0]
			}
		}
	}

	for _, v := range *data {
		vMap, ok := v.(map[string]interface{})
		if ok {
			flattenDataStructure(&vMap)
		}
	}
}

It's not nice but at least for me it gets the job done.

[Acconut]

Thanks, @gellweiler. Unfortunately, I do not think this code works for all use cases.

[finferflu]

My solution is not clean, but it's quite effective if your use-case is specific. For example, I'm stripping spurious arrays from a template_file resource:

user_data=`json2hcl -reverse < user_data.tf`
user_data=`echo "$user_data" | jq '.data = .data[]'`
user_data=`echo "$user_data" | jq '.data.template_file = .data.template_file[]'`
user_data=`echo $user_data | jq '.data.template_file.user_data = .data.template_file.user_data[]'`
user_data=`echo $user_data | jq '.data.template_file.user_data.vars = .data.template_file.user_data.vars[]'`

There might be a way to do this in one line, but I feel that the above is more readable and manageable.

I hope it helps anyone who might get stuck here.