Can't decode into map[string]interface{}

[brikis98]

I've been experimenting with the hcl2 code for use with Terragrunt, and so far, things have been going well. The parser is intuitive, the error messages are helpful, and the first-class support for expressions, functions, variables, etc is great! A big improvement over the original hcl code 👍

The one issue I'm hitting is when trying to use Decode function with the following struct:

type Example struct {
  Foo string `hcl:"foo,attr"`
  Bar map[string]interface{} `hcl:"bar,attr"`
}

I try to decode it as follows:

parser := hclparse.NewParser()
file, diagnostics := parser.ParseHCL([]byte(myConfig), "test")

if diagnostics.HasErrors() {
  panic(diagnostics)
}

var example Example
diags := gohcl.DecodeBody(file.Body, nil, &example)
if diags.HasErrors() {
  panic(diags)
}

I get the following error:

panic: unsuitable DecodeExpression target: no cty.Type for interface {} [recovered]

The Bar field of the Example struct is intentionally a map[string]interface{} because this is data that is user provided, and I have no way of knowing the types of the values; they could be strings, ints, bools, slices, maps, etc. I could parse this as a block and use the remain metadata to get back hcl.Body, but that requires changing the Example struct in a way that (a) is backwards incompatible, (b) leaks the underlying implementation details, and (c) is hard to use.

Is there any way to decode into a map[string]interface{} directly?

[apparentlymart]

Hi @brikis98,

At the moment there is no default decoding into interface{} like this, because the decoder needs some type information to hint how to convert from the HCL type system into the Go type system. In this particular case, where you're decoding an attribute whose type isn't known, the current best way to do that is to decode into cty.Value and then work with cty (the underlying type system) directly to decide which Go types to select.

Decoding into interface{} is, of course, theoretically possible here, but requires making some decisions that may affect how well this addresses your last point "(c) is hard to use". There are a few minor questions here which I think have reasonably straightforward answers, but a big one is what to do with numbers:

The most direct conversion of an HCL number is a *math/big.Float, but I expect the reason for decoding into interface{} is to use "normal-ish" Go types, and so a float64 would be more expected. That may requite the value to be truncated in some way, though. I think the most robust thing would be to select float64 by default and then produce an error if the given value is out of range for it, the same as would happen if the target type were float64 explicitly.

So far in situations where we've found decoding implementation details leaking into exported structs we've avoided it by using temporary structs just for decoding, and then populating the exported struct before returning. This also gives an opportunity to do any additional validations that HCL's schema cannot represent:

func decodeExampleBlock(block *hcl.Block) (*Example, hcl.Diagnostics) {
    type DecodeExample struct {
        Foo string `hcl:"foo,attr"`
        Bar cty.Value `hcl:"bar,attr"`
    }

    var raw DecodeExample
    diags := gohcl.DecodeBody(block.Body, nil, &raw)
    if diags.HasErrors() {
        return diags
    }

    ret := &Example{
        Foo: raw.Foo,
        Bar: map[string]interface{},
    }

    // then use your own logic to map cty into the map[string]interface{}
    // using whatever mapping rules make sense for your application.

    return ret, diags
}

However, as long as we can define it in a way that doesn't silently lose information I think having a default decoding into interface{} would be a big help for the common case. Mimicking what would happen if you serialized the cty.Value as JSON and then decoded it with encoding/json seems like a good place to start, as long as we can get an error when a number would be truncated.

I don't expect I'll have time to work on this in the near future since there's lots of Terraform-side work keeping me busy right now. If a short-term solution is needed, I guess I'd suggest using the local decoding struct pattern I described above and then doing the final step by actually serializing the cty.Value as JSON and then decoding it with encoding/json, which would then get a similar result to what I described above (albeit with slightly more overhead).

[brikis98]

Understood, thanks for the detailed answer!

[hasheddan]

@apparentlymart Hi! I am a HUG organizer and contributor to a number of HashiCorp products. I am very interested in programming languages and would love to start contributing to hcl2! Let me know if this issue or any others would be a good place to start helping out :)

[bluebrown]

how can terraform decode the terraform files? The structure is not known upfront so they cant create structs.

I also want to parse a user provided hcl file where I dont know whats written inside, I am not sure how I can do this. For example using map[string]cty.Value works only for a flat structure. As soon as I have blocks, it errors,

[apparentlymart]

Hi @bluebrown,

Terraform's configuration decoder is subject to the same requirement of providing a schema in order to decode a HCL body. There is nothing in the Terraform language that cannot be decoded using the HCL API, although Terraform does make use of some more advanced HCL API features that are not reachable using gohcl alone and so fully interpreting a .tf file (and the overall module it contributes to) will require using HCL's lower-level main API in some places, rather than the gohcl wrapper around it.

There is no way to decode blocks without providing a schema for those blocks, because it is the schema which allows resolving the inherent ambiguity in the JSON serialization of the HCL infoset, where HCL arguments and blocks are both represented as properties of a JSON object. When you are designing an HCL-based language you must decide the block structure as part of the language design and encode it into your program.

[bluebrown]

Thanks ,@apparentlymart, I as a user can decide in which TF file I put which blocks, or not? I can make one full of variables and another one containing a server resource or something like that. At that point, I may have definitions but I don't know which one is in which file, so how could I decode it?

[apparentlymart]

If your goal is to parse a Terraform configuration then that is more of a Terraform question than it is an HCL question. HCL is just a collection of syntax primitives for building languages with, and so there's a lot more to the Terraform language than just HCL alone: you'd need to reimplement its schema and the schema of any providers that would normally be involved in processing that configuration.

Terraform-specific questions are not really on topic for this repository, which is about the generic HCL library rather than any particular application using it, but if you ask in Terraform's community forum and say some more about what your goals are then someone there (possibly me!) may be able to offer some concrete suggestions on how to achieve your goal.

[joeatwork]

Just a +1 - I think may break round-tripping certain Nomad jobspecs (since Nomad Job Tasks contain a map[string]interface{} Configuration field.) I'm seeing a failure trying to encode Jobspecs that typecheck ok.

[b0bu]

I have almost the same question as @bluebrown and haven't been able to find (even remotely) an answer pointing me in the right direction. My usecase is similar I need to be able to parse hcl that is essentially freeform / schemaless into something that I can work with get values from within go. Now, within that massive space of possibilties my usecase is a narrow scope. A good example of this is locals in terraform.

locals {
 some_list = []
 some_map = {k="v"}
 some_nested = [{}]
 some_bool = true
 some_string = "yolo"
}

This is the type of struct I'd just like to "glob" in.

There are two specific problems, using terraform as an example, assuming it had a .hcl extension, I need to first find the locals which could be in any .tf file and I need to read them in in some way that allows me access the values similar to locals["some_list"]. So if I had multiple files, I'd read them all for such a locals block and essentially have a slice of locals. Even using basic code and basic examples of this larger problem I cannot figure out how the hcl packages could dynamically read in a block like the locals block above where the contents of locals is not known.

[apparentlymart]

HCL is not designed for "schemaless" languages. The evaluation model relies on schema to resolve ambiguities such as those in the JSON syntax.

locals in the Terraform language does have a "schema", but it's a flexible one: it can contain arbitrary arguments and cannot contain any nested blocks. That uses the Dynamic Attributes Processing mode, which in the Go API is represented by the Body.JustAttributes method.

This sort of unusual decoding strategy is harder to express with the gohcl abstraction because gohcl is trying to mediate between HCL's schema concepts and Go's type system concepts. However, there is a specific way to define a struct that gohcl can map to the dynamic attributes processing mode:

type Example struct {
    Attributes hcl.Attributes `hcl:",remain"`
}

If you decode into a valid of the above type, gohcl will internally use Body.JustAttributes and write the result into the Attributes field. You can then do dynamic decoding of the attributes as a separate step.

The main thing to recognize here is that treating attributes in a body as dynamic is itself still a static schema-like decision. HCL does not provide any way to just "decode the whole thing and see what happens" because the HCL infoset is a superset of the Go type system and so gohcl needs guidance (based on type information and struct tags) to understand what interpretation of the user input you need. However, you can designate that certain parts of your language have more or less rigid structure based on the needs of your HCL-based language.

I also want to restate what I said above: Terraform does not use gohcl and so the Terraform language includes elements that gohcl cannot represent. If you want to implement Terraform-language-like features then you may need to use the low-level HCL API directly instead of gohcl, because that's what Terraform itself does. It might be possible to adapt what Terraform is doing to the gohcl model, but not necessarily since gohcl only supports a subset of the possibilities of HCL aimed at applications with much simpler needs than Terraform's.

[b0bu]

@apparentlymart massive thanks, not only did you help to solve my problem you changed my thinking of the problem almost entirely.