New HCL and JSON parser in pure Go, printer support

[mitchellh]

This is an API backwards incompatible change. - We've actually never done this with any of our libraries, but given HCL is a fairly HashiCorp-specific library, we're going to do this. Plus, the AST is vastly superior to what we had before, and this is an important change going forward. Sorry, downstreams. Really, really sorry.

---

This brings in @fatih's awesome work rewriting the HCL parser into pure Go in the style of the Go parser package (go in the stdlib). This also adds hcl/printer which lets us output HCL!

I built on top of @fatih's work to make the decoding work with the new AST as well as rewrite the JSON parser into pure Go as well in the same style, so there aren't two differing styles within the repository. The JSON parser still decodes directly into the HCL AST.

This also vastly improves the HCL AST generated by the JSON parser. In every case we have (all our projects), we no longer need to special case certain AST weirdness caused by JSON. Everything just works with the HCL/JSON interoperability. This is a huge win, and thanks a lot to the AST generated by @fatih's work.

[virtuald]

The README hasn't changed much, and there aren't too many adjustments to the test fixtures... so I assume that the actual definition of HCL hasn't changed much. I haven't looked too deeply at the changes yet..

It would be great if you could implement an hcl2json (and back) tool on top of these changes -- which, presumably is easier with the new AST et al. It would also be nice to see more 'round trip' tests if you haven't added them already -- eg, going from JSON to data to HCL to data to JSON and ensuring that the output at the end is the same.

[mitchellh]

@virtuald That's correct, the syntax of HCL is completely the same. The API has changed if you were using this library, though.

[mitchellh]

And regarding hcl2json and back, this isn't truly possible since JSON has multiple representations for HCL. That is: a single HCL config can be converted to JSON in multiple ways and a single JSON representation can be converted to multiple styles of HCL. The semantics would be the same, but the look and feel would definitely be different. As a getting started tool if you planned on moving things to JSON it might not be bad, but it would feel weird.

[virtuald]

It seems like those types of ambiguities should be in the README file.

As the maintainer of the python implementation of HCL, I suppose that's directly in conflict from what I want from hcl then -- namely, I like to think about HCL as a replacement (ish) for JSON that isn't YAML that has really convenient constructs for creating nested structures and supports comments.

This of course is important for me because I just take HCL and turn it into a python dictionary -- which is basically JSON already. Having an unambiguous (canonical?) way to translate HCL -> JSON in a one-to-one way would be convenient for that use case.

[mitchellh]

Unfortunately due to the syntax of HCL it is simply impossible. A basic example:

HCL:

foo { bar = "baz" }

Can be represented in JSON as:

{ "foo": { "bar": "baz" } }
{ "foo": [{ "bar": "baz" }] }

The latter representation is because in HCL that block can be repeated multiple times to make a list of them. The former is allowed in JSON because it feels right as a JSON user, you expect it to work. It is a shortcut for the 2nd form.

I agree we should put this in the README.

The goal of HCL isn't to have a 1:1 mapping between HCL/JSON: it is so that your configuration can be represented semantically equivalent in either format, to allow humans and machines to interoperate with software. Or, to just allow JSON where it is convenient. The idea is that you should be able to parse HCL and also magically get JSON support. It isn't a goal that the JSON has to be a strict representation in one structure, hence multiple are allowed above.

[mitchellh]

Doing it!

[fatih]

👍 Awesome step. I'm at DotGo, didn't had the time to look.