HCL v0.12 brings incompatible updates.

[NadOby]

Steps to reproduce:
upgrade terraform to latest version.

cd /tmp/

git clone https://github.com/terraform-providers/terraform-provider-aws.git
Cloning into 'terraform-provider-aws'...
remote: Enumerating objects: 17, done.
remote: Counting objects: 100% (17/17), done.
remote: Compressing objects: 100% (11/11), done.
remote: Total 97677 (delta 7), reused 11 (delta 6), pack-reused 97660
Receiving objects: 100% (97677/97677), 83.75 MiB | 8.58 MiB/s, done.
Resolving deltas: 100% (62884/62884), done.
Checking out files: 100% (6246/6246), done.

cd terraform-provider-aws/examples/two-tier

terraform init

Initializing provider plugins...
- Checking for available provider plugins on https://releases.hashicorp.com...
- Downloading plugin for provider "aws" (2.14.0)...

The following providers do not have any version constraints in configuration,
so the latest version was installed.

To prevent automatic upgrades to new major versions that may contain breaking
changes, it is recommended to add version = "..." constraints to the
corresponding provider blocks in configuration, with the constraint strings
suggested below.

* provider.aws: version = "~> 2.14"

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

terraform 0.12upgrade

This command will rewrite the configuration files in the given directory so
that they use the new syntax features from Terraform v0.12, and will identify
any constructs that may need to be adjusted for correct operation with
Terraform v0.12.

We recommend using this command in a clean version control work tree, so that
you can easily see the proposed changes as a diff against the latest commit.
If you have uncommited changes already present, we recommend aborting this
command and dealing with them before running this command again.

Would you like to upgrade the module in the current directory?
  Only 'yes' will be accepted to confirm.

  Enter a value: yes

-----------------------------------------------------------------------------

Upgrade complete!

The configuration files were upgraded successfully. Use your version control
system to review the proposed changes, make any necessary adjustments, and
then commit.

from glob import iglob
import hcl

for filename in iglob('**.tf'):
    with open(filename, 'r') as f:
        print(filename)
        hcl.loads(f.read())

python3  ~/tmp/testhcl.py
main.tf
Traceback (most recent call last):
  File "/home/evgeniy/tmp/testhcl.py", line 7, in <module>
    hcl.loads(f.read())
  File "/home/evgeniy/.pyenv/versions/3.7.3/lib/python3.7/site-packages/hcl/api.py", line 66, in loads
    return HclParser().parse(s)
  File "/home/evgeniy/.pyenv/versions/3.7.3/lib/python3.7/site-packages/hcl/parser.py", line 326, in parse
    return self.yacc.parse(s, lexer=Lexer())
  File "/home/evgeniy/.pyenv/versions/3.7.3/lib/python3.7/site-packages/ply/yacc.py", line 333, in parse
    return self.parseopt_notrack(input, lexer, debug, tracking, tokenfunc)
  File "/home/evgeniy/.pyenv/versions/3.7.3/lib/python3.7/site-packages/ply/yacc.py", line 1201, in parseopt_notrack
    tok = call_errorfunc(self.errorfunc, errtoken, self)
  File "/home/evgeniy/.pyenv/versions/3.7.3/lib/python3.7/site-packages/ply/yacc.py", line 192, in call_errorfunc
    r = errorfunc(token)
  File "/home/evgeniy/.pyenv/versions/3.7.3/lib/python3.7/site-packages/hcl/parser.py", line 318, in p_error
    raise ValueError(msg)
ValueError: Line 3, column 70: unexpected IDENTIFIER

[sbalbach]

From Hashicorp:

At some point we'll likely have to roll the HCL2 codebase into that one, having two versions of HCL is something we'll likely maintain for a little while as customers migrate across from 0.11 to 0.12. However HCL2 will be our definitive version of HCL and will supersede the original repo at some point in the future. We just don't have a hard date for that as we've never EOL'd a version of Terraform so we don't have concrete plans for when we'll do that.

[NadOby]

I don't see how will be possible to update HCL=>HCL2 grammar easily.
There is no formal specification of the grammar that can be used outside of hashicorp
grammar is spread in https://github.com/hashicorp/hcl2/tree/master/hcl/hclsyntax

For now I workaround this problem by not using terraform 0.12upgrade old syntax works in terraform v0.12, but this took some time.

[pop]

As a user of pyhcl I would expect pyhcl's API to be something like hcl.load(input_tf_file, hcl_version=1) for loading pre-0.12 Terraform and hcl.load(other_tf_file, hcl_version=2) for loading 0.12-compatible files. (with version defaulting to 2 going forward).

Of course somebody then needs to do the work of figuring out how to parse HCL2 files. Easier said than done. 😓

[sbalbach]

If you don't want to wait for the pull request, try out the code in my fork:
https://github.com/StateFarmIns/pyhcl

[jcaxmacher]

@sbalbach, thanks for working updates that support HCL2 and Terraform v0.12. Do you think you'll add support for "for" and "for-each"?

[sbalbach]

@jcaxmacher: what kind of support are you looking for? i.e. are you getting an error?

[cr-liadsh]

@sbalbach when will this fix be merged in to pyhcl?

[virtuald]

@cr-liadsh someone needs to fix the unit tests as mentioned in the PR. I'm not going to merge it if it's obviously incorrect.

[sbalbach]

It isn't just the unit tests. To fix the unit tests, the code must be changed. I'm in a catch 22. If I fix it for the unit tests, it breaks it elsewhere. So, I don't plan to make further changes.

Someone else is free to give it a shot, though.