Replace internal decoder with hcl-lang

[radeksimko]

End-User Impact

The implementation generally enables a "progressively enhanced" experience. In practice this means:

• Completion is available as soon as the server starts - so the user is never left with zero completion, but the data available is gradually enriched between these stages:
  • (1) universal schema which covers any 0.12+ version (just resource, data, and provider for now, but we can expand later)
  • (2) version-specific core schema (once terraform version ran)
  • (3) provider schemas (once terraform providers schema -json ran)
• Symbol names no longer use any kind of "address" (e.g. data.random_string.refname) but a block is represented the same way it would be in config (e.g. data "random_string" "refname"). This prevents any confusion when displaying "unreferable" blocks or attributes such as terraform and allows us to load symbols early, before any schema (which would define the addressing scheme) is available.

Implementation Notes

• The upstream implementation actually already supports nested symbols (any blocks or attributes arbitrarily nested), but the sourcegraph/go-lsp library doesn't have the right type to expose this, so I left enabling this for another PR
• We could now probably leverage the diagnostics generated from the initial parsing and avoid double parsing it, but I also left that for another PR
• While the diff may seem big, most of it is actually deletions 💥 🎉 with the logic being upstreamed into hcl-lang and terraform-schema. The main "plumbing point" is basically RootModule methods.

Future

This makes it easier to solve the following issues:

• parser: Support 1st class blocks without exposed schema #36
• Add completion and hover support for built-in / reserved blocks/attributes #121
• Add completion support for variable blocks #122
• Add completion support for output blocks. #123
• unknown block type: "terraform" (terraform 0.13 unsupported?) #280

It should be mostly (ignoring RHS completion which isn't supported at all yet) a matter of adding the appropriate schema to hashicorp/terraform-schema and bumping the dependency here.

Additional Links

https://github.com/hashicorp/hcl-lang
https://github.com/hashicorp/terraform-schema

[appilon]

Overall approve, but a pairing session will go a long way for me I think

[radeksimko]

One potential bug was discovered, or at least an untested scenario during our call with @appilon where user requests completion in a file (or performs any other operation requiring schema) within a module that was not initialized. We need to make sure that the relevant schema is matched. AFAIK the current behaviour is just to provide reduced completion candidates (without any provider schema), which is incorrect.

This scenario is tested in some other contexts here:
https://github.com/hashicorp/terraform-ls/tree/master/internal/terraform/rootmodule/testdata/main-module-multienv
-> there we assume that the user can open modules/application/main.tf and have core + provider schema matched from env/{dev,prod,staging} root modules

I will need to look into this.

[radeksimko]

That bug was addressed in the last commit I believe, it wasn't actually that trivial, but I tried my best to avoid changing too much code, given how much is already being changed.

That said I think some of the rootmodule package is due for some refactoring.

Basically the key change this PR is introducing is that what we previously understood as "root module" was a module that is always terraform init'd (with .terraform folder in it), but now because we need to be able to work with uninitialized modules too - to provide symbols and basic early completion - we now see "root module" as any module - initialized or not.

So at the very least the naming should probably change from "root modules" to "modules", but I'd prefer to leave that for another PR.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the context necessary to investigate further.