Syntax Rewrite for v0.12.x #39

patrickrgaffney · 2019-09-05T04:56:04Z

HCL2

Since v0.12, the underlying HCL syntax has undergone a lot of changes. This PR reworks the sublime-syntax to better support the new language features.

Rationale

I've only been updating the .sublime-syntax file — as I only use Sublime. A quick Google search shows a few different converter tools to turn it back into a PList-style .tmLanguage file if that's still desired.

While reworking the syntax I've been trying to update the scopes to match the official conventions better (if possible). All of the changes made so far can be seen in the changes.md file. I can remove that file when this is complete — I've been trying to keep a thorough list there.

The major goal of this PR was to update the syntax for the new HCL2 syntax. In addition to Terraform, other Hashicorp projects are using HCL for configuation now: Vault, Nomad, (and soon 🤞) Packer. The changes in this PR should better support non-Terraform-specific HCL files going forward.

I added a lot of documentation to the sublime-syntax and re-ordered it to better follow the HCL spec. I also added fairly extensive unit testing. 🎉

Merging this PR would close the following open issues:

Closes Add support for tfvars+tfstate #1
Closes new terraform block isn't highlighted #31
Closes Unit testing #34

Progress / TODO

- CHANGED: scope `comment.line.number-sign.terraform` to `comment.line.terraform` - Better reflects scoping conventions - FIXED: previously incorrect used `number-sign` even for `//` inline comments - ADDED: syntax tests

- ADDED: syntax tests

- FIXED: removed invalid constants `yes`, `no`, `on`, `off`. - ADDED: `null` value from v0.12 - ADDED: syntax tests

Numbers - Separated into `Integers` and `Floats` - REMOVED: unit of measurement suffixes - REMOVED: hexadecimal literals Integers - CHANGED: scope `constant.numeric.terraform` is now `constant.numeric.integer.terraform` - Better reflects scoping conventions

- CHANGED: scope `constant.numeric.terraform` is now `constant.numeric.float.terraform` - Better reflects scoping conventions - ADDED: exponents in float literals - ADDED: syntax tests

More closely matches what is in the spec.

- REMOVED: the extra `string.terraform` scope - ADDED: character escapes - newline - carriage return - backslash - quote - unicode points

- CHANGED: scope `entity.tag.embedded.[begin|end]` is now `punctuation.section.interpolation.[begin|end]` - Better reflects scoping conventions - FIXED: use `clear_scopes` to remove the `string.*` scope inside interpolation - ADDED: the trim-left and trim-right operators inside interpolation

- ADDED: template if/for directives

- ADDED: match literals inside string interpolation

- ADDED: comparison operators `==`, `!=`, `<`, `>`, `<=`, `>=` - ADDED: arithmetic operators `+`, `-`, `*`, `/`, `%` - ADDED: logic operators `&&`, `||`, `!` - ADDED: conditional operator `predicate ? expression : expression`

- ADDED: first-class array syntax - ADDED: matches index operations

https://github.com/hashicorp/hcl2/blob/master/hcl/hclsyntax/spec.md#splat-operators

- ADDED: `variable.other.member` scope on attribute access - ADDED: `punctuation.accessor.dot` scope on dot-operator - ADDED: attribute-only splat operator - ADDED: matches attribute-access inside string interpolation - ADDED: syntax-tests

- ADDED: allow `-` after first alpha character in identifiers - CHANGED: `variable.assignment` to `variable.other.readwrite` - Better reflects scoping conventions - ADDED: paren-matching for computed attr names - CHANGED `keyword.operator` to `keyword.operator.assignment` for `=` - Better reflects scoping conventions

- ADDED: match outside of string interpolation - CHANGED: `keyword.other.function.inline` to `variable.function` - Better reflects scoping conventions - ADDED: match operators, literals, comments, attributes, etc inside parameter list - ADDED: match variable parameters as `variable.parameter` - ADDED: match commas in parameter list - ADDED: syntax tests

patrickrgaffney · 2019-10-15T02:57:33Z

Okay, just pushed up some changes:

Changed attribute-access by . scope to match JS.
From what I can tell, it appears we are matching string interpolation punctuation the same as JS — via punctuation.section.interpolation.[begin|end] plus the meta.interpolation scope. This could be different depending on which version of Sublime you're on. I'm running Build 3210 from the Dev channel.
I'm cool with adding some illegal scopes to the definitions. Maybe they could come in a separate PR? I can create an issue to track it if that works for you.
I agree with you on the parameter highlighting. Looking through other standard language packages it looks like they only match parameter names on definition and for named parameters. Since HCL currently doesn't allow you provide named parameters or define your own functions, I removed the variable.parameter matching.

Object keys seem to be a bit of a mixed bag, here's what some of the standard languages do:

JS: meta.object-literal.key.js
Python: for dictionaries, keys are quoted, and thus are highlighted as strings with meta.mapping.key.python.
Go: for maps, keys can be any type that is comparable — which is quite a few of them. They're highlighted as a literal of that type, without a meta_scope.

We're currently using entity.name.tag, which is part of the "minimal color scheme coverage" in the Sublime docs. I left it as that for now as I didn't have a better idea, but I'm open to other suggestions.

I went through and added additional tests to cover the issues in #28, #26, #21, and #18. The tests we're already covering #23.

patrickrgaffney · 2019-10-22T04:42:39Z

I did some more digging into the object key syntax scopes today. It seems like most of the official sublime-syntaxes use a meta scope of meta.mapping.key on the entire key along with string.unquoted or string.quoted. Seems like this will be the standard way forward per this RFC.

Found in the following syntaxes: JS, Git Config, Lua, Python, Erlang, JSON.

I'm good with updating this package to use that scope, sound good @alexlouden?

alexlouden · 2019-10-22T05:11:00Z

I'm good with updating this package to use that scope, sound good @alexlouden?

Perfect, thank you!

From what I can tell, it appears we are matching string interpolation punctuation the same as JS — via punctuation.section.interpolation.[begin|end] plus the meta.interpolation scope.

Ah sorry, my previous comment was using babel-sublime (imo big improvement over the built-in JS one), which uses keyword:

The ${ is keyword.other.substitution.begin and the period is keyword.operator.accessor

if I just use the JS plugin then it doesn't highlight the ${ or the period at all (just white). Are you okay with using these?

alexlouden · 2019-10-22T05:14:30Z

Also again - thank you so much for doing all this!

alexlouden · 2019-10-22T05:24:15Z

I've just set up CircleCI to unit test the repo, failing at the moment since there aren't any tests but maybe will test this PR next commit?

patrickrgaffney · 2019-10-22T05:32:07Z

Interesting. I used to use babel-sublime pretty much exclusively, but recently the builtin-in JS package has gotten a lot better — any version after v3.1 has most of the newer TC-39 features. 😁

It does look like the built-in Monokai color scheme doesn't handle the recommended scopes for interpolation very well — which is odd since the other built-ins (Breakers, Celeste, Mariana, Sixteen) all seem to be catching those scopes.

Although I think this is a relatively "new" standard — looks like the RFC for interpolation scopes was accepted back in February. I can update them to use the keyword. prefix.

Per the latest scope naming RFC and the standard packages, we use meta.mapping.key as the meta_scope, and either string.quoted or string.unquoted around the key name. Computed keys only get the meta_scope and are matched for expressions.

patrickrgaffney · 2019-10-22T05:56:50Z

Regarding CI — I have access to the GitHub actions beta. I could take a stab at adding one of those in a later PR if you want. I played around with it on another repo I maintain and it's pretty cool, plus you don't have to leave the GH UI.

alexlouden · 2019-10-22T06:28:00Z

the builtin-in JS package has gotten a lot better

Ah, that might be it! I switched over to primarily VS Code about two years ago :)

Regarding CI — I have access to the GitHub actions beta. I could take a stab at adding one of those in a later PR if you want. I played around with it on another repo I maintain and it's pretty cool, plus you don't have to leave the GH UI.

I had a quick poke around at creating a GH Action for it, but thought since it requires downloading and installing Sublime Text I thought maybe CircleCI's caching would help speed up tests a bit? By all means if you'd like to convert it to then go for it! The linting you set up in the other repo looks really neat

It was updated to use "punctuation.accessor", but the support for this new scope appears to be hit or miss. It was changed to "keyword.operator.accessor"

The scope was updated to be "punctuation.section.interpolation", but that is a newer scope with limited color scheme support at this time, so it was changed to be "keyword.other.interpolation".

patrickrgaffney · 2019-10-29T04:04:01Z

Updated the scopes for the attribute-access and string interpolation operators per our previous conversation. Sorry this took so long, been a long week.

patrickrgaffney · 2019-11-24T20:20:50Z

@alexlouden friendly ping 🙂 Are you okay with merging this?

alexlouden · 2019-11-25T01:32:48Z

Oh I'm sorry I missed this, if you're happy with it then let's merge it in!

patrickrgaffney · 2019-11-26T21:59:37Z

Yeah, I'm good with it. I'll let you merge and cut the new release. 😎

We should probably change the ## Unreleased in the CHANGELOG.md to the new version number (## vX.X.X) before the new release is made.

jeffbyrnes · 2019-11-27T17:39:04Z

For my own projects, I typically finalize the CHANGELOG & update the version number as a single commit, and tag that commit to use for the release.

patrickrgaffney · 2019-11-27T19:33:27Z

Yeah, I do something similar as well for my projects. Just didn't want to step on Alex's toes. 😁

patrickrgaffney · 2019-12-18T03:47:20Z

@alexlouden Did you want me to merge this?

grigorov · 2020-01-09T09:20:29Z

Hey, 2020 year... when merge?

TaysirTayyab · 2020-01-16T22:41:45Z

@alexlouden ping. Would love to get this merged in.

plasticine · 2020-01-20T05:15:48Z

This is amazing — thanks so much for all your work on this @patrickrgaffney & @alexlouden 👏

alexlouden · 2020-01-20T05:31:56Z

Sorry everyone, I've just merged it in and made a new release (hope I did it right!).

Thanks again for this @patrickrgaffney - please consider yourself a co-owner of the repo and do whatever you think is best with it going forward 😃

patrickrgaffney added 30 commits August 26, 2019 23:26

Add basic header and file_extensions comments

05cc141

Rework comment.line.terraform

5b17fe5

- CHANGED: scope `comment.line.number-sign.terraform` to `comment.line.terraform` - Better reflects scoping conventions - FIXED: previously incorrect used `number-sign` even for `//` inline comments - ADDED: syntax tests

Remove unused uuid key from Comments.tmPreferences

9d821fe

Rework block comments

3b5f69f

- ADDED: syntax tests

Rework language constants

b5b1945

- FIXED: removed invalid constants `yes`, `no`, `on`, `off`. - ADDED: `null` value from v0.12 - ADDED: syntax tests

Remove numeric literals (floats)

227f02f

- CHANGED: scope `constant.numeric.terraform` is now `constant.numeric.float.terraform` - Better reflects scoping conventions - ADDED: exponents in float literals - ADDED: syntax tests

Move integers to the 'match-numbers' context

c305889

Include all contexts within 'main' for better ordering

c559a67

Add variables for 'exponent' and 'identifier'

f74bcb4

Rename "numeric-constants" to "numeric-literals"

9bcbed6

More closely matches what is in the spec.

Add literal-values context for stack push/poping

2cd60d0

Add hyphens to identifiers

2c798dc

Rework string literals

007f4a5

- REMOVED: the extra `string.terraform` scope - ADDED: character escapes - newline - carriage return - backslash - quote - unicode points

Clean up YAML in the syntax file

3fd3ca7

Add template if/for directives

86d636d

- ADDED: template if/for directives

Add support for %{...} string interpolation

9430853

Allow literals inside string interpolation

dca1633

- ADDED: match literals inside string interpolation

Add tests for string literals in template if directives

de7367a

Add operators

ca2831f

- ADDED: comparison operators `==`, `!=`, `<`, `>`, `<=`, `>=` - ADDED: arithmetic operators `+`, `-`, `*`, `/`, `%` - ADDED: logic operators `&&`, `||`, `!` - ADDED: conditional operator `predicate ? expression : expression`

Add array syntax

44ad820

- ADDED: first-class array syntax - ADDED: matches index operations

Add full-splat operator

a67c4d3

https://github.com/hashicorp/hcl2/blob/master/hcl/hclsyntax/spec.md#splat-operators

Add attribute-access

f10a387

- ADDED: `variable.other.member` scope on attribute access - ADDED: `punctuation.accessor.dot` scope on dot-operator - ADDED: attribute-only splat operator - ADDED: matches attribute-access inside string interpolation - ADDED: syntax-tests

Match on commas in array literals

65fb69e

Match operators and attribute-access in string interpolation

252fc9e

Add support for nested array literals

aba8826

Rework built-in functions support

97396eb

patrickrgaffney added 5 commits October 14, 2019 21:33

Remove variable.parameter.terraform matching

ff9ac50

Add extra test for alexlouden#28 - regexes in strings

dae54a6

Add tests for alexlouden#26 - nested function calls in template

0131f3b

Add tests for alexlouden#21 - braces in strings

305e582

Add tests for alexlouden#18 - nested string interpolation

ed107fa

Update the object key-mapping scopes

f1fbe4e

Per the latest scope naming RFC and the standard packages, we use meta.mapping.key as the meta_scope, and either string.quoted or string.unquoted around the key name. Computed keys only get the meta_scope and are matched for expressions.

patrickrgaffney added 2 commits October 28, 2019 22:55

Update scope for attribute-access operator

e895f03

It was updated to use "punctuation.accessor", but the support for this new scope appears to be hit or miss. It was changed to "keyword.operator.accessor"

Update scope for string interpolation punctuation

a680585

The scope was updated to be "punctuation.section.interpolation", but that is a newer scope with limited color scheme support at this time, so it was changed to be "keyword.other.interpolation".

alexlouden merged commit 34d6a1d into alexlouden:master Jan 20, 2020

patrickrgaffney mentioned this pull request Jan 20, 2020

Highlight doesn't support 2.31 AWS provider syntax on TF 12.16 #40

Closed

patrickrgaffney deleted the rewrite branch January 20, 2020 06:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Syntax Rewrite for v0.12.x #39

Syntax Rewrite for v0.12.x #39

patrickrgaffney commented Sep 5, 2019 •

edited

Loading

patrickrgaffney commented Oct 15, 2019 •

edited

Loading

patrickrgaffney commented Oct 22, 2019

alexlouden commented Oct 22, 2019

alexlouden commented Oct 22, 2019

alexlouden commented Oct 22, 2019

patrickrgaffney commented Oct 22, 2019

patrickrgaffney commented Oct 22, 2019

alexlouden commented Oct 22, 2019

patrickrgaffney commented Oct 29, 2019

patrickrgaffney commented Nov 24, 2019

alexlouden commented Nov 25, 2019

patrickrgaffney commented Nov 26, 2019

jeffbyrnes commented Nov 27, 2019

patrickrgaffney commented Nov 27, 2019

patrickrgaffney commented Dec 18, 2019

grigorov commented Jan 9, 2020

TaysirTayyab commented Jan 16, 2020

plasticine commented Jan 20, 2020

alexlouden commented Jan 20, 2020

Syntax Rewrite for v0.12.x #39

Syntax Rewrite for v0.12.x #39

Conversation

patrickrgaffney commented Sep 5, 2019 • edited Loading

HCL2

Rationale

Progress / TODO

patrickrgaffney commented Oct 15, 2019 • edited Loading

patrickrgaffney commented Oct 22, 2019

alexlouden commented Oct 22, 2019

alexlouden commented Oct 22, 2019

alexlouden commented Oct 22, 2019

patrickrgaffney commented Oct 22, 2019

patrickrgaffney commented Oct 22, 2019

alexlouden commented Oct 22, 2019

patrickrgaffney commented Oct 29, 2019

patrickrgaffney commented Nov 24, 2019

alexlouden commented Nov 25, 2019

patrickrgaffney commented Nov 26, 2019

jeffbyrnes commented Nov 27, 2019

patrickrgaffney commented Nov 27, 2019

patrickrgaffney commented Dec 18, 2019

grigorov commented Jan 9, 2020

TaysirTayyab commented Jan 16, 2020

plasticine commented Jan 20, 2020

alexlouden commented Jan 20, 2020

patrickrgaffney commented Sep 5, 2019 •

edited

Loading

patrickrgaffney commented Oct 15, 2019 •

edited

Loading