terraform lexer errors on heredoc

[MichaHoffmann]

Terraform lexer has problems when heredoc templates start with "<<" instead of "<<-", to prove i applied the following patch:

index 1d73734d..ab0ba809 100644
--- a/tests/snippets/terraform/test_heredoc.txt
+++ b/tests/snippets/terraform/test_heredoc.txt
@@ -1,6 +1,6 @@
 ---input---
 resource "local_file" "heredoc" {
-  content = <<-DOC
+  content = <<DOC
     heredoc content
     DOC
 }

And the parser showed alot of errors in the tests:

========================================== FAILURES ===========================================
__________________________ tests/snippets/terraform/test_heredoc.txt __________________________
The tokens produced by the "terraform" lexer differ from the expected ones in the file "tests/snippets/terraform/test_heredoc.txt".
Run `pytest tests/snippets --update-goldens` to update it.

    'resource'    Keyword.Reserved
    ' '           Text.Whitespace
    '"local_file"' Name.Class
    ' '           Text.Whitespace
    '"heredoc"'   Name.Variable
    ' '           Text.Whitespace
    '{'           Punctuation
    '\n'          Text.Whitespace

    '  '          Text.Whitespace
    'content'     Name.Attribute
    ' '           Text.Whitespace
    '='           Operator
    ' '           Text.Whitespace
  - '<<-'         Operator
  ?    -
  + '<<'          Operator
  ?     +
    'DOC'         Literal.String.Delimiter
    '\n'          Literal.String.Heredoc

  - '    heredoc content\n' Literal.String.Heredoc
  + '    heredoc content\n' Error

  - '    DOC\n'   Literal.String.Delimiter
  + '    DOC\n'   Error
  +
  + '}\n'         Error
  +
  + '    '        Text.Whitespace
  + 'h'           Error
  + 'e'           Error
  + 'r'           Error
  + 'e'           Error
  + 'd'           Error
  + 'o'           Error
  + 'c'           Error
  + ' '           Text.Whitespace
  + 'c'           Error
  + 'o'           Error
  + 'n'           Error
  + 't'           Error
  + 'e'           Error
  + 'n'           Error
  + 't'           Error
  + '\n'          Text.Whitespace
  +
  + '    '        Text.Whitespace
  + 'D'           Error
  + 'O'           Error
  + 'C'           Error
  + '\n'          Text.Whitespace

    '}'           Punctuation
    '\n'          Text.Whitespace
=================================== short test summary info ===================================
FAILED tests/snippets/terraform/test_heredoc.txt::

According to the spec both are valid introductions to heredoc templates.

[birkenfeld]

From the spec you linked:

The heredoc template expression type is introduced by either << or <<-, followed by an identifier. The template expression ends when the given identifier subsequently appears again on a line of its own.

If a heredoc template is introduced with the <<- symbol, any literal string at the start of each line is analyzed to find the minimum number of leading spaces, and then that number of prefix spaces is removed from all line-leading literal strings. The final closing marker may also have an arbitrary number of spaces preceding it on its line.

This reads to me as if your changed test is not valid, since the final delimiter DOC appears not at the start of a line, which is not allowed in heredocs starting with <<.

[mhoffm-aiven]

The terraform tool accepts that style without problems though:

[michael.hoffmann@fedora ~]$ terraform fmt test.tf 
test.tf
[michael.hoffmann@fedora ~]$ cat test.tf 
resource "foo" "bar" {
  x = <<EOF
    testing
  EOF
}

[mhoffm-aiven]

Ah testing with DOC without leading whitespace works though! I get what you say

[birkenfeld]

Hm, but if terraform itself is happy with the leading whitespace we shouldn't highlight it as error.

[birkenfeld]

Fixed!

[mhoffm-aiven]

Thanks @birkenfeld !