YAML file does not colorise correctly

[dpoetzschke]

VSCode Version: 1.39.2
OS Version: macOS Catalina 10.15

Steps to Reproduce:

1. open the example files with YAML Syntax

Does this issue occur when all extensions are disabled?: Yes

One or more problems occur with the YAML syntax highlighting when using Blocks in yaml files.
There was already an issue but that has been closed (microsoft/vscode#63382) in favor of a TextMateBundle issue (textmate/yaml.tmbundle#30)
Since there are no problems within TextMate and the YAML Syntax i am creating a new issue here.

Example one:
Block is not initialised correctly as following pipes | within are then again recognized as blocks which leads to illegal characters error.

        - run:
          name: Delete old images
          command: |
            gcloud container images list-tags << parameters.image >> \
              --limit=999999 \
              --sort-by=TIMESTAMP \
              --format="get(digest)" \
              --filter="NOT(tags:(release,qa)) AND timestamp.datetime < -<< parameters.keep >>" \
              | xargs -I% -n1 gcloud container images delete -q --force-delete-tags "<< parameters.image >>@%"

Example two:
Same as in the issue before, double quotes blocks within Blocks are not closed properly.

      - run:
        name: 'Build and push docker image'
        command: |
          docker run \
                --rm \
                -w /app \
                -v $(pwd):/app \
                -v ${HOME}:/config \
                -v /var/run/docker.sock:/var/run/docker.sock \
                google/cloud-sdk \
                sh -c "\
                  command1 && \
                  command2 && \
                  command3
                "
      - run:
        name: 'test'
        command: |
          ....
          ....

These issues only exist with VisualStudio Code. TextMate does render these correctly.

Visual Studio Code

TextMate

[alexr00]

@alexandrudima It looks like the same textmate grammar is working in TextMate, but not in VS Code.
Since I cannot install TextMate to check (no Mac), would you be able to try it?

[alexdima]

This took me quite a while to track down. It turns out it is caused by the following rule in the YAML grammar:

<key>block-scalar</key>
<dict>
	<key>begin</key>
	<string>(?:(\|)|(&gt;))([1-9])?([-+])?(.*\n?)</string>
	...
	<key>end</key>
	<string>^(?=\S)|(?!\G)</string>
	<key>patterns</key>
	<array>
		<dict>
			<key>begin</key>
			<string>^([ ]+)(?! )</string>
			<key>end</key>
			<string>^(?!\1|\s*$)</string>
			<key>name</key>
			<string>string.unquoted.block.yaml</string>
		</dict>
	</array>
</dict>

And the issue appears to be around this code:

  command: |
    docker run \

What would happen is that the line with a | would enter the block-scalar BeginEndRule, and it would consume even the \n on the line command: |\n. Technically, that means that the next line has an anchorPosition of 0 because the previous rule has consumed the \n. So \G should match on the next line, which would prevent the end pattern ^(?=\S)|(?!\G) from matching and would allow the inner BeginEndRule of string.unquoted.block.yaml to match with ^([ ]+)(?! ).

Here's a debug trace of line 4 matching the outer end pattern instead of the inner begin pattern:

===========================================
TOKENIZING LINE 4: |    docker run \|

@@scanNext 0: |    docker run \\n|
  scanning for (linePos: 0, anchorPosition: -1)
   - -1: ^(?=\S)|(?!\�)
   - 41: ^([ ]+)(?! )
matched rule id: -1 with length 0 : 0 / 0

The rule id -1 should not match because it is at the previous anchor position.