while rule from parent grammer is applying during a capture tokenization in an embedded grammar

[msftrncs]

I am troubleshooting an expiremental PowerShell grammer while in a fenced code block in a MarkDown document, and I noticed that While rules are still being checked while tokenizing a capture.

Actually, they appear to be checked after the capture, but before it is tokenized. This seems to be causing a problem with the tokenizing of the capture.

@@scanNext 16 {A:16}: |$grammar_json -Indent "`t" -StateEncodingAs 'UTF-8' |\n|
  scanning for
   - 371: \$(?:(?>[\w][\w?]*:|:)?(?:[\w](?:[\w?]|:(?!:))*|:(?!:))|[$?^:])(?=\.(?!\.)|::|\[)
   - 408: \$(?:(?>[\w][\w?]*:|:)?(?:[\w](?:[\w?]|:(?!:))*|:(?!:))|[$?^:])
   - 412: (?=\$\{)
   - -1: (?!.\G)
matched: 16 / 29
  pushing BeginEndRule#408 @ powershell.tmLanguage.json:1862 - \$(?:(?>[\w][\w?]*:|:)?(?:[\w](?:[\w?]|:(?!:))*|:(?!:))|[$?^:])
  scanning for while rule
   - -2: (^|\￿)(?!\s*([`~]{3,})\s*$)

@@scanNext 16 {A:-1}: |$grammar_json|
  scanning for
   - -1: (^|\￿)(|\s{0,3})(```)\s*$
   - 343: (^|\￿)(\s*)(.*)
  no more matches.
  token: |$grammar_json|
      * text.html.markdown
      * markup.fenced_code.block.markdown

The $grammar_json failed to scope as a powershell variable, and it appears that it is because the while rule was checked, but didn't match inside the capture tokenizing (^ didn't match, not at beginning of line). Technically I don't think the while rule should apply if linepos !== 0. I will provide a PR shortly with a possible solution. Note: The debug above could probably use some improvement to clarify that it is scanning a capture, and when scanning the capture has completed.

[msftrncs]

some reference material

    ```PowerShell
    $grammar_json = Get-Content "powershell.tmlanguage.json" | ConvertFrom-Json
    ConvertTo-Plist $grammar_json -Indent "`t" -StateEncodingAs 'UTF-8' |
        Set-Content 'out\PowerShellSyntax.tmLanguage' -Encoding 'UTF8'
    ```

    ```PowerShell
    [xml]$grammar_xml = Get-Content "powershell.tmlanguage.json" | ConvertFrom-Json |
        ConvertTo-Plist -Indent "`t" -StateEncodingAs $null 
    ```

The $grammar_json on the second line of code in the first section and in both places in the second section, and the 'xml' in the second section rely on tokenizing a capture, and fails in both places, but the '$grammar_json' on the first line works because its at the start of the line. The $null on the last line of the last section is the same way.

[alexdima]

Closed via #88