Syntax highlighting broken for more complicated string types (string with escaping)

[goto40]

Langium version: 0.5.0
Package name: langium-cli

Steps To Reproduce

1. create hello-world example (yo langium)
2. change grammar to

grammar HelloWorld

entry Model:
    (persons+=Person | greetings+=Greeting)*;

Person:
    'person' name=STRING;

Greeting:
    'Hello' person=[Person:STRING] '!';

hidden terminal WS: /\s+/;
terminal ID: /[_a-zA-Z][\w_]*/;
terminal INT returns number: /[0-9]+/;
terminal STRING: /"(\\"|[^"])*"|'(\\'|[^'])*'/;

hidden terminal ML_COMMENT: /\/\*[\s\S]*?\*\//;
hidden terminal SL_COMMENT: /\/\/[^\n\r]*/;

3. Start vscode extension and enter a model

person "Pierre with \" inside"
Hello "Pierre with \" inside"!

Link to code example: (see above)
The STRING definition is the crucial point...

The current behavior

Scoping resolution works as expected (misspell the name and you get an error as expected), but Syntax highlighting is broken (after the first string everything formatted as string).

The expected behavior

No broken highlighting.

[luan-xiaokun]

It seems that this is because the textmate file (the json file in the syntaxes folder) is not correctly generated. The terminal rule STRING generates the following

    {
      "name": "string.quoted.double.langium-debug",
      "begin": "\"",
      "end": "\\\\\"[^\"]\""
    },
    {
      "name": "string.quoted.single.langium-debug",
      "begin": "'",
      "end": "\\\\'[^']'"
    }

Basically, it means that it will take anything between a starting " and an ending \"x" as a string and highlight it as a string, where x is anything that is not ". Clearly there is a missing alternative operator | in the generated textmate file.

[msujew]

Note that the generated syntax highlighting file is just a helper to get you started with writing/extending the textMate grammar. The heuristics used in the generator to figure out the highlighting rules are fairly simple and are not intended to be complete. We accept improvements to the code though. So if you can figure out what's wrong there, any contributions are welcome.

[luan-xiaokun]

String and comment are using the same regex visitor. Since the textmate generator is doing a good job for multiline comments, we can modify the string regex to make it look like a comment regex, such as "(\\"|[^"])*?" (note the non-greedy option), this can fix the issue, though I haven't figured out how the visitor does it. (Oops, not working actually)

[msujew]

This has been fixed with #888.