Tcl lexer: \ and $ interpreted as errors fix, add string escape handling, adding test case (fixing issue #2686)

[trungtruong1]

Update posted in the comments!

Good day! I am proposing a fix for the issue #2686.

I have added some changes to the lexer so that it no longer considers backslashes as errors. The lexer will now consider \ to be String.Escape. Standalone $ will be considered as Text.

Details changes
Path: pygments/lexers/tcl.py

Result:
Before:

After:

Tokens:

'set'         Keyword
' '           Text.Whitespace
'amount'      Text
' '           Text.Whitespace
'\\$'         Literal.String.Escape
'16.42'       Literal.Number.Float
'\n'          Text

'puts'        Name.Builtin
' '           Text.Whitespace
'lots'        Text
'\\n'         Literal.String.Escape
'of'          Text
'\\n'         Literal.String.Escape
'\\n'         Literal.String.Escape
'\\n'         Literal.String.Escape
'\\n'         Literal.String.Escape
'\\n'         Literal.String.Escape
'\\n'         Literal.String.Escape
'newlines'    Text
'\n'          Text

'set'         Keyword
' '           Text.Whitespace
'somevar'     Text
' '           Text.Whitespace
'{'           Keyword
'\n    '      Text.Whitespace
'This'        Name.Variable
' '           Text.Whitespace
'is'          Text
' '           Text.Whitespace
'a'           Text
' '           Text.Whitespace
'literal'     Text
' '           Text.Whitespace
'$ '          Text
'sign,'       Text
' '           Text.Whitespace
'and'         Text
' '           Text.Whitespace
'this'        Text
' '           Text.Whitespace
'\\}'         Literal.String.Escape
' '           Text.Whitespace
'escaped'     Text
'\n'          Text

'    '        Text.Whitespace
'brace'       Name.Variable
' '           Text.Whitespace
'remains'     Text
' '           Text.Whitespace
'uninterpreted' Text
'\n'          Text

'}'           Keyword
'\n'          Text

Please inform me if any alterations or modifications are necessary. I would greatly appreciate any feedback or comments!

[McSinyx]

Just driving by, but since tests for Tcl are rather lacking, I'd suggest adding the relevant ones.

[McSinyx]

Consider also line break and character escape sequences, e.g.

puts \
\164\x68\u1eed

[trungtruong1]

Hello, thank you for the comment, I have made the changes required to address the issue.

[trungtruong1]

Update new change:

@McSinyx have pointed out that \ character at the end of a line is still considered as an error, I have added another rule to the lexer to address this.

Also, his example:

puts \
\164\x68\u1eed

has shown that my change is not considering the case of 16-bit Unicode escape sequences, 8-bit Hexadecimal escape sequences, and 8-bit Octal escape sequences properly.

This is the new output for the updated lexer:

[trungtruong1]

Additionally, I'm adding a new file named test_string_escape.txt, which contains examples of string escapes related to this issue. This addition will enhance our test suite, as the current test coverage for the Tcl lexer is relatively limited.