You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Here’s the upstream Pygments TypeScriptLexer working as normal:
$ python
Python 3.8.10 (default, Mar 13 2023, 10:26:41)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pygments, pygments.lexers
>>> print(*pygments.lex("let m: Map<K, V>;", pygments.lexers.TypeScriptLexer()), sep='\n')
(Token.Keyword.Declaration, 'let')
(Token.Text.Whitespace, ' ')
(Token.Name.Other, 'm')
(Token.Operator, ':')
(Token.Text.Whitespace, ' ')
(Token.Keyword.Type, 'Map')
(Token.Operator, '<')
(Token.Name.Other, 'K')
(Token.Punctuation, ',')
(Token.Text.Whitespace, ' ')
(Token.Name.Other, 'V')
(Token.Operator, '>')
(Token.Punctuation, ';')
(Token.Text.Whitespace, '\n')
But if I start a new Python process and instead import jsx before the first time I use TypeScriptLexer, TypeScriptLexer incorrectly produces Token.Error tokens:
$ python
Python 3.8.10 (default, Mar 13 2023, 10:26:41)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pygments, pygments.lexers, jsx
>>> print(*pygments.lex("let m: Map<K, V>;", pygments.lexers.TypeScriptLexer()), sep='\n')
(Token.Keyword.Declaration, 'let')
(Token.Text.Whitespace, ' ')
(Token.Name.Other, 'm')
(Token.Operator, ':')
(Token.Text.Whitespace, ' ')
(Token.Keyword.Type, 'Map')
(Token.Punctuation, '<')
(Token.Name.Tag, 'K')
(Token.Error, ',')
(Token.Text, ' ')
(Token.Name.Attribute, 'V')
(Token.Punctuation, '>')
(Token.Punctuation, ';')
(Token.Text.Whitespace, '\n')
This is because jsx-lexer destructively mutates pygments.lexers.javascript.JavascriptLexer.tokens which doesn’t belong to it:
Here’s the upstream Pygments
TypeScriptLexer
working as normal:But if I start a new Python process and instead import
jsx
before the first time I useTypeScriptLexer
,TypeScriptLexer
incorrectly producesToken.Error
tokens:This is because jsx-lexer destructively mutates
pygments.lexers.javascript.JavascriptLexer.tokens
which doesn’t belong to it:jsx-lexer/jsx/lexer.py
Lines 8 to 9 in 5d764e7
The text was updated successfully, but these errors were encountered: