JSON Lexer produces Error tokens when whitespace surrounds ":" char #2010
Description
The JSON lexer will produce Error tokens if whitespace surrounds the : char. Repro:
from pygments.lexers.data import JsonLexer
from pprint import pprint
lexer = JsonLexer()
pprint(list(lexer.get_tokens('{"foo" : "bar"}')))Current behavior:
[(Token.Punctuation, '{'),
(Token.Name.Tag, '"foo"'),
(Token.Error, ' '), # <---- causes an exception to be raised
(Token.Punctuation, ':'),
(Token.Text.Whitespace, ' '),
(Token.Literal.String.Double, '"bar"'),
(Token.Punctuation, '}'),
(Token.Text.Whitespace, '\n')]
Previous behavior:
[(Token.Punctuation, '{'),
(Token.Name.Tag, '"foo"'),
(Token.Text, ' '),
(Token.Punctuation, ':'),
(Token.Text, ' '),
(Token.Literal.String.Double, '"bar"'),
(Token.Punctuation, '}'),
(Token.Text, '\n')]
I believe this was introduced in b4f0583. Now that a whitespace token is being used instead of text, the lexer will hit the fallback cause of producing an Error token:
pygments/pygments/lexers/data.py
Lines 589 to 597 in fb3edec
I noticed this through Sphinx, where any JSON code block that has whitespace around : produces this warning:
myfile.rst:Could not lex literal_block as "json". Highlighting skipped.