PythonConsoleLexer does not highlight correctly custom exceptions

[thepabloaguilar]

Hi, I'm having some problems with pygments because it is not highlighting my custom exceptions correctly, you can find in the following script how to reproduce the error:

from pygments import highlight
from pygments.lexers import PythonConsoleLexer
from pygments.formatters import HtmlFormatter


pycon3 = PythonConsoleLexer(python3=True, stripnl=False)
pycon3.add_filter('raiseonerror')
formatter = HtmlFormatter()

code = '''
>>> IOFailure(1).unwrap()
Traceback (most recent call last):
  ...
returns.primitives.exceptions.UnwrapFailedError
'''

highlight(code, pycon3, formatter)  # => raises pygments.filters.ErrorToken

You'll get something like this as the output:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/pabloaguilar/.cache/pypoetry/virtualenvs/returns-h6SXr9Wf-py3.8/lib/python3.8/site-packages/pygments/__init__.py", line 84, in highlight
    return format(lex(code, lexer), formatter, outfile)
  File "/home/pabloaguilar/.cache/pypoetry/virtualenvs/returns-h6SXr9Wf-py3.8/lib/python3.8/site-packages/pygments/__init__.py", line 63, in format
    formatter.format(tokens, realoutfile)
  File "/home/pabloaguilar/.cache/pypoetry/virtualenvs/returns-h6SXr9Wf-py3.8/lib/python3.8/site-packages/pygments/formatter.py", line 95, in format
    return self.format_unencoded(tokensource, outfile)
  File "/home/pabloaguilar/.cache/pypoetry/virtualenvs/returns-h6SXr9Wf-py3.8/lib/python3.8/site-packages/pygments/formatters/html.py", line 879, in format_unencoded
    for t, piece in source:
  File "/home/pabloaguilar/.cache/pypoetry/virtualenvs/returns-h6SXr9Wf-py3.8/lib/python3.8/site-packages/pygments/formatters/html.py", line 710, in _wrap_div
    for tup in inner:
  File "/home/pabloaguilar/.cache/pypoetry/virtualenvs/returns-h6SXr9Wf-py3.8/lib/python3.8/site-packages/pygments/formatters/html.py", line 728, in _wrap_pre
    for tup in inner:
  File "/home/pabloaguilar/.cache/pypoetry/virtualenvs/returns-h6SXr9Wf-py3.8/lib/python3.8/site-packages/pygments/formatters/html.py", line 753, in _format_lines
    for ttype, value in tokensource:
  File "/home/pabloaguilar/.cache/pypoetry/virtualenvs/returns-h6SXr9Wf-py3.8/lib/python3.8/site-packages/pygments/filter.py", line 20, in _apply
    for token in filter_.filter(lexer, stream):
  File "/home/pabloaguilar/.cache/pypoetry/virtualenvs/returns-h6SXr9Wf-py3.8/lib/python3.8/site-packages/pygments/filters/__init__.py", line 196, in filter
    raise self.exception(value)
pygments.filters.ErrorToken: r

As you can see in the output, the problem is the last line:

# WORKS
Something: returns.primitives.exceptions.UnwrapFailedError

# DOES NOT WORK
returns.primitives.exceptions.UnwrapFailedError

pygments version: 2.6.1

This is not a recent problem, some people have opened issues about it:

• PythonConsoleLexer is unable to parse full paths to excecptions #1187
• HTML Highlighting of User-Defined Python Exceptions generates a pygments.filters.ErrorToken #1015

---

refs dry-python/returns#568
refs sphinx-doc/sphinx#8156

[Anteru]

Looks like this does reproduce with 2.7.1 as well. @birkenfeld any ideas? You've looked at the referenced bugs, did we regress?

[birkenfeld]

I think I was tripped because the terminal256 formatter does not visible display error tokens (terminal does).

Easy to fix though.

[birkenfeld]

@thepabloaguilar thanks for the issue!

@Anteru I've put the changelog entry under a 2.7.2 heading; we should release this fix soon-ish but since it isn't a regression it doesn't need an immediate release, let's wait for a week or so for more potential fixes to be made.

[thepabloaguilar]

Thanks for the fast answer and solution! 👏🏽