Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Custom tokens don't backtrack to the lower level #1986

Closed
isidentical opened this issue Dec 10, 2021 · 4 comments
Closed

Custom tokens don't backtrack to the lower level #1986

isidentical opened this issue Dec 10, 2021 · 4 comments
Labels
A-formatting area: changes to formatters
Milestone

Comments

@isidentical
Copy link

isidentical commented Dec 10, 2021

Normally styles backtrack to the lower level tokens when doing style resolution. For example if there is no custom styling for Name.Function, it will try to find the style of Name and use it. But this doesn't work with custom tokens:

import pygments
from pygments.lexer import RegexLexer
from pygments.token import *
from pygments.style import Style
from pygments.formatters.terminal256 import Terminal256Formatter

class TestLexer(RegexLexer):
    tokens = {
        'root': [
            (r'A', Name.Function),
            (r'B', Name.Function.CUSTOM),
        ]
    }

class Style1(Style):
    styles = {
        Name: "#FA9BFA"
    }

class Style2(Style):
    styles = {
        Name.Function: "#FA9BFA",
        Name.Function.CUSTOM: "#FA9BFA"
    }


for style in [Style1, Style2]:
    print(pygments.highlight(
        code="AB",
        lexer=TestLexer(),
        formatter=Terminal256Formatter(style=style),
    ).strip())

I would expect both of these to work and output the same colors, but the initial style only highlights the Name.Function, and does not do anything for Name.Function.CUSTOM.
image

@birkenfeld
Copy link
Member

I agree.

@Anteru
Copy link
Collaborator

Anteru commented Dec 10, 2021

Uh, that's actually a serious issue as we have a bunch of lexers relying on this...

@Anteru Anteru added the A-formatting area: changes to formatters label Jan 4, 2022
@Anteru Anteru added this to the 2.12.0 milestone Jan 4, 2022
@jeanas
Copy link
Contributor

jeanas commented Jan 4, 2022

The Terminal256Formatter and IRCFormatter use the pattern ttype = ttype[:-1] instead of ttype = ttype.parent. Experimenting with it:

>>> from pygments.token import Token
>>> Token.Name.Custom[:-1]
('Name',)

So it makes ttype a tuple and lookups fail because they expect a token.

Is this the result of an earlier design of the Token object? Should we change it to make indexing return a token, or should we update the formatters to use the parent attribute?

There is also the GroffFormatter, which as far as I can see does not inherit styles at all.

@jeanas
Copy link
Contributor

jeanas commented Jan 6, 2022

Thanks for the fix! I've opened #2024 about the GroffFormatter (Pygments is already two years ahead of its time, yay :-).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-formatting area: changes to formatters
Projects
None yet
Development

No branches or pull requests

5 participants
@birkenfeld @Anteru @jeanas @isidentical and others