-
Notifications
You must be signed in to change notification settings - Fork 695
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
markdown lexer fails on fenced code bocks of type raw #1616
Comments
Perhaps the fix here is to not have an alias for RawTokenLexer? (https://github.com/pygments/pygments/blob/master/pygments/lexers/special.py#L58) I'm not familiar with why this exists, but it's not clear why you'd want to expose RawTokenLexer for filename matching, and letting it be exposed in this way allows it to match as a lexer in a variety of contexts. Or perhaps it shouldn't have a binary regex? What's the motivation for swapping from string input in the interface to a lexer to binary strings? To me (naive about what's going on here) that seems like a violation of the interface. |
As a workaround, I tried a monkey patch in my application that seemed to work:
I'm not suggesting this is the fix, but it shows that this does indeed avoid the issue. I believe this makes MarkdownLexer treat the raw fenced code block case the same as it would an unrecognized language. I'm not familiar enough with RawTokenLexer to know why it needs to be recognized with the "raw" alias. It seems like this is a pygments internal utility, not meant to be used for general language parsing. |
Thanks for the report. I'm fixing the string/bytes issue for the next 2.7 release, and for 2.8 we'll remove the alias. |
for example, the following produces the indicated error:
I believe this happens because the markdown lexer will parse the fenced code and see "raw" which invokes the RawTokenLexer which I don't think is meant to be used like a normal lexer (see https://github.com/pygments/pygments/blob/master/pygments/lexers/special.py#L45-L49)
You can see this in the wild using this README.md:
https://github.com/netdata/netdata/blob/master/collectors/perf.plugin/README.md
This repo has nearly 50k stars and over 4.5k forks so it seems pretty reasonable to handle this kind of issue.
The text was updated successfully, but these errors were encountered: