Define a MyST lexer for pygments

[pradyunsg]

Is your feature request related to a problem? Please describe.

"I'm always frustrated when..." I have a code block example of MyST, and it is not highlighted to make the syntax elements visible because there's no syntax highlighter that I can use with pygments that gives me highlighted MyST sources. :)

Describe the solution you'd like

A definition of a MySTLexer for pygments in this package (or upstream!) that correctly handles code blocks with MyST directives as well as inline MyST directives. :)

Describe alternatives you've considered

Maintaining status quo... which is "fine".

Additional context

A/B screenshots -- highlighting of equivalent example snippets in MyST and reST, using the default theme in pygments (md lexer vs rest lexer).

[welcome]

Thanks for opening your first issue here! Engagement like this is essential for open source projects! 🤗

If you haven't done so already, check out EBP's Code of Conduct. Also, please try to follow the issue template as it helps other community members to contribute more effectively.

If your issue is a feature request, others may react to it, to raise its prominence (see Feature Voting).

Welcome to the EBP community! 🎉

[choldgraf]

I wonder how this relates to https://github.com/executablebooks/myst-language-support ?

That's the LSP implementation for MyST, which gets us syntax highlighting in any IDE that supports the LSP (currently this is mostly VSCode I guess, but JupyterLab should get it soon). Is pygments support something in addition to this? Or could we piggyback on this LSP implementation to easier have a pygments implementation as well?

[pradyunsg]

We'll need to write something like https://github.com/pygments/pygments/blob/b2c91c70ee536b0472100d1273818f8bb45529fe/pygments/lexers/markup.py#L499.

[chrisjsewell]

Yeh good idea thanks.
I wonder if we can essentially write a new markdown-it-py renderer (a bit like https://github.com/executablebooks/mdformat) that outputs the requisite pygments tokens, and have pygments recognise this as lexer plugin.

[chrisjsewell]

I've had an initial play around with this now:

MyST-Parser/myst_parser/_docs.py

Line 383 in de31423

class MystLexer(MarkdownLexer):

Its nothing "rigourous" but its looking quite nice in the documentation, like: https://myst-parser.readthedocs.io/en/latest/syntax/cross-referencing.html#reference-roles