Custom lexer no longer worked in my computer.

[kilasuelika]

I use a gams lexer (gams.py) in my project. Some months ago I found that my lexer no longer worked and it works before. At that time I thought it was a bug and maybe fixed. But recently I found that the same code can work on Overleaf so I think the problem doesn't lie in lexer. The simplest latex example would be

\documentclass{article}
\usepackage{minted}
\begin{document}
\begin{minted}{gams.py:GAMSLexer -x}

set x=5;

\end{minted}
\end{document}

Error information:

PS E:\Connect\PT\SoftwareStack\test_minted> xelatex -shell-escape main.tex
This is XeTeX, Version 3.141592653-2.6-0.999994 (TeX Live 2022) (preloaded format=xelatex)
 \write18 enabled.
entering extended mode
(./main.tex
LaTeX2e <2022-11-01> patch level 1
L3 programming layer <2023-02-01>
(c:/texlive/2022/texmf-dist/tex/latex/base/article.cls
Document Class: article 2022/07/02 v1.4n Standard LaTeX document class
(c:/texlive/2022/texmf-dist/tex/latex/base/size10.clo))
(c:/texlive/2022/texmf-dist/tex/latex/minted/minted.sty
(c:/texlive/2022/texmf-dist/tex/latex/graphics/keyval.sty)
(c:/texlive/2022/texmf-dist/tex/latex/kvoptions/kvoptions.sty
(c:/texlive/2022/texmf-dist/tex/generic/ltxcmds/ltxcmds.sty)
(c:/texlive/2022/texmf-dist/tex/latex/kvsetkeys/kvsetkeys.sty))
(c:/texlive/2022/texmf-dist/tex/latex/fvextra/fvextra.sty
(c:/texlive/2022/texmf-dist/tex/latex/etoolbox/etoolbox.sty)
(c:/texlive/2022/texmf-dist/tex/latex/fancyvrb/fancyvrb.sty)
(c:/texlive/2022/texmf-dist/tex/latex/upquote/upquote.sty
(c:/texlive/2022/texmf-dist/tex/latex/base/textcomp.sty))
(c:/texlive/2022/texmf-dist/tex/latex/lineno/lineno.sty))
(c:/texlive/2022/texmf-dist/tex/latex/base/ifthen.sty)
(c:/texlive/2022/texmf-dist/tex/latex/tools/calc.sty)
(c:/texlive/2022/texmf-dist/tex/latex/tools/shellesc.sty)
(c:/texlive/2022/texmf-dist/tex/latex/ifplatform/ifplatform.sty
(c:/texlive/2022/texmf-dist/tex/generic/pdftexcmds/pdftexcmds.sty
(c:/texlive/2022/texmf-dist/tex/generic/infwarerr/infwarerr.sty)
(c:/texlive/2022/texmf-dist/tex/generic/iftex/iftex.sty))
(c:/texlive/2022/texmf-dist/tex/generic/catchfile/catchfile.sty
(c:/texlive/2022/texmf-dist/tex/generic/etexcmds/etexcmds.sty))
(c:/texlive/2022/texmf-dist/tex/generic/iftex/ifluatex.sty))
(c:/texlive/2022/texmf-dist/tex/generic/xstring/xstring.sty
(c:/texlive/2022/texmf-dist/tex/generic/xstring/xstring.tex))
(c:/texlive/2022/texmf-dist/tex/latex/framed/framed.sty)
(c:/texlive/2022/texmf-dist/tex/latex/float/float.sty))
(c:/texlive/2022/texmf-dist/tex/latex/xcolor/xcolor.sty
(c:/texlive/2022/texmf-dist/tex/latex/graphics-cfg/color.cfg)
(c:/texlive/2022/texmf-dist/tex/latex/graphics-def/xetex.def)
(c:/texlive/2022/texmf-dist/tex/latex/graphics/mathcolor.ltx))
E:\Connect\PT\SoftwareStack\test_minted>set  /p x=C:\Users\admin\AppData\Local\Programs\Python\Python311\Scripts\pygmentize.exe 0<nul: 1>>main.aex

E:\Connect\PT\SoftwareStack\test_minted>set  /p x= 0<nul: 1>>main.aex

E:\Connect\PT\SoftwareStack\test_minted>set  /p x= 0<nul: 1>>main.aex
system returned with code 1

(c:/texlive/2022/texmf-dist/tex/latex/l3backend/l3backend-xetex.def)
(./main.aux) (c:/texlive/2022/texmf-dist/tex/latex/base/ts1cmr.fd)
(./_minted-main/default.pygstyle)Error: no lexer for alias 'gams.py:GAMSLexer -x' found
system returned with code 1


! Package minted Error: Missing Pygments output; \inputminted was
probably given a file that does not exist--otherwise, you may need
the outputdir package option, or may be using an incompatible build tool,
or may be using frozencache with a missing file.

See the minted package documentation for explanation.
Type  H <return>  for immediate help.
 ...

l.8 \end{minted}

? exit
No pages of output.
Transcript written on main.log.
I cannot find emacs in the PATH.

I have updated all to the latest including pygments. I test installed pygments with

 C:\Users\admin\AppData\Local\Programs\Python\Python311\Scripts\pygmentize.exe -l gams.py:GAMSLexer -x test.gams

And it works perfectly.

[muzimuzhi]

If you check the log file, you'll see (on Windows)

runsystem(pygmentize -l "gams.py:GAMSLexer -x" ...)...executed.

Thus the culprit is the quotes around gams.py:GAMSLexer -x. The quotes are added in 25e1f30 and shipped with minted v2.7 to resolve other problems. Overleaf's texlive 2022 still uses minted v2.5 thus the different behaviors.

In example below a new boolean option custom lexer is provided (which supersedes #176 in part). If set, it will pass cli option -x to pygmentize. Now you can use custom lexer through \begin{minted}[custom lexer]{gams.py:GAMSLexer}. I don't speak GAMS so a custom Python lexer is used as an example.

% based on https://github.com/gpoore/minted/issues/176#issuecomment-695344998
\begin{filecontents}[noheader]{my_lexer.py}
from pygments.lexers.markup import TexLexer
from pygments.token import Keyword


class TexLexer2(TexLexer):
    """
    Improved lexer for the TeX and LaTeX typesetting languages.
    Character "@" is treated part of command names.
    """

    TexLexer.tokens['root'][4] = (r'\\([a-zA-Z@]+|.)', Keyword, 'command')
\end{filecontents}

\documentclass{article}
\usepackage{minted}

\makeatletter
\ifwindows
  \renewcommand{\minted@optlistcl@quote}[2]{%
    \ifstrempty{#2}{\detokenize{#1}}{\detokenize{#1="#2"}}}
\else
  \renewcommand{\minted@optlistcl@quote}[2]{%
    \ifstrempty{#2}{\detokenize{#1}}{\detokenize{#1='#2'}}}
\fi

% similar to \minted@def@optcl@switch
\newcommand{\minted@def@optcl@novalue}[2]{%
  \define@booleankey{minted@opt@g}{#1}%
    {\minted@addto@optlistcl{\minted@optlistcl@g}{#2}{}%
     \@namedef{minted@opt@g:#1}{true}}
    {\@namedef{minted@opt@g:#1}{false}}
  \define@booleankey{minted@opt@g@i}{#1}%
    {\minted@addto@optlistcl{\minted@optlistcl@g@i}{#2}{}%
     \@namedef{minted@opt@g@i:#1}{true}}
    {\@namedef{minted@opt@g@i:#1}{false}}
  \define@booleankey{minted@opt@lang}{#1}%
    {\minted@addto@optlistcl@lang{minted@optlistcl@lang\minted@lang}{#2}{}%
     \@namedef{minted@opt@lang\minted@lang:#1}{true}}
    {\@namedef{minted@opt@lang\minted@lang:#1}{false}}
  \define@booleankey{minted@opt@lang@i}{#1}%
    {\minted@addto@optlistcl@lang{minted@optlistcl@lang\minted@lang @i}{#2}{}%
     \@namedef{minted@opt@lang\minted@lang @i:#1}{true}}
    {\@namedef{minted@opt@lang\minted@lang @i:#1}{false}}
  \define@booleankey{minted@opt@cmd}{#1}%
      {\minted@addto@optlistcl{\minted@optlistcl@cmd}{#2}{}%
        \@namedef{minted@opt@cmd:#1}{true}}
      {\@namedef{minted@opt@cmd:#1}{false}}
}

\minted@def@optcl@novalue{custom lexer}{-x}
\makeatother

\begin{document}
% runsystem(pygmentize -l 'tex' ...)
\begin{minted}[custom lexer=false]{tex}
  % built-in tex lexer
  \@namedef{cmd}{$a + \alpha$} x
\end{minted}

% runsystem(pygmentize -l 'my_lexer.py:TexLexer2' ... -x ...)
\begin{minted}[custom lexer]{my_lexer.py:TexLexer2}
  % custom tex lexer which treats "@" as letter
  \@namedef{cmd}{$a + \alpha$} y
\end{minted}
\end{document}

[tobiasgrosser]

I encountered the same issue. @muzimuzhi, would it make sense to contribute your custom lexer option to minted as a pull request?

[muzimuzhi]

@tobiasgrosser Hi Tobias, I'm really happy my patch helps people with their documents, but first I think it has to be decided how minted will support custom lexers. I mean, the once-working \begin{minted}{gams.py:GAMSLexer -x} is never an officially supported feature (thus its deprecation is not a bug).

In a more general way, I feel sth like \begin{minted}[custom lexer=gams.py:GAMSLexer]{fallback lang} (with some python script to provide fallback logic) or \begin{minted}[custom lexer=gams.py]{GAMSLexer} is better.

[tobiasgrosser]

Sure. Now, admittedly the fact that there are uses of this in the wild makes the discussion if the loss of the other feature is a bug or not more academic 🙂. In an optimal world the -x syntax would just continue to work.

[nafur]

I'm having the same issue right now. A proper custom lexer option would be great!

[gpoore]

minted version 3.0 is now under development, thanks to a grant from the TeX Users Group. It will include official support for custom lexers. It will also be able to be extended using Python, not just LaTeX macro programming, which will make possible many new lexer-related features. Progress on custom lexers will be tracked in #372. Initial beta releases of minted version 3.0 are expected by early 2024.