Option "escapeinside=``" does not work in *nix

[muzimuzhi]

Example

When user sets minted option escapeinside=`` , the latex engine will run pygmentize with option -P "escapeinside=``". But character U+0060 (`) has special syntax in *nix shells, so the following example to use ` as escape delimiter, simplified from #260, won't work:

\documentclass[11pt]{beamer}
\usetheme[progressbar=head,numbering=fraction,block=fill]{metropolis}
\usepackage{minted}

\begin{document}

\begin{frame}[fragile]{title}
\begin{minted}[escapeinside=``, breaklines=true]{haskell}
class Applicative m => Monad m where
(>>=) :: m a -> (a -> m b) -> m b
(>>) :: m a -> m b -> m b
mx >> my =`\pause` mx >>=`\pause` \_ -> my
\end{minted}
\end{frame}

% uncomment the following line to keep the .pyg temp file
% \csname minted@drafttrue\endcsname

\end{document}

output

Using pygmentize only

This can be manually confirmed in macOS (with bash or zsh) or other *nix OS (for example, on overleaf.com, which runs latex in "Linux 4.19.112+ #1 SMP"). Suppose I have a file test-escapeinside.pyg containing

class Applicative m => Monad m where
(>>=) :: m a -> (a -> m b) -> m b
(>>) :: m a -> m b -> m b
mx >> my =`\pause` mx >>=`\pause` \_ -> my

• run with -P "escapeinside=``", which is pygmentize -l haskell -f latex -P commandprefix=PYG -F tokenmerge -P "escapeinside=``" -o output.pygtex test-escapeinside.pyg, then the output.pygtex contains

  \PYG{p}{`}\PYG{n+nf}{\PYGZbs{}}\PYG{n}{pause}\PYG{p}{`}

  which marks \pause as part of the code, not an escaped latex command.
• run with -P "escapeinside=\`\`", which is pygmentize -l haskell -f latex -P commandprefix=PYG -F tokenmerge -P "escapeinside=\`\`" -o output.pygtex test-escapeinside.pyg `, then the `output.pygtex, then the output.pygtex contains

  \PYG{esc}{\pause}

  which is the expected markup.

Discussion

So the *nix shell needs ` character to be escaped. But since \` is (re)defined by latex2e as a text accent command, escapeinside=\`\` results in pygmentize ... -P "escapeinside=<expansion of `><expansion of `>".

I find setting \AtBeginEnvironment{minted}{\chardef\`=`\`} and escapeinside=\`\` works, at the risk of cannot use \` as text accent anymore inside minted environment. Also, this only fixes a single environment. Changing how the value of option escapeinside is parsed may be a more general and elegant solution.

Other special characters with their corresponding escaped form defined as macros rather than characters may run into similar problems.

[gpoore]

Fixing this will need to make escapeinside=`` work, since that's what people naturally expect. A temporary fix would probably be to add a sanitization step to assembling the pygmentize command, and insert backslashes at that stage.

There are likely other escaping issues remaining elsewhere...there are a lot of ways that shell commands and macros can interact in undesirable ways. My longer-term plan is to have minted write all settings and the code to a temp file in a serialization format, and then create a new Python program that users will install alongside pygmentize and that will process the temp file data. That will eliminate all shell escaping issues completely. Also, a new Python wrapper for Pygments will easily enable new features like code extraction based on regex that could be implemented currently, but would be cumbersome due to multiple iterations of shell escape and temp files. I don't know when I'll get around to trying the new approach, though.

[romildo]

My longer-term plan is to have minted write all settings and the code to a temp file in a serialization format, and then create a new Python program that users will install alongside pygmentize and that will process the temp file data. That will eliminate all shell escaping issues completely. Also, a new Python wrapper for Pygments will easily enable new features like code extraction based on regex that could be implemented currently, but would be cumbersome due to multiple iterations of shell escape and temp files. I don't know when I'll get around to trying the new approach, though.

That is the approach taken in pygmentex.

[muzimuzhi]

@gpoore
That longer-term plan is absolutely an attraction, though it will take many efforts.