Escapeinside and Console Lexers

[lcnittl]

I just came across this strange behavior of escapeinside in conjunction with console lexers (like ps1con or batcon)

When using the snipped from the manual and swapping the lexer the output eats the beginning of the line (and things - like the prompt - from previous lines, if not in the first):

I also tried to use different escapeinside characters, like ` or $.

PS: I was of course using minted and not mínted, but did not know how to have \end{minted} inside a minted environment :) .

[lcnittl]

Never mind, I think I deduced that this comes from pygmentize, after reading how to use escapeinside on pygmentize directly in #262.

file.txt:

PS C:\Users\lcnittl\ucloud_knittlfrac92\bckp\LaTeX\PPTeX> git commit -m "Update CLS, use more LaTeX3" -S -a ; git push
Check for case conflicts.................................................|Passed|
Check for merge conflicts................................................Passed
Trim Trailing Whitespace.................................................Passed
Fix End of Files.........................................................Passed
Check Toml...........................................(no files to check)Skipped
Check Yaml...........................................(no files to check)Skipped
Fix requirements.txt.................................(no files to check)Skipped
prettier.............................................(no files to check)Skipped
isort seed config....................................(no files to check)Skipped
isort................................................(no files to check)Skipped
black................................................(no files to check)Skipped
gpg: AllowSetForegroundWindow(29316) failed: Access is denied.

[master 8b17c7a] Update CLS, use more LaTeX3
 1 file changed, 397 insertions(+), 362 deletions(-)
 rewrite PPTeX.cls (83%)
Enumerating objects: 5, done.
Counting objects: 100% (5/5), done.
Delta compression using up to 8 threads
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 2.24 KiB | 2.24 MiB/s, done.
Total 3 (delta 2), reused 0 (delta 0), pack-reused 0
remote: . Processing 1 references
remote: Processed 1 references in total
To https://git.maulide.univie.ac.at/knittlfrac92/PPTeX.git
   44ecee4..8b17c7a  master -> master
PS C:\Users\lcnittl\ucloud_knittlfrac92\bckp\LaTeX\PPTeX> _

pygmentize.exe -P "escapeinside=||" -l ps1con -f latex .\file.txt:

\begin{Verbatim}[commandchars=\\\{\},codes={\catcode`\$=3\catcode`\^=7\catcode`\_=8}]
\PY{l+s+s2}{\PYZdq{}}\PY{l+s+s2}{Update CLS, use more LaTeX3}\PY{l+s+s2}{\PYZdq{}}\PY{g+go}{ \PYZhy{}S \PYZhy{}a ; git push}
\PY{esc}{Passed}
\PY{g+go}{Check for merge conflicts................................................Passed}
\PY{g+go}{Trim Trailing Whitespace.................................................Passed}
\PY{g+go}{Fix End of Files.........................................................Passed}
\PY{g+go}{Check Toml...........................................(no files to check)Skipped}
\PY{g+go}{Check Yaml...........................................(no files to check)Skipped}
\PY{g+go}{Fix requirements.txt.................................(no files to check)Skipped}
\PY{g+go}{prettier.............................................(no files to check)Skipped}
\PY{g+go}{isort seed config....................................(no files to check)Skipped}
\PY{g+go}{isort................................................(no files to check)Skipped}
\PY{g+go}{black................................................(no files to check)Skipped}
\PY{g+go}{gpg: AllowSetForegroundWindow(29316) failed: Access is denied.}

\PY{g+go}{[master 8b17c7a] Update CLS, use more LaTeX3}
\PY{g+go}{ 1 file changed, 397 insertions(+), 362 deletions(\PYZhy{})}
\PY{g+go}{ rewrite PPTeX.cls (83\PYZpc{})}
\PY{g+go}{Enumerating objects: 5, done.}
\PY{g+go}{Counting objects: 100\PYZpc{} (5/5), done.}
\PY{g+go}{Delta compression using up to 8 threads}
\PY{g+go}{Compressing objects: 100\PYZpc{} (3/3), done.}
\PY{err}{|}\PY{g+go}{ 2.24 MiB/s, done.}
\PY{g+go}{Total 3 (delta 2), reused 0 (delta 0), pack\PYZhy{}reused 0}
\PY{g+go}{remote: . Processing 1 references}
\PY{g+go}{remote: Processed 1 references in total}
\PY{g+go}{To https://git.maulide.univie.ac.at/knittlfrac92/PPTeX.git}
\PY{g+go}{   44ecee4..8b17c7a  master \PYZhy{}\PYZgt{} master}
\PY{g+gp}{PS C:\PYZbs{}Users\PYZbs{}lcnittl\PYZbs{}ucloud\PYZus{}knittlfrac92\PYZbs{}bckp\PYZbs{}LaTeX\PYZbs{}PPTeX\PYZgt{} }\PY{n}{\PYZus{}}
\end{Verbatim}

Feel free to close this.

[gpoore]

There are a combination of issues here.

• If you want console highlighting for Python, you want the pycon lexer. Using lexers meant for other languages will tend to introduce problems.
• Even with pycon, text before the escapeinside is discarded. This may be an issue with console lexers in general. The implementation of escapeinside in Pygments has always been somewhat fragile, but this is likely the worst case I've seen. I may need to add yet another note to the documentation about how escapeinside is fragile and in some cases can fail in significant ways.

I'll close this once I decide if the documentation needs an additional note about this case.

[muzimuzhi]

• Even with pycon, text before the escapeinside is discarded. This may be an issue with console lexers in general. The implementation of escapeinside in Pygments has always been somewhat fragile, but this is likely the worst case I've seen. I may need to add yet another note to the documentation about how escapeinside is fragile and in some cases can fail in significant ways.

The specific pycon example in this issue has been fixed in Pygments. Though, similar problems still exist in other console/REPL lexers, see for example #388.

Running following example on overleaf.com (remember to use "Recompile from scratch" after switching to another TeX Live version),

• Using overleaf's TeX Live 2021 image, problem reproduces.
  LaTeX2e 2020-02-02 + minted v2.5 + pygments v2.3.1
  
• Using overleaf's TeX Live 2021 image, problem doesn't reproduce anymore.
  LaTeX2e 2021-06-01 + minted v2.5 + pygments v2.9.0
  

\documentclass{article}
\usepackage{minted}

\begin{document}
\begin{itemize}
    \item \fmtname, \fmtversion
    \item \csname ver@minted.sty\endcsname
    \item \input{|pygmentize -V}
\end{itemize}

Valid input
\begin{minted}[escapeinside=||]{pycon}
>>> def f(x): y = x|\colorbox{green}{**}|2; return y
\end{minted}

Invalid input
\begin{minted}[escapeinside=||]{pycon}
>>> def f(x): y = x|\colorbox{green}{**}|2
... return y
\end{minted}
\end{document}

Related issue in pygments' repo is pygments/pygments#1516, which has been fixed since v2.7.2.