Markdown no longer works with pdfTeX

[sorsted]

I tried compiling the following document with pdfTeX:

% (file name: test.tex)
\documentclass{article}
\usepackage{markdown}
\begin{document}

Using inline markdown:

\begin{markdown}

Hello World.

\end{markdown}
\end{document}

The result was an emergency stop and the error message displayed below. It compiles just fine with LuaTeX.

System information: I am using TeX Live 2023 (installed yesterday, October 22) on Windows 10. I got the same error message with TeX Live 2022, which was why I installed the 2023 version yesterday.

This is pdfTeX, Version 3.141592653-2.6-1.40.25 (TeX Live 2023) (preloaded format=pdflatex 2023.10.22)  23 OCT 2023 10:31
entering extended mode
 restricted \write18 enabled.
 %&-line parsing enabled.
**test.tex
(./test.tex
LaTeX2e <2023-06-01> patch level 1
L3 programming layer <2023-10-10>
(c:/texlive/2023/texmf-dist/tex/latex/base/article.cls
Document Class: article 2023/05/17 v1.4n Standard LaTeX document class
(c:/texlive/2023/texmf-dist/tex/latex/base/size10.clo
File: size10.clo 2023/05/17 v1.4n Standard LaTeX file (size option)
)
\c@part=\count185
\c@section=\count186
\c@subsection=\count187
\c@subsubsection=\count188
\c@paragraph=\count189
\c@subparagraph=\count190
\c@figure=\count191
\c@table=\count192
\abovecaptionskip=\skip48
\belowcaptionskip=\skip49
\bibindent=\dimen140
)
(c:/texlive/2023/texmf-dist/tex/latex/markdown/markdown.sty
(c:/texlive/2023/texmf-dist/tex/latex/l3kernel/expl3.sty
Package: expl3 2023-10-10 L3 programming layer (loader) 

(c:/texlive/2023/texmf-dist/tex/latex/l3backend/l3backend-pdftex.def
File: l3backend-pdftex.def 2023-10-10 L3 backend support: PDF output (pdfTeX)
\l__color_backend_stack_int=\count193
\l__pdf_internal_box=\box51
))
(c:/texlive/2023/texmf-dist/tex/generic/markdown/markdown.tex
\markdownOptionFrozenCacheCounter=\count194
\markdownInputFileStream=\read2
\markdownOutputFileStream=\write3
Package markdown Info: Using mode 3: The lt3luabridge package on input line 266
7.

(c:/texlive/2023/texmf-dist/tex/generic/lt3luabridge/lt3luabridge.tex
\g_luabridge_method_int=\count195

Package luabridge Info: Using shell escape as the bridging method

))
Package: markdown 2023-04-27 v2.23.0-0-g0b22f91 markdown renderer

(c:/texlive/2023/texmf-dist/tex/latex/paralist/paralist.sty
Package: paralist 2017/01/22 v2.7 Extended list environments
\pltopsep=\skip50
\plpartopsep=\skip51
\plitemsep=\skip52
\plparsep=\skip53
\pl@lab=\toks17
)
(c:/texlive/2023/texmf-dist/tex/latex/amsmath/amsmath.sty
Package: amsmath 2023/05/13 v2.17o AMS math features
\@mathmargin=\skip54

For additional information on amsmath, use the `?' option.
(c:/texlive/2023/texmf-dist/tex/latex/amsmath/amstext.sty
Package: amstext 2021/08/26 v2.01 AMS text

(c:/texlive/2023/texmf-dist/tex/latex/amsmath/amsgen.sty
File: amsgen.sty 1999/11/30 v2.0 generic functions
\@emptytoks=\toks18
\ex@=\dimen141
))
(c:/texlive/2023/texmf-dist/tex/latex/amsmath/amsbsy.sty
Package: amsbsy 1999/11/29 v1.2d Bold Symbols
\pmbraise@=\dimen142
)
(c:/texlive/2023/texmf-dist/tex/latex/amsmath/amsopn.sty
Package: amsopn 2022/04/08 v2.04 operator names
)
\inf@bad=\count196
LaTeX Info: Redefining \frac on input line 234.
\uproot@=\count197
\leftroot@=\count198
LaTeX Info: Redefining \overline on input line 399.
LaTeX Info: Redefining \colon on input line 410.
\classnum@=\count199
\DOTSCASE@=\count266
LaTeX Info: Redefining \ldots on input line 496.
LaTeX Info: Redefining \dots on input line 499.
LaTeX Info: Redefining \cdots on input line 620.
\Mathstrutbox@=\box52
\strutbox@=\box53
LaTeX Info: Redefining \big on input line 722.
LaTeX Info: Redefining \Big on input line 723.
LaTeX Info: Redefining \bigg on input line 724.
LaTeX Info: Redefining \Bigg on input line 725.
\big@size=\dimen143
LaTeX Font Info:    Redeclaring font encoding OML on input line 743.
LaTeX Font Info:    Redeclaring font encoding OMS on input line 744.
\macc@depth=\count267
LaTeX Info: Redefining \bmod on input line 905.
LaTeX Info: Redefining \pmod on input line 910.
LaTeX Info: Redefining \smash on input line 940.
LaTeX Info: Redefining \relbar on input line 970.
LaTeX Info: Redefining \Relbar on input line 971.
\c@MaxMatrixCols=\count268
\dotsspace@=\muskip16
\c@parentequation=\count269
\dspbrk@lvl=\count270
\tag@help=\toks19
\row@=\count271
\column@=\count272
\maxfields@=\count273
\andhelp@=\toks20
\eqnshift@=\dimen144
\alignsep@=\dimen145
\tagshift@=\dimen146
\tagwidth@=\dimen147
\totwidth@=\dimen148
\lineht@=\dimen149
\@envbody=\toks21
\multlinegap=\skip55
\multlinetaggap=\skip56
\mathdisplay@stack=\toks22
LaTeX Info: Redefining \[ on input line 2953.
LaTeX Info: Redefining \] on input line 2954.
)
(c:/texlive/2023/texmf-dist/tex/latex/amsfonts/amssymb.sty
Package: amssymb 2013/01/14 v3.01 AMS font symbols

(c:/texlive/2023/texmf-dist/tex/latex/amsfonts/amsfonts.sty
Package: amsfonts 2013/01/14 v3.01 Basic AMSFonts support
\symAMSa=\mathgroup4
\symAMSb=\mathgroup5
LaTeX Font Info:    Redeclaring math symbol \hbar on input line 98.
LaTeX Font Info:    Overwriting math alphabet `\mathfrak' in version `bold'
(Font)                  U/euf/m/n --> U/euf/b/n on input line 106.
))
(c:/texlive/2023/texmf-dist/tex/latex/csvsimple/csvsimple.sty
(c:/texlive/2023/texmf-dist/tex/latex/l3packages/l3keys2e/l3keys2e.sty
Package: l3keys2e 2023-10-10 LaTeX2e option processing using LaTeX3 keys
)
Package: csvsimple 2023/10/16 v2.5.0 LaTeX CSV file processing
)
(c:/texlive/2023/texmf-dist/tex/latex/csvsimple/csvsimple-legacy.sty
Package: csvsimple-legacy 2023/10/16 version 2.5.0 LaTeX2e CSV file processing

(c:/texlive/2023/texmf-dist/tex/latex/pgf/utilities/pgfrcs.sty
(c:/texlive/2023/texmf-dist/tex/generic/pgf/utilities/pgfutil-common.tex
\pgfutil@everybye=\toks23
\pgfutil@tempdima=\dimen150
\pgfutil@tempdimb=\dimen151
)
(c:/texlive/2023/texmf-dist/tex/generic/pgf/utilities/pgfutil-latex.def
\pgfutil@abb=\box54
)
(c:/texlive/2023/texmf-dist/tex/generic/pgf/utilities/pgfrcs.code.tex
(c:/texlive/2023/texmf-dist/tex/generic/pgf/pgf.revision.tex)
Package: pgfrcs 2023-01-15 v3.1.10 (3.1.10)
))
(c:/texlive/2023/texmf-dist/tex/latex/pgf/utilities/pgfkeys.sty
(c:/texlive/2023/texmf-dist/tex/generic/pgf/utilities/pgfkeys.code.tex
\pgfkeys@pathtoks=\toks24
\pgfkeys@temptoks=\toks25

(c:/texlive/2023/texmf-dist/tex/generic/pgf/utilities/pgfkeyslibraryfiltered.co
de.tex
\pgfkeys@tmptoks=\toks26
))) (c:/texlive/2023/texmf-dist/tex/latex/base/ifthen.sty
Package: ifthen 2022/04/13 v1.1d Standard LaTeX ifthen package (DPC)
)
(c:/texlive/2023/texmf-dist/tex/latex/etoolbox/etoolbox.sty
Package: etoolbox 2020/10/05 v2.5k e-TeX tools for LaTeX (JAW)
\etb@tempcnta=\count274
)
(c:/texlive/2023/texmf-dist/tex/latex/tools/shellesc.sty
Package: shellesc 2023/04/15 v1.0d unified shell escape interface for LaTeX
Package shellesc Info: Restricted shell escape enabled on input line 77.
)
\csv@file=\read3
\c@csvinputline=\count275
\c@csvrow=\count276
\c@csvcol=\count277
\csv@out=\write4
)
(c:/texlive/2023/texmf-dist/tex/latex/fancyvrb/fancyvrb.sty
Package: fancyvrb 2023/01/19 4.5a verbatim text (tvz,hv)

(c:/texlive/2023/texmf-dist/tex/latex/graphics/keyval.sty
Package: keyval 2022/05/29 v1.15 key=value parser (DPC)
\KV@toks@=\toks27
)
\FV@CodeLineNo=\count278
\FV@InFile=\read4
\FV@TabBox=\box55
\c@FancyVerbLine=\count279
\FV@StepNumber=\count280
\FV@OutFile=\write5
)
(c:/texlive/2023/texmf-dist/tex/latex/graphics/graphicx.sty
Package: graphicx 2021/09/16 v1.2d Enhanced LaTeX Graphics (DPC,SPQR)

(c:/texlive/2023/texmf-dist/tex/latex/graphics/graphics.sty
Package: graphics 2022/03/10 v1.4e Standard LaTeX Graphics (DPC,SPQR)

(c:/texlive/2023/texmf-dist/tex/latex/graphics/trig.sty
Package: trig 2021/08/11 v1.11 sin cos tan (DPC)
)
(c:/texlive/2023/texmf-dist/tex/latex/graphics-cfg/graphics.cfg
File: graphics.cfg 2016/06/04 v1.11 sample graphics configuration
)
Package graphics Info: Driver file: pdftex.def on input line 107.

(c:/texlive/2023/texmf-dist/tex/latex/graphics-def/pdftex.def
File: pdftex.def 2022/09/22 v1.2b Graphics/color driver for pdftex
))
\Gin@req@height=\dimen152
\Gin@req@width=\dimen153
)
(c:/texlive/2023/texmf-dist/tex/generic/ltxcmds/ltxcmds.sty
Package: ltxcmds 2020-05-10 v1.25 LaTeX kernel commands for general use (HO)
)
\markdownLaTeXCitationsCounter=\count281

(c:/texlive/2023/texmf-dist/tex/generic/gobble/gobble.sty
Package: gobble 2019/01/04 v0.2 Provides more gobble macros

(c:/texlive/2023/texmf-dist/tex/generic/gobble/gobble.tex))
(c:/texlive/2023/texmf-dist/tex/latex/url/url.sty
\Urlmuskip=\muskip17
Package: url 2013/09/16  ver 3.4  Verb mode for urls, etc.
)
\markdownLaTeXRowCounter=\count282
\markdownLaTeXRowTotal=\count283
\markdownLaTeXColumnCounter=\count284
\markdownLaTeXColumnTotal=\count285
\markdownLaTeXTable=\toks28
\markdownLaTeXTableAlignment=\toks29
\markdownLaTeXTableEnd=\toks30
) (./test.aux)
\openout1 = `test.aux'.

LaTeX Font Info:    Checking defaults for OML/cmm/m/it on input line 5.
LaTeX Font Info:    ... okay on input line 5.
LaTeX Font Info:    Checking defaults for OMS/cmsy/m/n on input line 5.
LaTeX Font Info:    ... okay on input line 5.
LaTeX Font Info:    Checking defaults for OT1/cmr/m/n on input line 5.
LaTeX Font Info:    ... okay on input line 5.
LaTeX Font Info:    Checking defaults for T1/cmr/m/n on input line 5.
LaTeX Font Info:    ... okay on input line 5.
LaTeX Font Info:    Checking defaults for TS1/cmr/m/n on input line 5.
LaTeX Font Info:    ... okay on input line 5.
LaTeX Font Info:    Checking defaults for OMX/cmex/m/n on input line 5.
LaTeX Font Info:    ... okay on input line 5.
LaTeX Font Info:    Checking defaults for U/cmr/m/n on input line 5.
LaTeX Font Info:    ... okay on input line 5.

(c:/texlive/2023/texmf-dist/tex/context/base/mkii/supp-pdf.mkii
[Loading MPS to PDF converter (version 2006.09.02).]
\scratchcounter=\count286
\scratchdimen=\dimen154
\scratchbox=\box56
\nofMPsegments=\count287
\nofMParguments=\count288
\everyMPshowfont=\toks31
\MPscratchCnt=\count289
\MPscratchDim=\dimen155
\MPnumerator=\count290
\makeMPintoPDFobject=\count291
\everyMPtoPDFconversion=\toks32
) (c:/texlive/2023/texmf-dist/tex/latex/epstopdf-pkg/epstopdf-base.sty
Package: epstopdf-base 2020-01-24 v2.11 Base part for package epstopdf
Package epstopdf-base Info: Redefining graphics rule for `.eps' on input line 4
85.

(c:/texlive/2023/texmf-dist/tex/latex/latexconfig/epstopdf-sys.cfg
File: epstopdf-sys.cfg 2010/07/13 v1.3 Configuration of (r)epstopdf for TeX Liv
e
))
\openout3 = `test.markdown.in'.

Package markdown Info: Buffering markdown input into the temporary input file "
test.markdown.in" and scanning for the closing token sequence "\end{markdown}" 
on input line 9.
Package markdown Info: The ending token sequence was found on input line 13.
Package markdown Info: Including markdown document "./test.markdown.in" on inpu
t line 13.
\g_tmpa_iow=\write6
\openout6 = `test.markdown.lua'.


Package luabridge Info: Writing a helper Lua script to file
(luabridge)             \markdownOptionHelperScriptFileName 


Package luabridge Info: Executing a helper Lua script from file
(luabridge)             \markdownOptionHelperScriptFileName 


! I can't find file `"|texlua ./test.markdown.lua"'.
<to be read again> 
                   \scan_stop: 
l.13 \end{markdown}
                   
(Press Enter to retry, or Control-Z to exit)
Please type another input file name
! Emergency stop.
<to be read again> 
                   \scan_stop: 
l.13 \end{markdown}
                   
*** (job aborted, file error in nonstop mode)

 
Here is how much of TeX's memory you used:
 6347 strings out of 474503
 149506 string characters out of 5756704
 1927043 words of memory out of 5000000
 28261 multiletter control sequences out of 15000+600000
 558069 words of font info for 36 fonts, out of 8000000 for 9000
 1141 hyphenation exceptions out of 8191
 82i,1n,80p,321b,827s stack positions out of 10000i,1000n,20000p,200000b,200000s
!  ==> Fatal error occurred, no output PDF file produced!

[Witiko]

Thank you for using the Markdown package for TeX and for describing your problem to us.

You didn't tell us the command you used to compile your document. However, line 3 in your log says "restricted \write18 enabled". This tells me that you did not enable shell access, as described in the user manual. Try typesetting your document with shell access enabled as follows and see if the error persists:

$ pdflatex --shell-escape test.tex

[sorsted]

No, I didn’t use --shell-escape, and I don’t think I’m going to, as I don’t know what external programs the package wants to use (I trust you, but not with my life, sorry 😉). In that case, I will probably have to stick to LuaTeX, I guess.

In any case, some of us don’t like using --shell-escape, so perhaps you could try to find a way to work around this with pdfTeX? And if not, perhaps the error message could be made more clear? I had no idea that --shell-escape was necessary, and I believe this has not always been the case, right? I seem to recall having used markdown in the past with pdfTeX without it. In fact, most LaTeX users (like myself) have barely heard of --shell-escape and probably never used it. A lot of us are used to compiling LaTeX by pressing a button in our editor of choice.

[Witiko]

No, I didn’t use --shell-escape, and I don’t think I’m going to, as I don’t know what external programs the package wants to use (I trust you, but not with my life, sorry 😉).

I understand. The package uses the shell to execute Lua code. A common solution to your problem is to run pdfTeX in a Docker container, see the example in the README. This way, you can grant pdfTeX shell access without exposing any data other than the document that you wish to compile.

In any case, some of us don’t like using --shell-escape, so perhaps you could try to find a way to work around this with pdfTeX?

The Markdown package uses Lua to parse markdown text. Since Lua is unavailable in pdfTeX, the only alternative to using the shell seems to be the rewrite of the markdown parser from Lua to TeX. However, this would be time-consuming and the resulting code would be more difficult to maintain as well as substantially slower. Therefore, there has not been much interest in pursuing this direction.

If you know about other alternatives to using the shell in pdfTeX, please let me know.

And if not, perhaps the error message could be made more clear?

It definitely should, thanks for bringing this to our attention. Currently, the execution of Lua code and all error handling is delegated to the lt3luabridge package, which uses the \sys_get_shell:nnNTF expl3 function in pdfTeX to determine whether the shell is available and to either use the shell or report an error:

\msg_new:nnnn
  { luabridge }
  { level-disabled }
  {
    Shell~escape~seems~to~be~disabled
  }
  {
    You~may~need~to~run~TeX~with~the~--shell-escape~or~the~
    --enable-write18~flag,~or~write~shell_escape=t~in~the~texmf.cnf~file.
  }

However, from your report it seems that the function \sys_get_shell:nnNTF fails to properly handle the case where the shell escape is disabled/restricted and does not produce a human-readable error.

I opened a ticket for your issue in Witiko/lt3luabridge#22.

I had no idea that --shell-escape was necessary, and I believe this has not always been the case, right? I seem to recall having used markdown in the past with pdfTeX without it.

Strange! The Markdown package for TeX has required the use of --shell-escape in TeX engines other than LuaTeX since its first release in 2016.

In fact, most LaTeX users (like myself) have barely heard of --shell-escape and probably never used it. A lot of us are used to compiling LaTeX by pressing a button in our editor of choice.

There are many LaTeX packages that require --shell-escape such as minted for source code highlighting, asymptote for creating vector graphics, and imakeidx for building indices. LaTeX editors such as Overleaf enable --shell-escape by default, which may be why some TeX users are unaware of it.

[Witiko]

@sorsted: After #372, compiling your example document test.tex with pdfLaTeX produces a human-readable error with instructions how to fix it:

! Package markdown Error: Shell escape is restricted

For immediate help type H <return>.
 ...                                              
                                                  
l.12 \end{markdown}
                   
? H

You may need to run TeX with the --shell-escape or the --enable-write18 flag,
or write shell_escape=t in the texmf.cnf file.

?

This update will be released in Markdown 3.3.0, which will be included in TeX Live 2024.