Literal square brackets are dropped when converting from markdown to markdown

[jez]

Explain the problem.

Converting from markdown to markdown changes [1] to \[1\]:

❯ cat foo.md
This is a footnote[1].

❯ pandoc -f markdown -t markdown foo.md
This is a footnote\[1\].

This is normally fine, however, if in the future you ever render that markdown file (which contains \[1\]) while also enabling the tex_math_single_backslash extension, the [ and ] will get dropped in the HTML output:

❯ pandoc -f markdown -t markdown foo.md | pandoc -f markdown+tex_math_single_backslash -t html
<p>This is a footnote<span class="math display">1</span>.</p>

I use -f markdown -t markdown to clean up some poorly-formatted markdown generated outside of my control.

I was hoping that I could change this to -f markdown -t markdown+tex_math_single_backslash to instruct pandoc that the markdown output should be compatible with \[ meaning "math", but it appears that's not how it works, and in fact no matter whether that extension is passed with the -t flag, the output shows \[1\].

❯ pandoc -f markdown -t markdown+tex_math_single_backslash foo.md
This is a footnote\[1\].

❯ pandoc -f markdown+tex_math_single_backslash -t markdown+tex_math_single_backslash foo.md
This is a footnote\[1\].

Pandoc version?
macOS 12.6 Monterey

❯ pandoc --version
pandoc 2.19.2
Compiled with pandoc-types 1.22.2.1, texmath 0.12.5.2, skylighting 0.13,
citeproc 0.8.0.1, ipynb 0.2, hslua 2.2.1
Scripting engine: Lua 5.4
User data directory: /Users/jez/.local/share/pandoc
Copyright (C) 2006-2022 John MacFarlane. Web:  https://pandoc.org
This is free software; see the source for copying conditions. There is no
warranty, not even for merchantability or fitness for a particular purpose.

Apologies if this is expected, or if this is a duplicate of an existing issue (I searched, but nothing looked the same).

As always, thanks for this simple and excellent piece of software.

[jgm]

Well, it's tough to know how to deal with this. tex_math_single_backslash is a really misguided extension -- I only supported it because some Markdown dialects do, but they should have known better because \[ and \] already have very clearly defined meanings in Markdown.

So of course it's a mess when you turn on this extension. \[x\] could mean either math or literal [x].

In the case you note, we could have left off the backslash. But that's not always the case. Consider

\[1\]

[1]: url

Here we can't leave off the backslash, or we'd get a link, so there's no way to avoid the ambiguity introduced by tex_math_single_backslash. I'd recommend against using it if possible.

[jez]

Thanks for the quick reply. Is the recommended extension to use the double backslash one? Or to just use dollar signs for math?

[jgm]

I prefer dollar signs, because that doesn't conflict with existing markdown syntax.
That's the pandoc default, too. But of course it depends what kind of markdown dialect you're trying to generate.

[jez]

Thanks. I was hesitant to use $ because the editor syntax highlighter I use for highlighting pandoc markdown files isn't as smart as pandoc itself w.r.t. sometimes treating $ as a literal dollar sign vs as the start of a math region:

(You can see: it's highlighting in yellow the regions it thinks are math, which it naively thinks is "any region starting with $, no matter what")

I suppose I can tolerate the bad syntax highlighting (it's better than incorrectly rendered documents) so I'll do that in the mean time. Maybe I can even make my editor's highlighter better.

If you still consider this a bug feel free to keep open. If you would rather close the ticket feel free. Thanks again for the quick responses.