Converting from LaTeX to CommonMark leaves in \maketitle, even without raw_tex

[quasicomputational]

The extra puzzling thing is that the Markdown writer doesn't do this:

$ cat test.tex 
\documentclass{article}

\title{Foo.}

\begin{document}

\maketitle

Hello.

\end{document}
$ pandoc -f latex-raw_tex -t commonmark-raw_tex test.tex
\maketitle

Hello.

$ pandoc -f latex -t markdown_strict-raw_tex test.tex
Hello.
$ pandoc --version
pandoc 2.1.3

[quasicomputational]

This might not be a CommonMark-specific issue, since the RST writer does it too:

$ pandoc -f latex-raw_tex -t rst-raw_tex test.tex 
.. raw:: latex

   \maketitle

Hello.

[quasicomputational]

There are two bugs going on here, I think, and they're both necessary to observe this bug.

First, the LaTeX reader is emitting raw blocks even without raw_tex:

$  printf '\\foo{}\n\nhello \\foo{} bar' | pandoc -f latex-raw_tex -t native
[RawBlock (Format "latex") "\\foo{}"
,Para [Str "hello",Space,Str "bar"]]

As can be seen, the raw inlines are correctly omitted.

Second, the CommonMark and RST writers aren't ignoring the raw blocks. Here's the relevant code from Text.Pandoc.Writers.CommonMark:

blockToNodes opts (RawBlock fmt xs) ns
  | fmt == Format "html" && isEnabled Ext_raw_html opts
              = return (node (HTML_BLOCK (T.pack xs)) [] : ns)
  | fmt == Format "latex" || fmt == Format "tex" && isEnabled Ext_raw_tex opts
              = return (node (CUSTOM_BLOCK (T.pack xs) T.empty) [] : ns)
  | otherwise = return ns

That's got an operator precedence bug in its logic. The corresponding code for RawInline is correctly bracketted. I'll make up a PR for that.

[quasicomputational]

Upon review, the RST raw logic is very different to the Markdown/CommonMark raw logic, defaulting to inclusion. That's probably a separate bug, and there's also the LaTeX reader bug, so merging the PR won't suffice to fully clear up the bugs here.

[quasicomputational]

I had a look at the manual to see whether the RST writer is actually working as specified. The raw_tex extension is described twice: in one place it says that it only affects markdown, latex, org and textile (which is wrong, it should include commonmark now); in another place, it says "Inline LaTeX is ignored in output formats other than Markdown, LaTeX, Emacs Org mode, and ConTeXt," which suggests that the RST reader is buggy in always allowing it through.

The LaTeX reader is definitely buggy in how it's emitting raw blocks even with raw_tex disabled.

But this isn't ruining my day nor even mildly inconveniencing me, and you can drop the raw elements with filters as a workaround if it matters to you, so I'm not going to work on it for the time being.

[jgm]

quasicomputational <notifications@github.com> writes:

I had a look at the manual to see whether the RST writer is actually working as specified. The `raw_tex` extension is described twice: in one place it says that it only affects markdown, latex, org and textile (which is wrong, it should include commonmark now); in another place, it says "Inline LaTeX is ignored in output formats other than Markdown, LaTeX, Emacs Org mode, and ConTeXt," which suggests that the RST reader is buggy in always allowing it through.

I think the documentation here just hasn't been kept up to date properly as things have changed. We should have changed it when we supported `raw` blocks in RST.

The LaTeX reader is definitely buggy in how it's emitting raw blocks even with `raw_tex` disabled.

Yes, agreed.

[jgm]

I tried making inclusion of raw latex blocks conditional on raw_tex, and got the following test failures:

Progress 1/2: pandoc-2.2.2.1pandoc tests
  Command:
    macros.md
      #3:                                                          FAIL (0.05s)
        test/Tests/Command.hs:54:
        
        ------------------------------------------------------------------------
        --- expected
        +++ pandoc -f latex -t latex
        +   1 
        -   1 \b
        ------------------------------------------------------------------------
    4527.md
      #2:                                                          FAIL (0.05s)
        test/Tests/Command.hs:54:
        
        ------------------------------------------------------------------------
        --- expected
        +++ pandoc -f latex -t commonmark+raw_tex
        -   1 \someunknowncommand
        -   2 
        ------------------------------------------------------------------------
    3577.md
      #1:                                                          FAIL (0.08s)
        test/Tests/Command.hs:54:
        
        ------------------------------------------------------------------------
        --- expected
        +++ pandoc -f latex -t html5 --quiet
        -   1 
        -   5 
        ------------------------------------------------------------------------
    3494.md
      #1:                                                          FAIL (0.09s)
        test/Tests/Command.hs:54:
        
        ------------------------------------------------------------------------
        --- expected
        +++ pandoc -f latex --quiet
        +  14 <td style="text-align: right;"><em>blah</em></td>
        -  14 <td style="text-align: right;"><p><em>blah</em></p></td>
        ------------------------------------------------------------------------
    3983.md
      #1:                                                          FAIL (0.04s)
        test/Tests/Command.hs:54:
        
        ------------------------------------------------------------------------
        --- expected
        +++ pandoc -f latex -t native
        +   1 []
        -   1 [RawBlock (Format "latex") "\\makeatletter"
        -   2 ,RawBlock (Format "latex") "\\makeatother"]
        ------------------------------------------------------------------------
      #2:                                                          FAIL (0.05s)
        test/Tests/Command.hs:54:
        
        ------------------------------------------------------------------------
        --- expected
        +++ pandoc -f latex -t native
        +   1 []
        -   1 [RawBlock (Format "latex") "\\makeatletter"
        -   2 ,RawBlock (Format "latex") "\\makeatother"]
        ------------------------------------------------------------------------
      #3:                                                          FAIL (0.04s)
        test/Tests/Command.hs:54:
        
        ------------------------------------------------------------------------
        --- expected
        +++ pandoc -f latex -t native
        +   1 []
        -   1 [RawBlock (Format "latex") "\\bambar"]
        ------------------------------------------------------------------------

7 out of 1709 tests failed (19.34s)

We should think these through to make sure the change is not going to cause problems.

[jgm]

Here's the patch

diff --git a/src/Text/Pandoc/Readers/LaTeX.hs b/src/Text/Pandoc/Readers/LaTeX.hs
index 1dd31d402..18a5bb550 100644
--- a/src/Text/Pandoc/Readers/LaTeX.hs
+++ b/src/Text/Pandoc/Readers/LaTeX.hs
@@ -2271,7 +2271,9 @@ blockCommand = try $ do
   let names = ordNub [name', name]
   let rawDefiniteBlock = do
         guard $ isBlockCommand name
-        rawBlock "latex" <$> getRawCommand name (txt <> star)
+        rawcontents <- getRawCommand name (txt <> star)
+        (guardEnabled Ext_raw_tex >> return (rawBlock "latex" rawcontents))
+          <|> return mempty
   -- heuristic:  if it could be either block or inline, we
   -- treat it if block if we have a sequence of block
   -- commands followed by a newline.  But we stop if we
@@ -2286,7 +2288,9 @@ blockCommand = try $ do
         curr <- rawBlock "latex" <$> getRawCommand name (txt <> star)
         rest <- many $ notFollowedBy startCommand *> blockCommand
         lookAhead $ blankline <|> startCommand
-        return $ curr <> mconcat rest
+        let contents = curr <> mconcat rest
+        (guardEnabled Ext_raw_tex >> return contents)
+          <|> return mempty
   let raw = rawDefiniteBlock <|> rawMaybeBlock
   lookupListDefault raw names blockCommands