Incomplete RawBlocks of some environments inside LaTeX tables

[domhal]

Running Pandoc 3.1.12.1 on Windows. For the following source file

\documentclass{article}

\begin{document}

\begin{tabular}{l}
\begin{theorem}
Cats
\end{theorem}
\end{tabular}

\end{document}

running

pandoc --from=latex+raw_tex --to=native test.tex

gives

[ Table
    ( "" , [] , [] )
    (Caption Nothing [])
    [ ( AlignLeft , ColWidthDefault ) ]
    (TableHead ( "" , [] , [] ) [])
    [ TableBody
        ( "" , [] , [] )
        (RowHeadColumns 0)
        []
        [ Row
            ( "" , [] , [] )
            [ Cell
                ( "" , [] , [] )
                AlignDefault
                (RowSpan 1)
                (ColSpan 1)
                [ RawBlock (Format "latex") "\\begin{theorem}" ]
            ]
        ]
    ]
    (TableFoot ( "" , [] , [] ) [])
]

which is unexpected as the RawBlock contains neither the contents nor the end of the theorem environment. The same problem occurs for environments such as lemma and tikzpicture but not proof or figure.

[jgm]

This is a subtle one. From what I can see, it looks like a bug in withRaw.
The parser is parsing the whole environment, but withRaw is not getting all of the raw tokens that go with it. Most likely, it's a problem running withRaw in a parseFromToks environment.

[jgm]

Can confirm that withRaw is returning an empty list of tokens.

[jgm]

Aha. parseFromToks has this line:

  result <- disablingWithRaw parser

which is undoubtedly there for a reason. (What was it?)

[jgm]

Here's one of the tests that fails if we remove this disablingWithRaw:

         ------------------------------------------------------------------------
          --- testsuite.native
          +++ /Users/jgm/src/pandoc/dist-newstyle/build/aarch64-osx/ghc-9.6.3/pandoc-3.1.12.2/t/test-pandoc/noopt/build/test-pandoc/test-pandoc --emulate --quiet testsuite.txt -r markdown -w native -s
          +1506       "\\begin{tabular}{|l|l|}\\hline\nAnimal & Number Animal  Number Animal  Number Animal  Number \\\\ \\hline\nDog    & 2      Dog     2      Dog     2      Dog     2      \\\\\nCat    & 1      Cat     1      Cat     1      Cat     1      \\\\ \\hline\n\\end{tabular}"
          -1506       "\\begin{tabular}{|l|l|}\\hline\nAnimal & Number \\\\ \\hline\nDog    & 2      \\\\\nCat    & 1      \\\\ \\hline\n\\end{tabular}"
          ------------------------------------------------------------------------
          Use -p '/markdown.reader.basic/' to rerun this test only.

[jgm]

So we can either

(a) figure out how to do the table cell parsing without parseFromToks, or
(b) figure out how to make withRaw work within parseFromToks

[jgm]

Ah yes, we needed parseFromToks because we sometimes have to add prefix or suffix tokens to each cell.

[jgm]

In satisfyTok we have

    updateState $ \st ->
      if sEnableWithRaw st
         then
           let !newraws = IntMap.map (res:) $! sRawTokens st
            in  st{ sRawTokens = newraws }
         else st

So all of the token streams in the map are updated. Perhaps parseFromToks should simply temporarily reset (and then restore) sRawTokens?