(Regression) pandoc leaves temporary directory behind

[sboukortt]

As mentioned in #2288 (comment), pandoc 3.1.11.1 on Windows fails to clean up the tex2pdf temporary directory it creates if a document that includes PDF graphics is converted to PDF. (I haven’t tried whether other image formats trigger the bug.) I have offered to try and bisect the issue and am in the process of doing so. I can already confirm that 3.1.2 did not exhibit the problem (it cleaned up all temporary files, as expected).

I’ll update this once I find the revision that introduced the regression.

I also plan to test whether the issue still reproduces on HEAD, and if it doesn’t, I’ll likely “reverse-bisect” the revision that fixed it, if only for my own curiosity.

[sboukortt]

It seems that HEAD (e43ab9e) leaves more files behind. (input.tex, input.log, input.pdf; in addition to the input.aux that 3.1.11.1 also left.)

[jgm]

Can anybody else running pandoc on Windows reproduce this? It seems to be Windows-specific.

[sboukortt]

Interestingly, my attempt at creating a minimal reproducible example is failing. Not sure what it is about my not-minimal example that is causing the issue to occur… But maybe finding the culprit revision will shed some light on it.

[sboukortt]

It seems that HEAD (e43ab9e) leaves more files behind. (input.tex, input.log, input.pdf; in addition to the input.aux that 3.1.11.1 also left.)

A bisection between 3.1.11.1 and HEAD suggests that this increase was introduced by 2dd98b9, which doesn’t seem so informative at first glance.

(I did that bisection first because I expected it to be faster.)

My next step is going to be the bisection between 3.1.2 and 3.1.11.1 (likely tomorrow as it’s getting a bit late here).

[jgm]

One thing to check is how deterministic your results are. Are the same files always left behind? Are files always left behind with that version, or just sometimes? Does 3.1.2 never leave them behind?

[sboukortt]

It seems quite deterministic as far as I can tell, and the first bisection did provide a clue after all, at least as far as creating a repro case is concerned: it seems that adding a section name does the trick, and adding it actually makes the image unnecessary – even without it, pandoc still leaves input.aux behind. (But the image makes for another one, just to be sure.)

Here is therefore a repro case:

tmpandoc.zip

$ cd tmpandoc
$ pandoc document.md -o document.pdf

[jgm]

Also: what --pdf-engine are you using (or just the default)? OK: the default.

[jgm]

My hunch is that this may have to do with lazy IO, and I note that there are a couple of readFileLazy's in the runTeXProgram code. I would expect that withSystemTempDir would still clean up, but maybe there is a bug in the Windows implementation? We could try replacing all the lazy IO with strict IO in this context -- I remember doing that before for another issue on Windows.

[jgm]

If you want to try it, you could, in Text.Pandoc.PDF.hs, remove the readFileLazy from the imports from Text.Pandoc.Class and add this function definition:

readFileLazy :: (PandocMonad m, MonadIO m) => FilePath -> m BL.ByteString
readFileLazy fp = BL.fromStrict <$> readFileStrict fp

Then recompile and see if the problem is still there.

[jgm]

Commit bd8e317
adds a readFileLazy.
Dec. 15 2023 pandoc 3.1.11

[jgm]

OK, try with current HEAD, which incorporates the idea I had above.

[sboukortt]

That does seem to fix it, thanks!

git bisect found 3c17869 as the proximate cause, but it seems plausible that it only surfaced the issue, not caused it.

[jgm]

Great, I'm glad to have found this. I remember running into a similar issue with lazy IO on Windows over a decade ago.