parallel render()'s with different output_file's overwrite each other

[petrelharp]

I commonly write .Rmd templates, that then get rendered to create a report with different input parameters. However, if more than one such job is being rendered at the same time, since the name of the intermediate .md file is based on the input, all jobs write to the same intermediate.

The result is that I (silently) get wrong reports.

Here is a MWE.

test.Rmd:

I am writing to `r outname`.html.

test.sh:

#!/bin/bash
R --vanilla -e "outname=\"${1}\";require(rmarkdown);render(\"test.Rmd\",output_file=\"${1}.html\")"

Run two at the same time:

for x in one two; do (./test.sh $x &); done
grep "writing to" *.html
# one.html:<p>I am writing to two.html.</p>
# two.html:<p>I am writing to two.html.</p>

Non-workarounds:

• "specify intermediates_dir", but if there's images in the document this fails (render with intermediates_dir fails with images #500)
• "first render to md_document, then render to html", but the same problem occurs because render actually outputs to input.utf8.md, with no chance to change this

A fix?

It seems better to use tempfile(), e.g.

 knit_output <- intermediates_loc(tempfile(tmpdir=".",fileext=".knit.md"))

... at least when output_file is specified.

[petrelharp]

Additional non-workaround: knit_root_dir. Revised test.sh:

#!/bin/bash
R --vanilla -e "outname=\"${1}\";require(rmarkdown);render(\"test.Rmd\",output_file=\"${1}.html\",knit_root_dir=dirname(\"${1}\"))"

and running has the same problem.

for x in a/one b/two; do (mkdir $(dirname $x); ./test.sh $x &); done
grep "writing to" */*.html
# a/one.html:<p>I am writing to b/two.html.</p>
# b/two.html:<p>I am writing to b/two.html.</p>

[Pablo-Leon]

Hitting this problem too!

how do I up-vote the issue?

[darth-donut]

Is there a temporary solution to this problem?

[petrelharp]

I use my own package to avoid this problem: https://github.com/petrelharp/templater

If it's useful but you hit snags, please let me know over there.

[darth-donut]

I'll give it a go and let you know if I hit a snag, cheers!

[JTC-R]

I also have this issue; I run a parallel report generation that consists of over 1,000 reports. Often times reports will be "mis-matched" where the title of the report will contain the wrong information from a different report.

Setting the intermediates_dir param in an effort to over-come this fails as the doing so provokes its own bugs.