pandoc: openBinaryFile: does not exist (No such file or directory) when using render() in parallel #1632
Comments
I can confirm this is happening on local computer using local drives, not network ones. |
|
So I have examined a little bit how is actually I have found that this file is probably somehow used for navbars, but more important is this function: called in So what is actually happening? After I can confirm this dirty solution works (put this after Would be great if developers add something like Full modified |
|
We're using a basic Pandoc script in bash to merge multiple files:
Is it possible to add a function to prevent clearing the temp files which causes the "openBinaryFile: does not exist (No such file or directory)" error in this medium? |
|
@JayMMTL I am not sure how is this connected to rendering Rmds from within R. If this is the case, you can use the snippet I have provided to replace the |
|
@gorgitko Your solution worked perfectly for me thanks. |
|
Thanks @gorgitko for explaining the root cause. I also tried the set off a flag mentioned in several posts: SessionInfo: Any ideas?? |
|
@agilebean I am not sure if |
|
I continue to get random messages about files not existing. |
|
Some version of an insistently retrying function may work (although clearly not ideal, it seems to be helping me avoid this somewhat random error): then call |
|
Any suggestions on how to fix this? I am running it in R version 4.04 pandoc.exe: \: openBinaryFile: does not exist (No such file or directory) |
|
@ktd2001 Can you share the file with the issue so we can try reproduce ? With any other element that could help us understand ? |
|
Hi Chris,
Thank you for responding so quickly.
I am using a Window 10
Here is the file and dataset that is located in the same folder as the NB.
I could not find which Pandoc version I use ? rmarkdown::pandoc_version().
How would I find this? I did a search in the files but it continues to
search.
My best,
Keiana
…On Wed, Mar 17, 2021 at 1:15 PM Christophe Dervieux < ***@***.***> wrote:
@ktd2001 <https://github.com/ktd2001>
Which OS are you on ?
Which Pandoc version do you use ? rmarkdown::pandoc_version()
Where are you files located ? On a network drive ?
Can you share the file with the issue so we can try reproduce ? With any
other element that could help us understand ?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#1632 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AKK2EYJNTCQMPPLQDNY3NITTEDPQTANCNFSM4IRBJKFA>
.
|
|
@cderv I am just curious if anyone at RStudio has looked into the problem I proposed in the #1632 (comment) TLDR: Allow users to specify whether Anyway, I think each rendered Rmd should get its own directory in Thank you for looking at this 🙂 |
|
I cleared the temporary files and still getting same error message:
pandoc.exe: \\: openBinaryFile: does not exist (No such file or directory)
Warning: Error in : pandoc document conversion failed with error 1
128: stop
127: pandoc_convert
126: convert
125: render
124: discover_rmd_resources
123: find_external_resources
122: copy_render_intermediates
121: output_format$intermediates_generator
120: <Anonymous>
115: <reactive>
99: doc
98: renderUI
97: func
84: renderFunc
83: output$__reactivedoc__
3: <Anonymous>
1: rmarkdown::run
…On Thu, Mar 18, 2021 at 3:57 AM Jiri Novotny ***@***.***> wrote:
@cderv <https://github.com/cderv> I am just curious if anyone at RStudio
has looked into the problem I proposed in the #1632 (comment)
<#1632 (comment)>
TLDR: Allow users to specify whether clean_tmpfiles() will be run after
render() has finished. Possibly allow users to run it manually.
*Anyway, I think each rendered Rmd should get its own directory in
tempdir(); that would basically avoid this problem.*
Thank you for looking at this 🙂
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#1632 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AKK2EYIOCFTKTR4SFFH53K3TEGW5PANCNFSM4IRBJKFA>
.
|
|
@gorgitko I tried your solution, but I still get the error intermittently. Note that this works correctly in my local RStudio session, but not always in a Docker container for Continuous Integration on GitLab with Note that I created a R file called to run in parallel. The R file looks like this: knit_file_system <- function(x) {
runR <- tempfile(fileext = "run.R")
cat(
paste0(
paste0(".libPaths(c(\"", paste(.libPaths(), collapse = "\", \""), "\"));"),
# Avoid https://github.com/rstudio/rmarkdown/issues/1632#issuecomment-545824711
'
library(rmarkdown)
clean_tmpfiles_mod <- function() {
message("Calling clean_tmpfiles_mod()")
}
assignInNamespace("clean_tmpfiles", clean_tmpfiles_mod, ns = "rmarkdown")
',
# 'rmarkdown::render(',
'render(',
gsub("\\", "\\\\", .x, fixed = TRUE),
'envir = new.env(), encoding = "UTF-8"', #parent = baseenv()
')'
)
, file = runR)
system(
paste(normalizePath(file.path(Sys.getenv("R_HOME"), "bin", "Rscript"), mustWork = FALSE), runR)
)
}Then call it with {future} library(future)
future::plan(future::multicore)
all_my_rmds <- list.files(pattern = "[.]Rmd")
future_imap(all_my_rmds,
~try(knit_file_system(.x)),
.progress = TRUE) |
|
@statnmap does it happens also if you explicitly run each If the issue is really with each render having its own I am thinking more and more of providing a way to run a |
|
I use the |
This is definitely the issue (i.e. common |
|
Now I know why I still have this problem, this is because I knit twice the same file in parallel, such that one process deletes the utf8.md file while the other process is trying to access it. |
|
@statnmap That's why I am using an unique intermediates_dir <- glue("{i}_intermediates_dir")
render(..., intermediates_dir = intermediates_dir)
# Cleaning.
system(glue("rm -r {intermediates_dir}"))It's also possible to use completely random intermediate dirs, e.g.: paste0("intermediates_", stringi::stri_rand_strings(1, 10)) |
|
FYI this issue caused a major issue with some work products - rendering a bunch of rmarkdown to latex to pandoc in parallel caused some of the files to have a file name that didn't match the file content. Until this is fixed, there should be a warning or error if it is run in parallel. Using @gorgitko's method to create unique intermediate directories solved the issue, so the fix should implement this. |
|
I was getting similar errors, but had nothing to do with running in parallel. Instead, the While containerizing a report, I hit the following error: pandoc: /usr/local/lib/R/site-library/fontawesome/fontawesome/css/../webfonts/fa-v4compatibility.woff2: withBinaryFile: does not exist (No such file or directory)
Error: pandoc document conversion failed with error 1I then checked the directory it was complaining about: > list.files("/usr/local/lib/R/site-library/fontawesome/fontawesome/css/../webfonts/")
[1] "fa-brands-400.ttf" "fa-brands-400.woff" "fa-regular-400.ttf"
[4] "fa-regular-400.woff" "fa-solid-900.ttf" "fa-solid-900.woff" As you can see, the file |
|
@gorgitko I'm trying to use your solution, but the |
Apologies, actually a much dumber problem -- on mac, the file name was displayed as "x.bib" but the real name was *x.bib.txt", but that could only be seen when accessing meta-information on the file (Command I). Leaving here in case somebody else is confused into the same dumb problem. |
|
In the vain of dumb problems, omitting the -o before specifying the output file will cause the same error. This one caught me out (read the docs people!) |
|
I confirm that this bug is still present with makeForkCluster (makePSOCKcluster works fine) with the following configuration: Apple M3 Pro The bug was hard to pinpoint, but the "dirty" solution proposed by gorgitko, copied below, still works: |
By filing an issue to this repo, I promise that
xfun::session_info('rmarkdown'). I have upgraded all my packages to their latest versions (e.g., R, RStudio, and R packages), and also tried the development version:remotes::install_github('rstudio/rmarkdown').I understand that my issue may be closed if I don't fulfill my promises.
When I am using
rmarkdown::render()inBiocParallel::bplapply(), Pandoc throws this error:pandoc: /tmp/RtmpW06rTD/rmarkdown-str3bc26dd971b5.html: openBinaryFile: does not exist (No such file or directory). I am using Pandoc version 2.7.3 and development version of rmarkdown. Everything works fine when I useBPPARAM = SerialParam()inbplapply(i.e. it will disable parallel processing).to_render.Rmd:render.R:Output from
render.R:Pandoc and session info:
The text was updated successfully, but these errors were encountered: