[help] Extreme memory load from tar_quarto_rep() on an qmd with tar_load(df, branches = params$branch)

[asadow]

Help

• I understand and agree to https://books.ropensci.org/targets/help.html.

Description

In my situation, df is a pretty big target (1000 ggplots) that has been branched off. Without branching it would be ~64MB.

I've got a decent system; I was able to create df using distributed computing very fast (~15 secs). So here is where I'm surprised and I can't figure out what's going on.

I am creating separate renders for each of the branches. A single worker is taking up 10GB of memory. So I can't use multiple workers. I've tried all the options in the Performance chapter. Am I missing something?

[wlandau]

It would help to look at a small reprex.

[asadow]

Ok, great. I'm stuck however on how to name the files based on the employee. Without output_file, a single calendar.html is made and is rewritten over. But the below code will give Object active_employees not found.

tar_target(
    reports_calendars, 
    command = quarto::quarto_render(
      here::here("calendar.Rmd"),
      output_file = paste0(
      1:length(active_employee), ".html"
      )
      ), 
    pattern = map(calendars)
    )

[wlandau]

You would need something from the current branch of calendars. When you map over calendars, the mention of calendars inside your command is the current branch. If calendars is a data frame with a column to indicate the output file, it would look something like this:

tar_target(
  reports_calendars, 
  command = quarto::quarto_render(
    here::here("calendar.Rmd"),
    output_file = calendars$output_file[1]
  ), 
  pattern = map(calendars)
)

Each branches output_file should be different because calendars$output_file[1] uses only the subset of calendars corresponding to the current branch of reports_calendars.

[asadow]

Ah perfect. But how would you load in the branch inside the Rmd? I feel like I need a way to refer to the pattern, like tar_load(calendars, branches = pattern).

I've tried tar_load(calendars) with and without params$branch, and omitting tar_load() completely.

[wlandau]

If you really need to use Quarto, then you would have to make a dynamic file target to save each section of calendars to a file, then map over those files in report_calendars. But if you can switch to R Markdown, the report is rendered in the same process that runs the target, so then you could just reference the object in the R code of the report. Either way, tar_load() and tar_read() are not meant for this scenario.

[asadow]

Thank you dearly. As you implied, it works when I omit tar_load() from the report and use rmarkdown::render() instead of quarto::quarto_render() in _targets.R. The issue was in using Quarto.

I tend to avoid Quarto in tandem with targets. For one, I always get quarto.exe failed when the Rmd/qmd file is not in the root directory. And this adds another reason.

[asadow]

@wlandau I spoke too soon: It works for a handful of reports, but suddenly one report errors with Last error: pandoc document conversion failed with error 1.

Repo reprex here.

I can't reproduce the error; manually running reports works fine. I tried restarting and running targets::tar_make(callr_function = NULL).

[wlandau]

I was able to reproduce the error you saw, and it indeed does go away when I render the report locally or from within the built-in targets interactive debugging system. Then I tried tar_make(use_crew = FALSE), and everything worked.

So I think the pandoc error happens because two parallel processes from crew are trying to write the same intermediate files at the same time. Even though your HTML output files are all unique, pandoc may still write temporary local intermediate files based on the name of the common Rmd file. If the crew workers fight over those intermediate files, the results are unpredictable.

So for each rmarkdown::render() call, I recommend making a copy of Rmd/calendar.Rmd to a temporary file (c.f. tempfile() and tempdir()), rendering it from the temporary location, and then moving the HTML file to the output folder you want.

As a last resort, you could alternatively set deployment = "main" in the report targets so they don't run on crew workers.

[asadow]

I did what you recommended and it works fantastically. Thanks Will!