tar_quarto() ignores output_dir in _quarto.yml when passed an individual file

[petrbouchal]

When running tar_quarto() on a single qmd file in a (website) project with e.g. output_dir: docs param in _quarto.yml, an error is returned:

Error : _paths_ missing files: report.html
✖ Problem with the pipeline.
Error:
! problem with the pipeline.

Reprex:

  targets::tar_dir({  # tar_dir() runs code from a temporary directory.
    # Unparameterized Quarto document:
    lines <- c(
      "---",
      "title: report.qmd source file",
      "output_format: html",
      "---",
      "Assume these lines are in report.qmd.",
      "```{r}",
      "targets::tar_read(data)",
      "```"
    )
    writeLines(lines, "report.qmd")

    # Add _quarto.yml
    yml <- c(
      "project:
        type: website
        output-dir: docs"
    )
    writeLines(yml, "_quarto.yml")

    # Include the report in a pipeline as follows.
    targets::tar_script({
      library(tarchetypes)
      list(
        tar_target(data, data.frame(x = seq_len(26), y = letters)),
        tar_quarto(report, path = "report.qmd")
      )
    }, ask = FALSE)
    targets::tar_make()
  })

The output (HTML) file is rendered correctly in the docs directory, but targets expects it to be in the working directory, since it does not seem to be inferring output directory from _quarto.yml and quarto::quarto_render() does not return the output path.

Running files in quarto projects as individual targets is useful, because running the tar_quarto()factory on the whole project currently produces a single target, where all input *.qmd files depend on all targets used in all of them, triggering a complete rebuild of all files whenever a single target used in one of them changes.

This might be easier to handle if quarto_render() returned the path of its output file - which would be sensible behaviour for a render function, similar to what rmarkdown::render() does. Happy to file a request there if needed.

An alternative solution might be for the target factory to produce individual targets/branches for each file when the path argument in tar_quarto() is left unset.

[wlandau]

tarchetypes uses quarto::quarto_inspect() to figure out what the output file should be. In your case, quarto::quarto_inspect("report.qmd")$formats$html$pandoc[["output-file"]] is "report.html", which disagrees with the output generated by quarto::quarto_render(). I believe this is a Quarto issue and is related to my comments in quarto-dev/quarto-r#14, particularly quarto-dev/quarto-r#14 (comment).

Running files in quarto projects as individual targets is useful, because running the tar_quarto()factory on the whole project currently produces a single target, where all input *.qmd files depend on all targets used in all of them, triggering a complete rebuild of all files whenever a single target used in one of them changes.

That could be useful if some reports do their own heavy computation. However, the recommended pattern is to do the hard work in upstream targets and just call tar_read() in the reports so the Quarto part runs quickly. In addition, every tar_quarto() call runs quarto_inspect() to tarchetypes can learn what files Quarto will generate, and quarto_inspect() has noticeable overhead: quarto-dev/quarto-cli#1266. With the current version of Quarto, many calls to tar_quarto() for individual reports could add up.

[gorkang]

tarchetypes uses quarto::quarto_inspect() to figure out what the output file should be. In your case, quarto::quarto_inspect("report.qmd")$formats$html$pandoc[["output-file"]] is "report.html", which disagrees with the output generated by quarto::quarto_render(). I believe this is a Quarto issue and is related to my comments in quarto-dev/quarto-r#14, particularly quarto-dev/quarto-r#14 (comment).

Having this same error when using _quarto.yml to change the output-dir of the quarto documents, which, as far as I can see, it makes impossible to choose where the quarto outputs will be stored.

Not sure if I am missing something, but there seems to be an easy solution to this. tarchetypes could use quarto::quarto_inspect("report.qmd")$project$config$project[["output-dir"]] to figure out the location of the output file.

[gorkang]

Sorry, I can see tarchetypes:::tar_quarto_files() actually gets the output-dir through tarchetypes:::tar_quarto_files_project().

Then, I am not sure why the expected output is still report.html, and not docs/report.html.

[wlandau]

_quarto.yml is just for projects, right? I am not sure it applies to standalone qmd files. So you could either supply path = getwd() and use _quarto.yml to set the output directory, or you could set path = "path/to/file/report.qmd" and forgo _quarto.yml.

[gorkang]

The specific issue is that the qmd files are in a folder 'Qmd', but I want the outputs (rendered html or pdf) in a different folder (e.g. 'outputs').

My impression was that you can only do that using the _quarto.yml... but I probably missed something in the docs.

In any case, when there is a _quarto.yml with the 'output-dir' parameter, targets will fail with a cryptic error.

[wlandau]

Do you have a reprex?

[gorkang]

Sorry Will, the reprex is in the first message of this discussion. I was having the same issue in a more complex project, and arrived here :)

Here a copy of the reprex:

targets::tar_dir({  # tar_dir() runs code from a temporary directory.
  # Unparameterized Quarto document:
  lines <- c(
    "---",
    "title: report.qmd source file",
    "output_format: html",
    "---",
    "Assume these lines are in report.qmd.",
    "```{r}",
    "targets::tar_read(data)",
    "```"
  )
  writeLines(lines, "report.qmd")
  
  # Add _quarto.yml
  yml <- c(
    "project:
        type: website
        output-dir: docs"
  )
  writeLines(yml, "_quarto.yml")
  
  # Include the report in a pipeline as follows.
  targets::tar_script({
    library(tarchetypes)
    list(
      tar_target(data, data.frame(x = seq_len(26), y = letters)),
      tar_quarto(report, path = "report.qmd")
    )
  }, ask = FALSE)
  targets::tar_make()
})
#> • start target data
#> • built target data [0.001 seconds]
#> • start target report
#> ✖ error target report
#> • end pipeline [2.17 seconds]
#> Error:
#> ! Error running targets::tar_make()
#>   Error messages: targets::tar_meta(fields = error, complete_only = TRUE)
#>   Debugging guide: https://books.ropensci.org/targets/debugging.html
#>   How to ask for help: https://books.ropensci.org/targets/help.html
#>   Last error: _paths_ missing files: report.html
#> Backtrace:
#>     ▆
#>  1. └─targets::tar_make()
#>  2.   └─targets:::callr_outer(...)
#>  3.     ├─targets:::if_any(...)
#>  4.     └─targets:::callr_error(condition = out, fun = fun)
#>  5.       └─targets::tar_throw_run(message, class = class(condition))
#>  6.         └─targets::tar_error(...)
#>  7.           └─rlang::abort(message = message, class = class, call = tar_empty_envir)

^{Created on 2023-03-21 with reprex v2.0.2}

Session info

sessioninfo::session_info()
#> ─ Session info ───────────────────────────────────────────────────────────────
#>  setting  value
#>  version  R version 4.2.3 (2023-03-15)
#>  os       Ubuntu 22.04.2 LTS
#>  system   x86_64, linux-gnu
#>  ui       X11
#>  language en_US:en
#>  collate  en_US.UTF-8
#>  ctype    en_US.UTF-8
#>  tz       Atlantic/Canary
#>  date     2023-03-21
#>  pandoc   2.19.2 @ /usr/lib/rstudio/resources/app/bin/quarto/bin/tools/ (via rmarkdown)
#> 
#> ─ Packages ───────────────────────────────────────────────────────────────────
#>  package     * version date (UTC) lib source
#>  backports     1.4.1   2021-12-13 [1] CRAN (R 4.2.0)
#>  base64url     1.4     2018-05-14 [1] CRAN (R 4.2.0)
#>  callr         3.7.3   2022-11-02 [1] RSPM (R 4.2.0)
#>  cli           3.6.0   2023-01-09 [1] RSPM (R 4.2.0)
#>  codetools     0.2-19  2023-02-01 [4] CRAN (R 4.2.2)
#>  data.table    1.14.8  2023-02-17 [1] RSPM (R 4.2.0)
#>  digest        0.6.31  2022-12-11 [1] RSPM (R 4.2.0)
#>  evaluate      0.20    2023-01-17 [1] RSPM (R 4.2.0)
#>  fansi         1.0.4   2023-01-22 [1] RSPM (R 4.2.0)
#>  fastmap       1.1.1   2023-02-24 [1] RSPM (R 4.2.0)
#>  fs            1.6.1   2023-02-06 [1] RSPM (R 4.2.0)
#>  glue          1.6.2   2022-02-24 [1] CRAN (R 4.2.0)
#>  htmltools     0.5.4   2022-12-07 [1] RSPM (R 4.2.0)
#>  igraph        1.4.1   2023-02-24 [1] RSPM (R 4.2.0)
#>  knitr         1.42    2023-01-25 [1] CRAN (R 4.2.1)
#>  lifecycle     1.0.3   2022-10-07 [1] RSPM (R 4.2.0)
#>  magrittr      2.0.3   2022-03-30 [1] CRAN (R 4.2.0)
#>  pillar        1.8.1   2022-08-19 [1] RSPM (R 4.2.0)
#>  pkgconfig     2.0.3   2019-09-22 [1] CRAN (R 4.2.0)
#>  processx      3.8.0   2022-10-26 [1] RSPM (R 4.2.0)
#>  ps            1.7.2   2022-10-26 [1] RSPM (R 4.2.0)
#>  purrr         1.0.1   2023-01-10 [1] RSPM (R 4.2.0)
#>  R.cache       0.16.0  2022-07-21 [1] CRAN (R 4.2.0)
#>  R.methodsS3   1.8.2   2022-06-13 [1] CRAN (R 4.2.0)
#>  R.oo          1.25.0  2022-06-12 [1] CRAN (R 4.2.0)
#>  R.utils       2.12.2  2022-11-11 [1] RSPM (R 4.2.0)
#>  R6            2.5.1   2021-08-19 [1] CRAN (R 4.2.0)
#>  reprex        2.0.2   2022-08-17 [1] RSPM (R 4.2.0)
#>  rlang         1.1.0   2023-03-14 [1] RSPM (R 4.2.0)
#>  rmarkdown     2.20    2023-01-19 [1] RSPM (R 4.2.0)
#>  rstudioapi    0.14    2022-08-22 [1] RSPM (R 4.2.0)
#>  sessioninfo   1.2.2   2021-12-06 [1] CRAN (R 4.2.0)
#>  styler        1.9.1   2023-03-04 [1] RSPM (R 4.2.0)
#>  targets       0.14.3  2023-03-08 [1] RSPM (R 4.2.0)
#>  tibble        3.2.0   2023-03-08 [1] RSPM (R 4.2.0)
#>  tidyselect    1.2.0   2022-10-10 [1] RSPM (R 4.2.0)
#>  utf8          1.2.3   2023-01-31 [1] RSPM (R 4.2.0)
#>  vctrs         0.6.0   2023-03-16 [1] RSPM (R 4.2.0)
#>  withr         2.5.0   2022-03-03 [1] CRAN (R 4.2.0)
#>  xfun          0.37    2023-01-31 [1] RSPM (R 4.2.0)
#>  yaml          2.3.7   2023-01-23 [1] RSPM (R 4.2.0)
#> 
#>  [1] /home/emrys/R/x86_64-pc-linux-gnu-library/4.2
#>  [2] /usr/local/lib/R/site-library
#>  [3] /usr/lib/R/site-library
#>  [4] /usr/lib/R/library
#> 
#> ──────────────────────────────────────────────────────────────────────────────

[wlandau]

Sorry, didn't realize you were referring to the first reprex. As I said in #107 (reply in thread), I think we have to choose between _quarto.yml and passing individual files to path in tar_quarto(). The following works and renders reports to a directory called output if you have tar_quarto(path = getwd()). Then it will get the output dir from _quarto.yml. You might want to also put specific qmd files to render in _quarto.yml.

_targets.R:

library(targets)
library(tarchetypes)
list(
  tar_target(data, data.frame(x = seq_len(26), y = letters)),
  tar_quarto(report, path = getwd())
)

_quarto.yml:

project:
  output-dir: output
render:
  report.qmd

report.qmd:

---
title: report.qmd source file
output_format: html
---
Assume these lines are in report.qmd.
```{r}
targets::tar_read(data)
```

[gorkang]

Thanks Will! Using tar_quarto(report, path = getwd() works.

I was stuck for a while because tar_quarto() seems to look for all .qmd files in the wd. So:

1. There seems to be no need for render: report.qmd in _quarto.yml
2. Even if you use render: report.qmd, if there is a failing qmd document anywhere in the wd, the pipeline will fail

 
See reprex below for 2). Commenting the Failing Quarto doc chunk will make the pipeline run without errors.

 

targets::tar_dir({  # tar_dir() runs code from a temporary directory.
  # Working Quarto doc
  lines <- c(
    "---",
    "title: report.qmd source file",
    "output_format: html",
    "---",
    "Assume these lines are in report.qmd.",
    "```{r}",
    "targets::tar_read(data)",
    "```"
  )
  writeLines(lines, "report.qmd")
  
  
  # Failing Quarto doc
  lines <- c(
    "---",
    "title: report2.qmd source file",
    "output_format: html",
    "---",
    "Assume these lines are in report2.qmd.",
    "```{r}",
    "cli::cli_abort()",
    "```"
  )
  writeLines(lines, "report2.qmd")
  
  
  # Add _quarto.yml
  yml <- c(
    "project:
        output-dir: docs
render:
    report.qmd
    "
  )
  writeLines(yml, "_quarto.yml")
  
  # Include the report in a pipeline as follows.
  targets::tar_script({
    library(tarchetypes)
    list(
      tar_target(data, data.frame(x = seq_len(26), y = letters)),
      tar_quarto(report, path = getwd())
    )
  }, ask = FALSE)
  targets::tar_make()
})
#> • start target data
#> • built target data [0.001 seconds]
#> • start target report
#> ✖ error target report
#> • end pipeline [2.163 seconds]
#> Error:
#> ! Error running targets::tar_make()
#>   Error messages: targets::tar_meta(fields = error, complete_only = TRUE)
#>   Debugging guide: https://books.ropensci.org/targets/debugging.html
#>   How to ask for help: https://books.ropensci.org/targets/help.html
#>   Last error: ! ! System command 'quarto' failed
#> Backtrace:
#>     ▆
#>  1. └─targets::tar_make()
#>  2.   └─targets:::callr_outer(...)
#>  3.     ├─targets:::if_any(...)
#>  4.     └─targets:::callr_error(condition = out, fun = fun)
#>  5.       └─targets::tar_throw_run(message, class = class(condition))
#>  6.         └─targets::tar_error(...)
#>  7.           └─rlang::abort(message = message, class = class, call = tar_empty_envir)

^{Created on 2023-03-22 with reprex v2.0.2}

[wlandau]

If render: report.qmd is ignored, it sounds a Quarto issue rather than a tarchetypes issue. So I recommend seeing if you can reproduce the error using quarto::quarto_render() without targets or tarchetypes.

[gorkang]

You are completely right. So sorry for wasting your time.

The issue was that "render" should be nested under project in _quarto.yml. So this works fine:

project:
 output-dir: outputs
 render:
  - report.qmd