Parallelize build_articles() and build_reference()

[pat-s]

#744

Uses future_map and future_walk from the furrr package if parallel = TRUE is set in build_site(), build_articles() or build_reference().

New arguments

• parallel: Whether to run in parallel or not
• progress: Whether to show a progress bar when running in parallel
• workers: Number of cores used for parallelization. Default = all available cores / 2

Comments

• By default all cores are used. It may be better to use only 50% by default as using all cores might conflict with vignettes that use parallelization themselves.

• By default, the "Reading", "Writing" output is suppressed when building in parallel in favor of a progress bar. This is opinionated. What's your opinion on this? Another option would be to only suppress the "Reading", "Writing" output if argument progress = TRUE and to show it otherwise.

Example

build_site(parallel = TRUE, lazy = FALSE, workers = 2)
══ Building pkgdown site ══════════════════════════════════════════════════════════════════════════════════════════════════════════
Reading from: '/home/pjs/git/pkgdown'
Writing to:   '/home/pjs/git/pkgdown/docs'
── Initialising site ──────────────────────────────────────────────────────────────────────────────────────────────────────────────
Writing 'sitemap.xml'
● Edit '_pkgdown.yml'
── Building home ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Reading 'CODE_OF_CONDUCT.md'
Reading 'LICENSE.md'
Reading 'README.Rmd'
Writing 'index.html'
── Building function reference 
 Progress: ──────────────────────────────────100%

── Building articles 
 Progress: ──────────────────────────────────100%

── Building news ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Writing 'news/index.html'
══ DONE ═══════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════
── Previewing site ────────────────────────────────────────────────────────────────────────────────────────────────────────────────

[hadley]

This kills the current progress reporting which makes errors difficult to debug. I think it requires more thought before we add parallel support to pkgdown.

[pat-s]

This kills the current progress reporting which makes errors difficult to debug. I think it requires more thought before we add parallel support to pkgdown.

I turned it off on purpose. If you prefer to keep the progress reporting using the "Reading/writing" approach as it applies for the sequential execution, I'll change it again.

What other potential problems do you have in mind that I am not aware off?

[pat-s]

I would bet that I saw "Reading/Writing" output when I first integrated future_walk(). This surprised me at first. Well, probably I was wrong and just saw the sequential run in this moment...

I changed the code to use half of all cores so that it does not conflict with vignettes having parallel code.

I would not see the requirement for debugging; the sequential mode can be used for this. Once one is sure that everything works, the parallel mode can be used.

If this PR is most likely not going to be merged/useful right now, feel free to close it.