Travis/pkgdown sometimes fails when building the vignette due to memory, but not with the same objects earlier in R CMD check #1230

lcolladotor · 2020-02-28T00:16:13Z

Hi,

I'm a bit puzzled with a particular repository that sometimes fails deploying from Travis to GitHub due to out of memory errors and sometimes doesn't . The memory error always happens when pkgdown is building the vignette. The closest pkgdown issue I could find was #783 though that one was closed due to a lack of access to the code.

This particular package uses about 2.5 GB of RAM in memory of R objects simply to get going, so yes, it's a beast. Something that really puzzles me is that the examples run fine (R CMD check) and the Travis build never fails at that point. My understanding is that pkgdown::deploy_site_github() is run on a new R process at the end, such that it should in theory have the exact same memory to use as the R process that R CMD check earlier. However if the two R processes are active at the same time, this could double the memory and bring it close to the 7.5 GB RAM limit of the Google Compute Engine used by Travis as detailed in their docs leading to the cases where sometimes it runs out of memory and sometimes it doesn't.

I'd appreciate if you have any suggestions that I could try or if there's more information that I provide you.

Best,
Leo

Hm...

As I write this, I realize that a more likely case is that the R process running pkgdown::deploy_site_github() keeps all the same objects used in the examples around as when building the vignette. Currently my examples use code like this (includes roxygen format):

#' ## Obtain the necessary data
#' if (!exists('ori_sce')) ori_sce <- fetch_data('sce')

## That object uses 2.08 of RAM as determined by pryr::object_size()

but then in the vignette I use sce <- fetch_data('sce'). Hm... However, I actually don't use that big object in the vignette. I created a fake version that I use then (96.5 Mb) (details here).

I will test a version where I use the if (!exists()) syntax with the same object names through both the docs and the vignette.

Error message

Ok, here's the actual error message copy-pasted from https://travis-ci.com/LieberInstitute/spatialLIBD/builds/150899142#L1106-L1167.

── Building articles ───────────────────────────────────────────────────────────
Writing 'articles/index.html'
Reading 'vignettes/spatialLIBD.Rmd'
Error in rethrow_call(c_processx_exec, command, c(command, args), stdin,  : 
  Cannot fork when running '/home/travis/R-bin/lib/R/bin/R' (system error -12, Unknown error -12) @unix/processx.c:499 (processx_exec)
Warning message:
system call failed: Cannot allocate memory 
Error: <callr_status_error: callr subprocess failed: Cannot fork when running '/home/travis/R-bin/lib/R/bin/R' (system error -12, Unknown error -12) @unix/processx.c:499 (processx_exec)>
-->
<callr_remote_error in rethrow_call(c_processx_exec, command, c(command, args), stdin,  ...:
 Cannot fork when running '/home/travis/R-bin/lib/R/bin/R' (system error -12, Unknown error -12) @unix/processx.c:499 (processx_exec)>
 in process 14754 
 Stack trace:
 Process 14660:
 1. pkgdown::deploy_site_github(verbose = TRUE, ssh_id = Sys.getenv("TRAVIS_DEP ...
 2. pkgdown:::deploy_local(pkg, repo_slug = repo_slug, commit_message = commit_ ...
 3. pkgdown:::build_site(".", override = list(destination = dest_dir),  ...
 4. pkgdown:::build_site_external(pkg = pkg, examples = examples,  ...
 5. callr::r(function(..., crayon_enabled, crayon_colors, pkgdown_internet) { ...
 6. callr:::get_result(output = out, options)
 7. throw(newerr, parent = remerr[[2]])
 x callr subprocess failed: Cannot fork when running '/home/travis/R-bin/lib/R/bin/R' (system error -12, Unknown error -12) @unix/processx.c:499 (processx_exec) 
 Process 14754:
 19. (function (..., crayon_enabled, crayon_colors, pkgdown_internet)  ...
 20. pkgdown::build_site(...)
 21. pkgdown:::build_site_local(pkg = pkg, examples = examples, run_dont_run = r ...
 22. pkgdown:::build_articles(pkg, lazy = lazy, override = override,  ...
 23. purrr::walk(pkg$vignettes$name, build_article, pkg = pkg, quiet = quiet,  ...
 24. purrr:::map(.x, .f, ...)
 25. pkgdown:::.f(.x[[i]], ...)
 26. pkgdown:::render_rmarkdown(pkg, input = input, output = output_file,  ...
 27. callr::r_safe(function(...) rmarkdown::render(...), args = args,  ...
 28. callr:::run_r(options)
 29. base:::with(options, with_envvar(env, do.call(processx::run,  ...
 30. base:::with.default(options, with_envvar(env, do.call(processx::run,  ...
 31. base:::eval(substitute(expr), data, enclos = parent.frame())
 32. base:::eval(substitute(expr), data, enclos = parent.frame())
 33. callr:::with_envvar(env, do.call(processx::run, c(list(bin, args = real_cmd ...
 34. base:::force(code)
 35. base:::do.call(processx::run, c(list(bin, args = real_cmdargs,  ...
 36. (function (command = NULL, args = character(), error_on_status = TRUE,  ...
 37. process$new(command, args, echo_cmd = echo_cmd, wd = wd, windows_verbatim_a ...
 38. .subset2(public_bind_env, "initialize")(...)
 39. processx:::process_initialize(self, private, command, args, stdin,  ...
 40. base:::rethrow_call(c_processx_exec, command, c(command, args),  ...
 41. base:::withCallingHandlers(do.call(".Call", list(.NAME, ...)),  ...
 42. base:::do.call(".Call", list(.NAME, ...))
 43. base:::.handleSimpleError(function (e)  ...
 44. base:::h(simpleError(msg, call))
 45. base:::throw(e)
 46. base:::signalCondition(cond)
 47. (function (e)  ...
 x Cannot fork when running '/home/travis/R-bin/lib/R/bin/R' (system error -12, Unknown error -12) @unix/processx.c:499 (processx_exec) 
Execution halted
Script failed with status 1

Recent build history

I inspected the recent build history to see if I could pick up any patterns. From the Travis docs I learnt that the worker information specifies which infrastructure was used. I see several GCE-production-1, 2 and 3, which from the docs, are the instances with 7.5 GB of RAM. I don't really see any clues, though maybe you do pick up some.

Build 31: worked!

Build link: https://travis-ci.com/LieberInstitute/spatialLIBD/builds/150877464
Travis worker information

hostname: 1bad30be-bd46-4294-8b20-7d9a1d55282b@1.worker-com-f6b8745bf-wc4sq.gce-production-1
version: v6.2.8 https://github.com/travis-ci/worker/tree/6d3048d96b26562be21fa1a8b8144f4c4cecd083
instance: travis-job-16677ea7-3070-4227-b365-02ec8eacab72 travis-ci-opal-xenial-1553530491-f909ac5 (via amqp)
startup: 6.137394972s

Build 32: worked!

Build link: https://travis-ci.com/LieberInstitute/spatialLIBD/builds/150879986
Travis worker information

hostname: 04eab6d4-1134-42dd-8816-aa9f5869db24@1.worker-com-f6b8745bf-w9swj.gce-production-1
version: v6.2.8 https://github.com/travis-ci/worker/tree/6d3048d96b26562be21fa1a8b8144f4c4cecd083
instance: travis-job-e68e627f-8f90-43c2-9511-30623bd716b4 travis-ci-opal-xenial-1553530491-f909ac5 (via amqp)
startup: 6.239525641s

Build 33: failed! (other error)

note that this one failed due to an actual error with R CMD check that I introduced

Build link: https://travis-ci.com/LieberInstitute/spatialLIBD/builds/150891053
Travis worker information

hostname: c32b5b6d-4d1e-4b9e-a3e1-98c919079b63@1.worker-com-755dc776cf-pslrn.gce-production-3
version: v6.2.8 https://github.com/travis-ci/worker/tree/6d3048d96b26562be21fa1a8b8144f4c4cecd083
instance: travis-job-7217b984-743f-4868-96a3-223d9557b6ca travis-ci-opal-xenial-1553530491-f909ac5 (via amqp)
startup: 6.533376154s

Build 34: worked!

Build link: https://travis-ci.com/LieberInstitute/spatialLIBD/builds/150894335
Travis worker information

hostname: d2ebb3cc-ee15-4628-b63a-48e82345a51f@1.worker-com-66449df859-zqp8m.gce-production-3
version: v6.2.8 https://github.com/travis-ci/worker/tree/6d3048d96b26562be21fa1a8b8144f4c4cecd083
instance: travis-job-79185772-9946-4a52-ae2b-e8c2ea33d8d5 travis-ci-opal-xenial-1553530491-f909ac5 (via amqp)
startup: 6.306192603s

Build 35: worked!

Build link: https://travis-ci.com/LieberInstitute/spatialLIBD/builds/150895828
Travis worker information

hostname: 3e555756-2122-44d1-8b56-c67b2350d673@1.worker-com-66449df859-hgqwr.gce-production-3
version: v6.2.8 https://github.com/travis-ci/worker/tree/6d3048d96b26562be21fa1a8b8144f4c4cecd083
instance: travis-job-4ea7cbaa-f39d-48c2-982d-3dc449edf002 travis-ci-opal-xenial-1553530491-f909ac5 (via amqp)
startup: 6.34185147s

Build 36: failed! (other error)

note that this one failed due to an actual error with R CMD check that I introduced

Build link: https://travis-ci.com/LieberInstitute/spatialLIBD/builds/150898566
Travis worker information

hostname: c32b5b6d-4d1e-4b9e-a3e1-98c919079b63@1.worker-com-755dc776cf-pslrn.gce-production-3
version: v6.2.8 https://github.com/travis-ci/worker/tree/6d3048d96b26562be21fa1a8b8144f4c4cecd083
instance: travis-job-7217b984-743f-4868-96a3-223d9557b6ca travis-ci-opal-xenial-1553530491-f909ac5 (via amqp)
startup: 6.533376154s

Build 37: failed! (`pkgdown` error)

note that this one failed due to an actual error with R CMD check that I introduced

Build link: https://travis-ci.com/LieberInstitute/spatialLIBD/builds/150899142
Travis worker information

hostname: 0f974d1b-d056-4dfb-8a42-a5e30382a18a@1.worker-com-c5d6c78f6-vwfgb.gce-production-2
version: v6.2.8 https://github.com/travis-ci/worker/tree/6d3048d96b26562be21fa1a8b8144f4c4cecd083
instance: travis-job-cad7e389-4857-4d86-b36d-cbf727a83dc2 travis-ci-opal-xenial-1553530491-f909ac5 (via amqp)
startup: 6.177977162s

Build 38: failed! (`pkgdown` error)

note that this one failed due to an actual error with R CMD check that I introduced

Build link: https://travis-ci.com/LieberInstitute/spatialLIBD/builds/150919167
Travis worker information

hostname: 2ad3fc52-d266-4859-8bff-dbd74d4a0219@1.worker-com-c5d6c78f6-zjhrr.gce-production-2
version: v6.2.8 https://github.com/travis-ci/worker/tree/6d3048d96b26562be21fa1a8b8144f4c4cecd083
instance: travis-job-6e195d02-3a32-4e8d-a885-5a75148a989a travis-ci-opal-xenial-1553530491-f909ac5 (via amqp)
startup: 6.108245454s

Build 39: failed! (`pkgdown` error)

note that this one failed due to an actual error with R CMD check that I introduced

Build link: https://travis-ci.com/LieberInstitute/spatialLIBD/builds/150922974
Travis worker information

hostname: 61439fc6-85a7-4990-a380-0ac6775357f4@1.worker-com-755dc776cf-d9rt4.gce-production-3
version: v6.2.8 https://github.com/travis-ci/worker/tree/6d3048d96b26562be21fa1a8b8144f4c4cecd083
instance: travis-job-06ced571-d304-4a0c-9f2b-1b233e2f44ce travis-ci-opal-xenial-1553530491-f909ac5 (via amqp)
startup: 6.37963737s

Repository in question

Package documentation website: http://research.libd.org/spatialLIBD/
GitHub repository for the package https://github.com/LieberInstitute/spatialLIBD
Travis build reports: https://travis-ci.com/LieberInstitute/spatialLIBD
.travis.yml code that calls pkgdown: https://github.com/LieberInstitute/spatialLIBD/blob/2030888d0b0ef98fc4409d749af1b5f62dec8ae0/.travis.yml#L10-L13
Note that you will need RcppAnnoy version 0.0.14 instead of the latest (from today) version 0.0.15 (due to this issue) which you can install with remotes::install_version('RcppAnnoy', version = '0.0.14') as done in the .travis.yml code here.

Unlike when I wrote https://twitter.com/fellgernon/status/1231483308120444929?s=20, I can now actually post an issue since we made the GitHub repository public so we could submit it to Bioconductor.

Locally

Unlike the previous issue #783, I do not encounter this problem locally on my macOS (16 GB of RAM). I even made a script that I would run manually to update my website using the internals of pkgdown. While monitoring the memory usage with the macOS Activity Monitor, I saw no jumps during this whole process, also unlike #783.

$ more update_spatialLIBD.sh
#!/bin/bash

# sh ~/Dropbox/code/update_spatialLIBD.sh

cd ~/Dropbox/Code/spatialLIBD
## Note the upgrade = FALSE, otherwise I run into the already mentioned RcppAnnoy issue
## when deploying spatialLIBD to shinyapps.io
Rscript -e "remotes::install_github('LieberInstitute/spatialLIBD', upgrade = FALSE); pkgdown:::deploy_local('.', repo_slug = 'LieberInstitute/spatialLIBD', commit_message = pkgdown:::construct_commit_message('.'))"
cd -

Local R session information

> library(pkgdown)
> library(RcppAnnoy) # so it'll show in the session info
> options(width = 120)
> sessioninfo::session_info()
─ Session info ───────────────────────────────────────────────────────────────────────────────────────────────────────
 setting  value                       
 version  R version 3.6.2 (2019-12-12)
 os       macOS Catalina 10.15.2      
 system   x86_64, darwin15.6.0        
 ui       RStudio                     
 language (EN)                        
 collate  en_US.UTF-8                 
 ctype    en_US.UTF-8                 
 tz       America/New_York            
 date     2020-02-27                  

─ Packages ───────────────────────────────────────────────────────────────────────────────────────────────────────────
 package     * version  date       lib source                            
 assertthat    0.2.1    2019-03-21 [1] CRAN (R 3.6.0)                    
 backports     1.1.5    2019-10-02 [1] CRAN (R 3.6.0)                    
 callr         3.4.2    2020-02-12 [1] CRAN (R 3.6.0)                    
 cli           2.0.1    2020-01-08 [1] CRAN (R 3.6.0)                    
 codetools     0.2-16   2018-12-24 [1] CRAN (R 3.6.2)                    
 colorout    * 1.2-1    2019-05-07 [1] Github (jalvesaq/colorout@7ea9440)
 crayon        1.3.4    2017-09-16 [1] CRAN (R 3.6.0)                    
 desc          1.2.0    2018-05-01 [1] CRAN (R 3.6.0)                    
 devtools    * 2.2.1    2019-09-24 [1] CRAN (R 3.6.1)                    
 digest        0.6.25   2020-02-23 [1] CRAN (R 3.6.0)                    
 ellipsis      0.3.0    2019-09-20 [1] CRAN (R 3.6.0)                    
 fansi         0.4.1    2020-01-08 [1] CRAN (R 3.6.0)                    
 fs            1.3.1    2019-05-06 [1] CRAN (R 3.6.0)                    
 glue          1.3.1    2019-03-12 [1] CRAN (R 3.6.0)                    
 magrittr      1.5      2014-11-22 [1] CRAN (R 3.6.0)                    
 MASS          7.3-51.5 2019-12-20 [1] CRAN (R 3.6.0)                    
 memoise       1.1.0    2017-04-21 [1] CRAN (R 3.6.0)                    
 packrat       0.5.0    2018-11-14 [1] CRAN (R 3.6.0)                    
 pkgbuild      1.0.6    2019-10-09 [1] CRAN (R 3.6.0)                    
 pkgdown     * 1.4.1    2019-09-15 [1] CRAN (R 3.6.0)                    
 pkgload       1.0.2    2018-10-29 [1] CRAN (R 3.6.0)                    
 prettyunits   1.1.1    2020-01-24 [1] CRAN (R 3.6.2)                    
 processx      3.4.2    2020-02-09 [1] CRAN (R 3.6.0)                    
 ps            1.3.2    2020-02-13 [1] CRAN (R 3.6.0)                    
 R6            2.4.1    2019-11-12 [1] CRAN (R 3.6.1)                    
 Rcpp          1.0.3    2019-11-08 [1] CRAN (R 3.6.0)                    
 RcppAnnoy   * 0.0.14   2019-11-12 [1] CRAN (R 3.6.2)                    
 remotes       2.1.1    2020-02-15 [1] CRAN (R 3.6.0)                    
 rlang         0.4.4    2020-01-28 [1] CRAN (R 3.6.0)                    
 rprojroot     1.3-2    2018-01-03 [1] CRAN (R 3.6.0)                    
 rstudioapi    0.11     2020-02-07 [1] CRAN (R 3.6.0)                    
 sessioninfo   1.1.1    2018-11-05 [1] CRAN (R 3.6.0)                    
 testthat    * 2.3.1    2019-12-01 [1] CRAN (R 3.6.0)                    
 usethis     * 1.5.1    2019-07-04 [1] CRAN (R 3.6.0)                    
 withr         2.1.2    2018-03-15 [1] CRAN (R 3.6.0)                    

[1] /Library/Frameworks/R.framework/Versions/3.6/Resources/library

The text was updated successfully, but these errors were encountered:

hadley · 2020-03-10T22:20:38Z

The only thing I can think to suggest is to rm() the big object once you've done with it.

Former-commit-id: 7e0ce2c04bed8e8178266e7dd87de71d2bea9f96 [formerly aad74a6] Former-commit-id: 8f0c2c7c44849fed96b6d69160323517563e63f8

Former-commit-id: b447bbeada6e17da9a127c065b39aa54c3b4d754 [formerly 9d39947] Former-commit-id: 7ae7a00bb0bcd656aa37fbaa007e02d912907d2a

Former-commit-id: 7e0ce2c04bed8e8178266e7dd87de71d2bea9f96 [formerly aad74a6] Former-commit-id: 8f0c2c7c44849fed96b6d69160323517563e63f8 Former-commit-id: c1f4322

Former-commit-id: b447bbeada6e17da9a127c065b39aa54c3b4d754 [formerly 9d39947] Former-commit-id: 7ae7a00bb0bcd656aa37fbaa007e02d912907d2a Former-commit-id: 60f8699

lcolladotor changed the title ~~Travis/pkgdown fails when building the vignette due to memory, but not with the same objects earlier in R CMD check~~ Travis/pkgdown sometimes fails when building the vignette due to memory, but not with the same objects earlier in R CMD check Feb 28, 2020

lcolladotor added a commit to LieberInstitute/spatialLIBD that referenced this issue Feb 28, 2020

v0.99.7 -- run a test for r-lib/pkgdown#1230. Works locally

aad74a6

lcolladotor added a commit to LieberInstitute/spatialLIBD that referenced this issue Feb 28, 2020

Disable pkgdown until r-lib/pkgdown#1230 gets resolved

9d39947

hadley closed this as completed Mar 10, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Travis/pkgdown sometimes fails when building the vignette due to memory, but not with the same objects earlier in R CMD check #1230

Travis/pkgdown sometimes fails when building the vignette due to memory, but not with the same objects earlier in R CMD check #1230

lcolladotor commented Feb 28, 2020 •

edited

Loading

hadley commented Mar 10, 2020

Travis/pkgdown sometimes fails when building the vignette due to memory, but not with the same objects earlier in R CMD check #1230

Travis/pkgdown sometimes fails when building the vignette due to memory, but not with the same objects earlier in R CMD check #1230

Comments

lcolladotor commented Feb 28, 2020 • edited Loading

Hm...

Error message

Recent build history

Build 31: worked!

Build 32: worked!

Build 33: failed! (other error)

Build 34: worked!

Build 35: worked!

Build 36: failed! (other error)

Build 37: failed! (pkgdown error)

Build 38: failed! (pkgdown error)

Build 39: failed! (pkgdown error)

Repository in question

Locally

Local R session information

hadley commented Mar 10, 2020

lcolladotor commented Feb 28, 2020 •

edited

Loading

Build 37: failed! (`pkgdown` error)

Build 38: failed! (`pkgdown` error)

Build 39: failed! (`pkgdown` error)