install.packages within callr::r does not find packages in provided lib

[akaever]

callr::r seems to interfere with the install.packages lib parameter (R version 3.6.0).
package1 is already installed in /tmp/lib. package2 imports package1:

callr::r(function() {
  install.packages("/tmp/dev/package2", lib = "/tmp/lib", repos = NULL, type = "source") 
})

prints

Error: package or namespace load failed for ‘package2’ in loadNamespace(i, c(lib.loc, .libPaths()), versionCheck = vI[[i]]):
there is no package called ‘package1’

Looks like the main problem is overwriting the .libPaths in the files defined in R_PROFILE and R_PROFILE_USER interpreted by R CMD install:

.libPaths(c("/usr/local/lib/R/site-library", "/usr/local/lib/R/library"))

When unsetting corresponding environment variables (or manually removing the .libPaths calls), this seems to work again:

callr::r(function() {
  Sys.unsetenv(c("R_ENVIRON", "R_ENVIRON_USER", "R_PROFILE", "R_PROFILE_USER"))
  install.packages("/tmp/dev/package2", lib = "/tmp/lib", repos = NULL, type = "source") 
})

This is not an issue with R version 3.5.1.

[gaborcsardi]

Can you please try to create a reproducible example?

Or, maybe just run

nstall.packages("/tmp/dev/package2", lib = "/tmp/lib", repos = NULL, type = "source") 

callr::r(function() {
  install.packages("/tmp/dev/package2", lib = "/tmp/lib", repos = NULL, type = "source") 
})

and show the output?

callr by default just re-created the same library path in the subprocess, so if the installation works in the main process, it should work in the subprocess.

[akaever]

Here's an example based on the docker image rocker/r-ver:3.6.0. I still believe this is an issue with R 3.6.0, callr overwriting .libPaths() in R_PROFILE, and that lib does not correspond to the site library.

## docker run -it 'rocker/r-ver:3.6.0' R

install.packages("callr")

tmpLib <- "/tmp/lib"
dir.create(tmpLib)
tmpFile <- tempfile()
download.file('https://mran.microsoft.com/snapshot/2019-06-30/src/contrib/testthat_2.1.1.tar.gz', destfile = tmpFile)

install.packages("testthat", lib = tmpLib)
# also installing the dependencies ‘assertthat’, ‘cli’, ‘crayon’, ‘digest’, ‘magrittr’, ‘praise’, ‘rlang’, ‘withr’
# ...

list.files(tmpLib)
# [1] "assertthat" "cli"        "crayon"     "digest"     "magrittr"
# [6] "praise"     "rlang"      "testthat"   "withr"

## all dependencies installed in tmpLib

## reinstall only testthat from gz file 

install.packages(tmpFile, lib = tmpLib, repos = NULL, type = "source")
# * installing *source* package ‘testthat’ ...
# ** package ‘testthat’ successfully unpacked and MD5 sums checked
# ** using staged installation
# ** libs
# gcc -I"/usr/local/lib/R/include" -DNDEBUG -I../inst/include -DCOMPILING_TESTTHAT  -I/usr/local/include  -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c init.c -o init.o
# gcc -I"/usr/local/lib/R/include" -DNDEBUG -I../inst/include -DCOMPILING_TESTTHAT  -I/usr/local/include  -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c reassign.c -o reassign.o
# g++ -std=gnu++11 -I"/usr/local/lib/R/include" -DNDEBUG -I../inst/include -DCOMPILING_TESTTHAT  -I/usr/local/include  -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c test-catch.cpp -o test-catch.o
# g++ -std=gnu++11 -I"/usr/local/lib/R/include" -DNDEBUG -I../inst/include -DCOMPILING_TESTTHAT  -I/usr/local/include  -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c test-example.cpp -o test-example.o
# g++ -std=gnu++11 -I"/usr/local/lib/R/include" -DNDEBUG -I../inst/include -DCOMPILING_TESTTHAT  -I/usr/local/include  -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c test-runner.cpp -o test-runner.o
# g++ -std=gnu++11 -shared -L/usr/local/lib/R/lib -L/usr/local/lib -o testthat.so init.o reassign.o test-catch.o test-example.o test-runner.o -L/usr/local/lib/R/lib -lR
# installing to /tmp/lib/00LOCK-testthat/00new/testthat/libs
# ** R
# ** inst
# ** byte-compile and prepare package for lazy loading
# ** help
# *** installing help indices
# *** copying figures
# ** building package indices
# ** installing vignettes
# ** testing if installed package can be loaded from temporary location
# ** checking absolute paths in shared objects and dynamic libraries
# ** testing if installed package can be loaded from final location
# ** testing if installed package keeps a record of temporary installation path
# * DONE (testthat)

list.files(tmpLib)
# [1] "assertthat" "cli"        "crayon"     "digest"     "magrittr"
# [6] "praise"     "rlang"      "testthat"   "withr"

## same with callr fails (imported packages not found in tmpLib)

callr::r(function(tmpFile, tmpLib) { install.packages(tmpFile, lib = tmpLib, repos = NULL, type = "source") }, args = list(tmpFile = tmpFile, tmpLib = tmpLib), show = TRUE)
# * installing *source* package ‘testthat’ ...
# ** package ‘testthat’ successfully unpacked and MD5 sums checked
# ** using staged installation
# ** libs
# gcc -I"/usr/local/lib/R/include" -DNDEBUG -I../inst/include -DCOMPILING_TESTTHAT  -I/usr/local/include  -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c init.c -o init.o
# gcc -I"/usr/local/lib/R/include" -DNDEBUG -I../inst/include -DCOMPILING_TESTTHAT  -I/usr/local/include  -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c reassign.c -o reassign.o
# g++ -std=gnu++11 -I"/usr/local/lib/R/include" -DNDEBUG -I../inst/include -DCOMPILING_TESTTHAT  -I/usr/local/include  -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c test-catch.cpp -o test-catch.o
# g++ -std=gnu++11 -I"/usr/local/lib/R/include" -DNDEBUG -I../inst/include -DCOMPILING_TESTTHAT  -I/usr/local/include  -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c test-example.cpp -o test-example.o
# g++ -std=gnu++11 -I"/usr/local/lib/R/include" -DNDEBUG -I../inst/include -DCOMPILING_TESTTHAT  -I/usr/local/include  -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c test-runner.cpp -o test-runner.o
# g++ -std=gnu++11 -shared -L/usr/local/lib/R/lib -L/usr/local/lib -o testthat.so init.o reassign.o test-catch.o test-example.o test-runner.o -L/usr/local/lib/R/lib -lR
# installing to /tmp/lib/00LOCK-testthat/00new/testthat/libs
# ** R
# ** inst
# ** byte-compile and prepare package for lazy loading
# Error in loadNamespace(i, c(lib.loc, .libPaths()), versionCheck = vI[[i]]) :
#   there is no package called ‘rlang’
# Calls: <Anonymous> ... loadNamespace -> withRestarts -> withOneRestart -> doWithOneRestart
# Execution halted
# ERROR: lazy loading failed for package ‘testthat’
# * removing ‘/tmp/lib/testthat’
# * restoring previous ‘/tmp/lib/testthat’
# Warning message:
#   In install.packages(tmpFile, lib = tmpLib, repos = NULL, type = "source") :
#   installation of package ‘/tmp/RtmpjbMpuT/file145158604’ had non-zero exit status
# NULL

[gaborcsardi]

This is a tricky one. We actually test for the library path in a subprocess started by system() from the callr R process. (I.e. R ---callr---> R ---system---> R), but in this case the last one is R CMD INSTALL, which might be special.

I'll try to fix this now, but in any case a workaround is to set the library path within the function passed to callr.

[gaborcsardi]

Maybe what we should do is to save all related env vars in the main process, and then restore them in the subprocess, at startup. That way we

1. get an environment with identical library config, and
2. do not carry the custom setup with us to a sub-subprocess.

[gaborcsardi]

Maybe what we should do is to save all related env vars in the main process, and then restore them in the subprocess, at startup.

This definitely seems like the right thing to do, according to the principle of

f()
callr(function() f())

being equivalent. But it is also not backward compatible. :(