BUG: makeClusterPSOCK() produces error if 'parallelly' is not available on worker

[HenrikBengtsson]

Issue

makeClusterPSOCK() produces an error on 'could not find function "getOptionOrEnvVar"' if parallelly is not available on R worker.

Reproducible example

Let's hide the parallelly package from the worker by setting a dummy R_LIBS_USER value;

Sys.setenv(R_LIBS_USER=tempdir())
cl <- parallelly::makeClusterPSOCK(1)
## Error in checkForRemoteErrors(lapply(cl, recvResult)) : 
##   one node produced an error: could not find function "getOptionOrEnvVar"

with

> traceback()
5: stop("one node produced an error: ", firstmsg, domain = NA)
4: checkForRemoteErrors(lapply(cl, recvResult))
3: clusterCall(cl[ii], fun = session_info)
2: add_cluster_session_info(cl[ii])
1: parallelly::makeClusterPSOCK(1)

Troubleshooting

This is because of the

clusterCall(cl[ii], fun = session_info)

call (cf. Issue #22), where session_info(), which is part of the parallelly package, looks like:

> parallelly:::session_info
function (pkgs = getOptionOrEnvVar("future.makeNodePSOCK.sessionInfo.pkgs", FALSE)) 
{
   ...
}

The pkgs argument is needed but since it is not specifying in the above clusterCall() it falls back to the default value, which requires (non-exported) getOptionOrEnvVar() that is part of the parallelly package.

Now, if the parallelly package would have been available on the worker, it would have worked because R will try to load the parallelly package when it recieves the parallelly:::session_info() object. To see that it actually fails finding the package, we can do:

Sys.setenv(R_LIBS_USER=tempdir())
cl <- parallelly::makeClusterPSOCK(1, outfile="")
## starting worker pid=10563 on localhost:11361 at 19:39:14.323
## Warning: namespace ‘parallelly’ is not available and has been replaced
## by .GlobalEnv when processing object ‘<unknown>’
## Error in checkForRemoteErrors(lapply(cl, recvResult)) : 
## one node produced an error: could not find function "getOptionOrEnvVar"

Since that's only a warning, we don't see it unless we use outfile="" - and even with that we might not see it on all operating systems.

BTW, this also explains why parallelly is loaded on a fresh worker.

Solution

Specify argument pkgs in the call, i.e.

pkgs <- getOptionOrEnvVar("future.makeNodePSOCK.sessionInfo.pkgs", FALSE)
clusterCall(cl[ii], fun = session_info, pkgs = pkgs)

so that it is not needed on the workers.

This bug was identified because the above error showed up kind of randomly when revdep checking the develop version of future that rely on parallelly for setting up clusters, e.g. HenrikBengtsson/future@a8e3b1a but also HenrikBengtsson/future@02fe610. Not sure why rerunning made the latter disappear but I suspect the revdep checks somehow picks up what current version of future I have installed on my account.

[HenrikBengtsson]

Fixed

[HenrikBengtsson]

Workaround

Until the next version of parallelly is on CRAN, we can workaround this bug by tweaking rscript_startup;

Sys.setenv(R_LIBS_USER = tempdir())
rscript_startup <- "getOptionOrEnvVar <- function(...) FALSE"
cl <- parallelly::makeClusterPSOCK(1L, rscript_startup = rscript_startup)

or at bit fancier to respect those options and env vars:

Sys.setenv(R_LIBS_USER = tempdir())
pkgs <- parallelly:::getOptionOrEnvVar("future.makeNodePSOCK.sessionInfo.pkgs", FALSE)
rscript_startup <- paste("getOptionOrEnvVar <- function(...)", pkgs)
cl <- parallelly::makeClusterPSOCK(1L, rscript_startup = rscript_startup)