Fails to find extraneous package functions when passed a function in a list

[khvorov45]

OS: Windows 10
R 3.6.1
furrr: 0.1.0.9002
future: 1.15.1

Putting a function into a list and then passing it into future_map apparently causes it to not find functions appropriately.

library(extraDistr)
suppressPackageStartupMessages(library(purrr))
suppressPackageStartupMessages(library(furrr))

plan(multiprocess)

# Innermost function which uses something from a package
inner_fun <- function() rbern(10, 0.5) # from extraDistr

# A function that calls another which is passed to it in a list
wrapper_fun <- function(fun_list) {
  fun_list$sim_fun()
}

# A list of functions
list_of_funs <- list(sim_fun = inner_fun)

# Will work
map(
  1:2, 
  function(i) wrapper_fun(list_of_funs)
)
#> [[1]]
#>  [1] 1 0 0 1 0 0 1 0 0 0
#> 
#> [[2]]
#>  [1] 1 1 0 1 0 0 0 0 1 0

# Will fail to find rbern
future_map(
  1:2, 
  function(i) wrapper_fun(list_of_funs)
)
#> Error in rbern(10, 0.5): could not find function "rbern"

^{Created on 2019-11-27 by the reprex package (v0.3.0)}

[khvorov45]

If anyone has the same issue, the solution (workaround?) is to add .options = future_options(packages = "extraDistr") to the future_map call

future_map(
  1:2, 
  function(i) wrapper_fun(list_of_funs),
  .options = future_options(packages = "extraDistr")
)
#> [[1]]
#>  [1] 1 0 0 1 1 1 1 0 0 0
#> 
#> [[2]]
#>  [1] 1 0 1 0 0 1 1 0 1 0

Although I'm not sure if furrr is supposed to behave like this.

[EmilRehnberg]

This is due to that actually multisession is being used (since forking is disabled on Win). The same code would work fine on a Unix system.

[EmilRehnberg]

I am having this problem too, that code working under multiprocess isn't working under multisession.

E.g.

library(furrr)
plan(multisession)

dl <- list(
  a = data.frame(r = runif(4)),
  b = runif(5)
)

run <- function(obj) UseMethod("run")
run.default <- function(obj) {
  paste("don't know what to do with object of class:", class(obj))
}
run.numeric <- function(nums) paste("mean is:", mean(nums))
furrr::future_map(dl, run) # fails using multisession, works under multiprocess.

Not sure if this is an issue with furrr or future.

[EmilRehnberg]

Based on more testing with this, multisession seems to require you to

pass in packages

via e.g. future_options(packages = "magrittr") for packages that are expected to be loaded.

Usually you can use pkg::fun syntax but for operators and certain S3 polymorphism to work (e.g. if you have to subset a ts object, you need forecast package to be loaded)

reload functions that has a dot in the name

functions with dots in the environment doesn't seem to be passed to the workers so all S3 polymorphism will need to be handled in special ways.

@DavisVaughan do you know more about this? Is this expected? Or does it need to be fixed?

[DavisVaughan]

I think this is a globals bug (but is probably hard to fix). Right here, globalsOf() filters down to only functions before calling itself recursively on those functions. This filters out the list of functions which should be searched. Not searching the list of functions means that rbern() is never identified as a global

https://github.com/HenrikBengtsson/globals/blob/24f150843038dd8f078c050f26d4a18c644b43a6/R/globalsOf.R#L110-L113

It doesn't seem like there is anything I can do in furrr, so I'm going to close this one

@HenrikBengtsson here is a globals only reprex that shows rbern() not being found

library(extraDistr)
#> Warning: package 'extraDistr' was built under R version 4.0.2
library(globals)

# uses rbern() from extraDistr
inner_fun <- function() rbern(10, 0.5)

# A function that calls another which is passed to it in a list
wrapper_fun <- function(fun_list) {
  fun_list$sim_fun()
}

# A list of functions
list_of_funs <- list(sim_fun = inner_fun)

expr <- quote(function(i) wrapper_fun(list_of_funs))

globalsOf(expr)
#> $wrapper_fun
#> function(fun_list) {
#>   fun_list$sim_fun()
#> }
#> 
#> $list_of_funs
#> $list_of_funs$sim_fun
#> function() rbern(10, 0.5)
#> 
#> 
#> $`{`
#> .Primitive("{")
#> 
#> $`$`
#> .Primitive("$")
#> 
#> attr(,"class")
#> [1] "Globals" "list"   
#> attr(,"where")
#> attr(,"where")$wrapper_fun
#> <environment: R_GlobalEnv>
#> 
#> attr(,"where")$list_of_funs
#> <environment: R_GlobalEnv>
#> 
#> attr(,"where")$`{`
#> <environment: base>
#> 
#> attr(,"where")$`$`
#> <environment: base>

^{Created on 2020-08-05 by the reprex package (v0.3.0)}