Kenstone Algorithm is not running in parallel

[soottikkal]

I am trying to parallelize my R code. All of my functions are running in parallel as expected, but when the KenStone algorithm is called the R process just hangs. You can test it with the following code.

library(prospectr)

myfunc=function(x){
data(NIRsoil)
sel <- kenStone(NIRsoil$spc, k = 30, pc = .99)
}

#This works
library(parallel)
mclapply(1:2, FUN=myfunc, mc.cores=1)
#This doesnt work
mclapply(1:2, FUN=myfunc, mc.cores=2)

Any suggestions? Thanks

[l-ramirez-lopez]

Hi,
I just ran your sample code in my small Ubuntu machine and it works...
Have you tested it with future_lapply? it also works for me

[philipp-baumann]

Hi @soottikkal ,
I guess you are on Windows (?). mclapply() will only work with mc.cores = 1. See ?mclapply. As @l-ramirez-lopez mentions above the following should work:

library("prospectr")
library("future.apply")

# Custom function
myfunc <- function(x) {
  data(NIRsoil)
  sel <- kenStone(NIRsoil$spc, k = 30, pc = .99)
}

# Set up PSOCK cluster (no forking on Windows); launch new R processes
plan(multiprocess)

# execute on all available cores
future_lapply(X = 1:2, FUN = myfunc)

# or classic way via `parallel::parLapply`
# make cluster
library("parallel")
ncores <- detectCores()
cl <- makeCluster(spec = ncores)

clusterExport(cl = cl, c("kenStone", "NIRsoil"))
parLapply(cl = cl, X = 1:2, fun = myfunc)
stopCluster(cl)

[l-ramirez-lopez]

I just also tested mclapply() with kenStone() on windows and it works, although as @philipp-baumann mentiones the mc.cores has no effect on windows i.e. it will always operate sequentially