Parallel computing with Future

[LeaLe88]

Dear all,

Sorry I need to open one more time the topic of parallel computing with the future package for findmarkers.

Following my previous post #8315 I have run below code in order to use multiple cores for the findmarkers function using the future package.

However, for some reason I am using all 64 cores even though I specified to use only 6:

plan(multisession(workers = 6))   #seems not to be working correctly 
idents <- levels(Idents(kid.filtered_new))

# To avoid this error run:
#Error in getGlobalsAndPackages(expr, envir = envir, globals = globals) : 
#The total size of the 2 globals exported for future expression ('FUN()') is 1.27 GiB.. This exceeds the maximum allowed size of 500.00 MiB (option 'future.globals.maxSize'). 
options(future.globals.maxSize = 8000 * 1024^2)

#Create future
markers <- future_lapply(idents, function(x) {
  Clu_marker <- FindMarkers(object = kid.filtered_new, ident.1 = x, test.use="poisson",latent.vars = "orig.ident")
  Clu_marker$cluster <- x
},future.seed = TRUE)

What am I missing here?

BTW I had to add options(future.globals.maxSize = 8000 * 1024^2) in order to avoid the error shown above.

Thanks so much!

[samuel-marsh]

@LeaLe88 see comment on #8315 on correct manner to specify workers and see if that solves the issue.

if not then it would seem like more of future issue than Seurat and would suggest posting to future repo.

Best,
Sam