Help needed setting up cores for workers

[ramiromagno]

Now I was trying to set the number of cores per worker but it is running on one core only in each node.

Here's my setup:

library(future.apply)
library(daeqtlr)

custom_workers <- function() {
  switch(Sys.info()[["nodename"]],
         "rey" = 64L,
         "r2d2" = 40L,
         ## default:
         parallelly::availableCores()
  )
}

future::plan(list(
  future::tweak(
    future::cluster,
    workers = c('rey', 'r2d2'),
    homogeneous = FALSE
  ),
  future::tweak(future::multisession, workers = custom_workers)
))


snp_pairs <- read_snp_pairs(file = daeqtlr_example("snp_pairs.csv"))
zygosity <- read_snp_zygosity(file = daeqtlr_example("zygosity.csv"))
ae <- read_ae_ratios(file = daeqtlr_example("ae.csv"))

snp_pairs2 <- snp_pairs[sample(1:200, 10000, replace = TRUE), ]

daeqtl_mapping2(snp_pairs = snp_pairs2, zygosity = zygosity, ae = ae, .n_cores = 64 + 40)

The function daeqtl_mapping2():

#' @export
daeqtl_mapping2 <-
  function(snp_pairs,
           zygosity,
           ae,
           fn = daeqtl_test,
           ...,
           .extra_cols = 2L,
           .n_cores = 1L) {

    snp_pairs_lst <-
      split(snp_pairs, split_index(nrow(snp_pairs), .n_cores))

    for (i in seq_along(snp_pairs_lst)) {
      data.table::setkeyv(snp_pairs_lst[[i]], c('dae_snp', 'candidate_snp'))
    }

    res <- future.apply::future_lapply(
      snp_pairs_lst,
      FUN = daeqtl_mapping_,
      zygosity = zygosity,
      ae = ae,
      fn = fn,
      ...,
      .extra_cols = .extra_cols
    )

    # Put together the data table from the list of data tables
    mapping_dt <- data.table::rbindlist(res)

    data.table::setkeyv(mapping_dt, cols = c('dae_snp', 'candidate_snp'))
    return(mapping_dt)
  }

[HenrikBengtsson]

With

future::plan(list(
  future::tweak(future::cluster, workers = c('rey', 'r2d2'), homogeneous = FALSE),
  future::tweak(future::multisession, workers = custom_workers)
))

you're configuring two layers of workers. In order to reach the second layer of workers, you need to use nested futures, which are explained in https://future.futureverse.org/articles/future-3-topologies.html.

Having said that, I think you're really asking for the following:

workers <- c(rep("rey", times = 64L), rep("r2d2", times = 40L))
future::plan(future::cluster, workers = workers, homogeneous = FALSE)

I'm not how sure how well this is documented, but it's mentioned in https://future.futureverse.org/articles/future-3-topologies.html and in the examples of https://parallelly.futureverse.org/reference/makeClusterPSOCK.html.

UPDATE +15 min: Added a paragraph on this to https://future.futureverse.org/articles/future-1-overview.html#cluster-futures. It was also illustrated in the 'Demos' section at the end of that vignette.

[ramiromagno]

Thank so so much for your help. Truly appreciated!