Skip to content
slurmR: A Lightweight Wrapper for Slurm
R TeX Other
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
R
data-raw Adding more documentation and a couple of new functions Sep 5, 2019
inst Adding a few more tests Oct 29, 2019
man-roxygen Adding more test before submiting to CRAN Oct 8, 2019
man
paper Update paper.md Oct 2, 2019
tests Improving coverage Oct 21, 2019
vignettes
.Rbuildignore
.gitignore New way to wait without having to submit another job Oct 17, 2019
.travis.yml Removing docs... that should be part of the gh-pages branch Nov 19, 2019
CODE_OF_CONDUCT.md Updating website Aug 24, 2018
CRAN-RELEASE Checking travis (for osx...) Oct 15, 2019
DESCRIPTION Coverage is now done with tinytest and at HPC Oct 1, 2019
LICENSE
LICENSE.md Updating website Aug 24, 2018
NAMESPACE Cleaning up code of new_bash and adding new function slurmr_cmd Oct 29, 2019
NEWS.md Bump version for JOSS review (see openjournals/joss-reviews#1493 (com… Jul 8, 2019
README.Rmd Working on documentation Oct 31, 2019
README.md Working on documentation Nov 19, 2019
_pkgdown.yml Updating website Aug 24, 2018
appveyor.yml Adding appveyor and MacOS to travis. Skipping test on non-unix systems Oct 9, 2019
comparing-hpc-tools.ods Working on documentation Oct 31, 2019
cran-comments.md Checking travis (for osx...) Oct 15, 2019
makefile All working OK Oct 29, 2019
slurmR.Rproj New way to wait without having to submit another job Oct 17, 2019

README.md

DOI Travis build status codecov lifecycle CRAN status

slurmR: A Lightweight Wrapper for Slurm

Slurm Workload Manager is a popular HPC cluster job scheduler found in many of the top 500 super computers. The slurmR R package provides an R wrapper to it that matches the parallel package’s syntax, this is, just like parallel provides the parLapply, clusterMap, parSapply, etc., slurmR provides Slurm_lapply, Slurm_Map, Slurm_sapply, etc.

While there are other alternatives such as future.batchtools, batchtools, clustermq, and rslurm, this R package has the following goals:

  1. It is dependency free, which means that it works out-of-the-box

  2. Puts an emphasis on been similar to the workflow in the R package parallel

  3. It provides a general framework for the user to create its own wrappers without using template files.

  4. Is specialized on Slurm, meaning more flexibility (no need to modify template files), and, in the future, better debuging tools (e.g. job resubmission).

  5. Provide a backend for the parallel package, providing an out-of-the-box method for creating Socket cluster objects for multi-node operations. (See the examples below on how this can be used with other R packages)

Checkout the VS section section for comparing slurmR with other R packages. Wondering who is using Slurm? Checkout the list at the end of this document.

Installation

From your HPC command line, you can install the development version from GitHub with:

$ git clone https://github.com/USCbiostats/slurmR.git
$ R CMD INSTALL slurmR/ 

The second line assumes you have R available in your system (usually loaded via module R or some other command). Or using the devtools from within R:

# install.packages("devtools")
devtools::install_github("USCbiostats/slurmR")

Citation


To cite slurmR in publications use:

  Vega Yon et al., (2019). slurmR: A lightweight wrapper for HPC with
  Slurm. Journal of Open Source Software, 4(39), 1493,
  https://doi.org/10.21105/joss.01493

A BibTeX entry for LaTeX users is

  @Article{,
    title = {slurmR: A lightweight wrapper for HPC with Slurm},
    author = {George {Vega Yon} and Paul Marjoram},
    journal = {The Journal of Open Source Software},
    year = {2019},
    month = {jul},
    volume = {4},
    number = {39},
    doi = {10.21105/joss.01493},
    url = {https://doi.org/10.21105/joss.01493},
  }

Example 1: Computing means (and looking under the hood)

library(slurmR)
#  Loading required package: parallel
#  slurmR default options for `tmp_path` (used to store auxiliar files) set to:
#    /home/george/Documents/slurmR
#  You can change this and checkout other slurmR options using: ?opts_slurmR, or you could just type "opts_slurmR" on the terminal.

# Suppose that we have 100 vectors of length 50 ~ Unif(0,1)
set.seed(881)
x <- replicate(100, runif(50), simplify = FALSE)

We can use the function Slurm_lapply to distribute computations

ans <- Slurm_lapply(x, mean, plan = "none")
#  Warning: [submit = FALSE] The job hasn't been submitted yet. Use sbatch() to submit the job, or you can submit it via command line using the following:
#  sbatch --job-name=slurmR-job-3be51d3ebe15 /home/george/Documents/slurmR/slurmR-job-3be51d3ebe15/01-bash.sh
Slurm_clean(ans) # Cleaning after you

Notice the plan = "none" option, this tells Slurm_lapply to only create the job object, but do nothing with it, i.e., skip submission. To get more info, we can actually set the verbose mode on

opts_slurmR$verbose_on()
ans <- Slurm_lapply(x, mean, plan = "none")
#  --------------------------------------------------------------------------------
#  [VERBOSE MODE ON] The R script that will be used is located at: /home/george/Documents/slurmR/slurmR-job-3be51f05b127/00-rscript.r and has the following contents:
#  --------------------------------------------------------------------------------
#  .libPaths(c("/home/george/R/x86_64-pc-linux-gnu-library/3.6", "/usr/local/lib/R/site-library", "/usr/lib/R/site-library", "/usr/lib/R/library"))
#  Slurm_env <- function (x) 
#  {
#      y <- Sys.getenv(x)
#      if ((x == "SLURM_ARRAY_TASK_ID") && y == "") {
#          return(1)
#      }
#      y
#  }
#  ARRAY_ID         <- as.integer(Slurm_env("SLURM_ARRAY_TASK_ID"))
#  JOB_PATH         <- "/home/george/Documents/slurmR/slurmR-job-3be51f05b127/"
#  INDICES          <- readRDS("/home/george/Documents/slurmR/slurmR-job-3be51f05b127/INDICES.rds")
#  X                <- readRDS(sprintf("/home/george/Documents/slurmR/slurmR-job-3be51f05b127/X_%04d.rds", ARRAY_ID))
#  FUN              <- readRDS("/home/george/Documents/slurmR/slurmR-job-3be51f05b127/FUN.rds")
#  mc.cores         <- readRDS("/home/george/Documents/slurmR/slurmR-job-3be51f05b127/mc.cores.rds")
#  seeds            <- readRDS("/home/george/Documents/slurmR/slurmR-job-3be51f05b127/seeds.rds")
#  set.seed(seeds[ARRAY_ID], kind = NULL, normal.kind = NULL)
#  ans <- parallel::mclapply(
#      X                = X,
#      FUN              = FUN,
#      mc.cores         = mc.cores
#  )
#  saveRDS(ans, sprintf("/home/george/Documents/slurmR/slurmR-job-3be51f05b127/03-answer-%03i.rds", ARRAY_ID), compress = TRUE)
#  --------------------------------------------------------------------------------
#  The bash file that will be used is located at: /home/george/Documents/slurmR/slurmR-job-3be51f05b127/01-bash.sh and has the following contents:
#  --------------------------------------------------------------------------------
#  #!/bin/sh
#  #SBATCH --job-name=slurmR-job-3be51f05b127
#  #SBATCH --output=/home/george/Documents/slurmR/slurmR-job-3be51f05b127/02-output-%A-%a.out
#  #SBATCH --array=1-2
#  #SBATCH --ntasks=1
#  #SBATCH --cpus-per-task=1
#  export OMP_NUM_THREADS=1
#  /usr/lib/R/bin/Rscript --vanilla /home/george/Documents/slurmR/slurmR-job-3be51f05b127/00-rscript.r
#  --------------------------------------------------------------------------------
#  EOF
#  --------------------------------------------------------------------------------
#  Warning: [submit = FALSE] The job hasn't been submitted yet. Use sbatch() to submit the job, or you can submit it via command line using the following:
#  sbatch --job-name=slurmR-job-3be51f05b127 /home/george/Documents/slurmR/slurmR-job-3be51f05b127/01-bash.sh
Slurm_clean(ans) # Cleaning after you

Example 2: Job resubmission

The following example from the package’s manual.

# Submitting a simple job
job <- Slurm_EvalQ(slurmR::WhoAmI(), njobs = 20, plan = "submit")

# Checking the status of the job (we can simply print)
job
status(job) # or use the state function
sacct(job) # or get more info with the sactt wrapper.

# Suppose some of the jobs are taking too long to complete (say 1, 2, and 15 through 20)
# we can stop it and resubmit the job as follows:
scancel(job)

# Resubmitting only 
sbatch(job, array = "1,2,15-20") # A new jobid will be assigned

# Once its done, we can collect all the results at once
res <- Slurm_collect(job)

# And clean up if we don't need to use it again
Slurm_clean(res)

Take a look at the vignette here.

Example 3: Using slurmR and future/doParallel/boot/…

The function makeSlurmCluster creates a PSOCK cluster within a Slurm HPC network, meaning that users can go beyond a single node cluster object and take advantage of Slurm to create a multi-node cluster object. This feature allows then using slurmR with other R packages that support working with SOCKcluster class objects. Here are some examples

With the future package

library(future)
library(slurmR)

cl <- makeSlurmCluster(50)

# It only takes using a cluster plan!
plan(cluster, cl)

...your fancy futuristic code...

# Slurm Clusters are stopped in the same way any cluster object is
stopCluster(cl)

With the doParallel package

library(doParallel)
library(slurmR)

cl <- makeSlurmCluster(50)

registerDoParallel(cl)
m <- matrix(rnorm(9), 3, 3)
foreach(i=1:nrow(m), .combine=rbind) 

stopCluster(cl)

Example 4: Using slurmR directly from the command line

The slurmR package has a couple of convenient functions designed for the user to save time. First, the function sourceSlurm() allows skipping the explicit creating of a bash script file to be used together with sbatch by putting all the required config files on the first lines of an R scripts, for example:

#!/bin/sh
#SBATCH --account=lc_ggv
#SBATCH --partition=scavenge
#SBATCH --time=01:00:00
#SBATCH --mem-per-cpu=4G
#SBATCH --job-name=Waiting
Sys.sleep(10)
message("done.")

Is an R script that on the first line coincides with that of a bash script for Slurm: #!/bin/bash. The following lines start with #SBATCH explicitly specifying options for sbatch, and the reminder lines are just R code.

The previous R script is included in the package (type system.file("example.R", package="slurmR")).

Imagine that that R script is named mybigjob.R, then you use the sourceSlurm function to submit it to Slurm as follows:

slurmR::sourceSlurm("example.R")

This will create the corresponding bash file required to be used with sbatch, and submit it to Slurm.

Another nice tool is the slurmr_cmd(). This function will create a simple bash script that can be used as a command line tool to submit this type of R scripts. Moreover, this command will can add the command to your session’s alias as follows:

library(slurmR)
slurmr_cmd("~", add_alias = TRUE)

Once that’s done, you can simply submit R scripts with “Slurm-like headers” (as shown previously) as follows:

$ slurmr example.R

VS

There are several ways to enhance R for HPC. Depending on what are your goals/restrictions/preferences, you can use any of the following from this manually curated list:

Package

Rerun (1)

apply family (2)

Multinode cluster (3)

Slurm options

Focus on [blank]

System [blank]

Dependencies (4)

Status

drake

yes

-

-

by template

workflows

agnostic

5/9

active

slurmR

yes

yes

yes

on the fly

calls

specific

0/0

active

rslurm

-

-

-

on the fly

calls

specific

1/1

active

future.batchtools

-

yes

yes

by template

calls

agnostic

2/24

active

batchtools

yes

yes

-

by template

calls

agnostic

12/20

active

clustermq

-

-

-

by template

calls

agnostic

6/16

active

[1] After errors, the part or the entire job can be resubmitted.

[2] Functionality similar to the apply family in base R, e.g. lapply, sapply, mapply or similar.

[3] Creating a cluster object using either MPI or Socket connection.

[4] Number of directed/recursive dependencies. As reported in https://tinyverse.netlify.com/status/ (June 4th, 2019)

Please submit an issue or a PR if you find anything off.

Contributing

We welcome contributions to slurmR. Whether it is reporting a bug, starting a discussion by asking a question, or proposing/requesting a new feature, please go by creating a new issue here so that we can talk about it.

Please note that this project is released with a Contributor Code of Conduct (see the CODE_OF_CONDUCT.md file included in this project). By participating in this project you agree to abide by its terms.

Who uses Slurm

Here is a list of educational institutions using Slurm:

Funding

Supported by National Cancer Institute Grant #1P01CA196596.

Computation for the work described in this paper was supported by the University of Southern California’s Center for High-Performance Computing (hpcc.usc.edu).

You can’t perform that action at this time.