Snakemake can make use of cluster engines that support shell scripts and have access to a common filesystem, (e.g. Slurm or PBS). There exists a generic cluster support which works with any such engine (see :ref:`cluster-generic`), and a specific support for Slurm (see :ref:`cluster-slurm`). When executing on a cluster, Snakemake implicitly assumes some default resources for all rules (see :ref:`default-resources`).
SLURM is a widely used batch system for
performance compute clusters. In order to use Snakemake with slurm, simply append --slurm
to your Snakemake invocation.
Most SLURM clusters have two mandatory resource indicators for accounting and scheduling, Account and Partition, respectivily. These resources are usually omitted from Snakemake workflows in order to keep the workflow definition independent from the platform. However, it is also possible to specify them inside of the workflow as resources in the rule definition (see :ref:`snakefiles-resources`).
To specify them at the command line, define them as default resources:
$ snakemake --slurm --default-resources slurm_account=<your SLURM account> slurm_partition=<your SLURM partition>
If individual rules require e.g. a different partition, you can override the default per rule:
$ snakemake --slurm --default-resources slurm_account=<your SLURM account> slurm_partition=<your SLURM partition> --set-resources <somerule>:slurm_partition=<some other partition>
Usually, it is advisable to persist such settings via a :ref:`configuration profile <profiles>`, which can be provided system-wide or per user.
Most jobs will be carried out by programs which are either single core scripts or threaded programs, hence SMP (shared memory programs)
in nature. Any given threads and mem_mb
requirements will be passed to SLURM:
rule a:
input: ...
output: ...
threads: 8
resources:
mem_mb=14000
This will give jobs from this rule 14GB of memory and 8 CPU cores.
It is advisable to use resonable default resources, such that you don't need to specify them for every rule.
Snakemake already has reasonable defaults built in, which are automatically activated when using the --default-resources
flag (see above, and also snakemake --help
).
Snakemake's Slurm backend also supports MPI jobs, see :ref:`snakefiles-mpi` for details.
When using MPI with slurm, it is advisable to use srun
as MPI starter.
rule calc_pi:
output:
"pi.calc",
log:
"logs/calc_pi.log",
resources:
tasks=10,
mpi="srun",
shell:
"{resources.mpi} -n {resources.tasks} calc-pi-mpi > {output} 2> {log}"
Note that the -n {resources.tasks}
is not necessary in case of SLURM, but it should be kept in order to allow execution of the workflow on other systems, e.g. by replacing srun
with mpiexec
:
$ snakemake --set-resources calc_pi:mpi="mpiexec" ...
A workflow rule may support a number of :ref:`resource <snakefiles-resources>` specification. For a SLURM cluster, a mapping between Snakemake and SLURM needs to be performed.
You can use the following specifications:
SLURM Resource | Snakemake resource | Background Information |
---|---|---|
-p /--partition |
slurm_partition |
the partition a rule/job is to use |
-t /--time |
runtime |
the walltime per job in minutes |
-C /--constraint |
constraint |
may hold features on some clusters |
--mem |
mem_mb |
memory in MB a cluster node must provide |
--mem-per-cpu |
mem_mb_per_cpu |
memory per reserved CPU |
-n /--ntasks |
tasks |
number of concurrent tasks / ranks |
-c /--cpus-per-task |
cpus_per_task |
number of cpus per task (in case of SMP, rather use threads ) |
-N /--nodes |
nodes |
number of nodes |
Each of these can be part of a rule, e.g.:
rule:
input: ...
output: ...
resources:
partition: <partition name>
runtime: <some number>
Please note: as --mem
and --mem-per-cpu
are mutually exclusive on SLURM clusters, there corresponding resource flags mem_mb
and mem_mb_per_cpu
are mutually exclusive, too.
You can only reserve memory a compute node has to provide or the memory required per CPU (SLURM does not make any distintion between real CPU cores and those provided by hyperthreads).
SLURM will try to sastify a combination of mem_mb_per_cpu
and cpus_per_task
and nodes
, if nodes
is not given.
Note that it is usually advisable to avoid specifying SLURM (and compute infrastructure) specific resources (like constraint
) inside of your workflow because that can limit the reproducibility on other systems.
Consider using the --default-resources
and --set-resources
flags to define such resources on the command line.
SLURM installations can support custom plugins, which may add support for additional flags to sbatch
.
In addition, there are various sbatch
options not directly supported via the resource definitions shown above.
You may use the slurm_extra
resource to specify additional flags to sbatch
:
rule:
input: ...
output: ...
resources:
slurm_extra="--qos=long --mail-type=ALL --mail-user=<your email>"
To use the generic cluster support, Snakemake simply needs to be given a submit command that accepts a shell script as first positional argument:
$ snakemake --cluster qsub --jobs 32
Here, --jobs
denotes the number of jobs submitted to the cluster at the same time (here 32).
The cluster command can be decorated with job specific information, e.g.
Note
Consider to :ref:`group jobs <snakefiles-grouping>` in order to minimize overhead, in particular for short-running jobs.
$ snakemake --cluster "qsub {threads}"
Thereby, all keywords of a rule are allowed (e.g. rulename, params, input, output, threads, priority, resources, ...).
For example, you could encode the expected running time in minutes into a :ref:`resource <snakefiles-resources>` runtime_min
:
rule:
input:
...
output:
...
resources:
runtime_min=240
shell:
...
and forward it to the cluster scheduler:
$ snakemake --cluster "qsub --runtime {resources.runtime}"
In order to avoid specifying runtime_min
for each rule, you can make use of the --default-resources
flag, see snakemake --help
.
If your cluster system supports DRMAA, Snakemake can make use of that to control jobs.
With DRMAA, no qsub
command needs to be provided, but system specific arguments can still be given as a string, e.g.
$ snakemake --drmaa " -q username" -j 32
Note that the string has to contain a leading whitespace. Else, the arguments will be interpreted as part of the normal Snakemake arguments, and execution will fail.
Adapting to a specific cluster can involve quite a lot of options. It is therefore a good idea to setup a :ref:`a profile <profiles>`.
Note
If there are more than 100 input and/or output files for a job, None
will be used instead of listing all values. This is to prevent the jobscript from becoming larger than Slurm jobscript size limits.
When executing a workflow on a cluster using the --cluster
parameter (see below), Snakemake creates a job script for each job to execute. This script is then invoked using the provided cluster submission command (e.g. qsub
). Sometimes you want to provide a custom wrapper for the cluster submission command that decides about additional parameters. As this might be based on properties of the job, Snakemake stores the job properties (e.g. name, rulename, threads, input, output, params etc.) as JSON inside the job script (for group jobs, the rulename will be "GROUP", otherwise it will be the same as the job name). For convenience, there exists a parser function snakemake.utils.read_job_properties that can be used to access the properties. The following shows an example job submission wrapper:
#!python
#!/usr/bin/env python3
import os
import sys
from snakemake.utils import read_job_properties
jobscript = sys.argv[1]
job_properties = read_job_properties(jobscript)
# do something useful with the threads
threads = job_properties[threads]
# access property defined in the cluster configuration file (Snakemake >=3.6.0)
job_properties["cluster"]["time"]
os.system("qsub -t {threads} {script}".format(threads=threads, script=jobscript))