Skip to content

intro_to_slurm

Jump to bottom
Ryan edited this page Dec 4, 2025 · 21 revisions

Getting started

  1. Request an account
  2. Go to the Submit a HPSC Request link and ask for a new project (this may only be performed by PIs). By default this will produce a 1TB directory in addition to the designated home and scratch directories. If more space is required consider submitting additional help tickets.
  3. Request access to the following partitions:
  • condo-ut-genomics (QOS: genomics)
  • condo-utia (QOS: condo)

Other initial good things to do

The new /lustre/isaac24 storage might require users to update some aliases, such as scratch directory.

# add to your ~/.bashrc
export SCRATCHDIR="/lustre/isaac24/scratch/<your username goes here>"

QOS and partitions

Running jobs documentation

Opportunistic partitions as of November 2024:

Name Priority MaxTRES MaxWall MaxTRESPU MaxJobsPU MaxSubmitPU
campus 1 node=6 1-00:00:00 node=6 48 96
campus-gpu 1 node=2 1-00:00:00 gres/gpu=4 3 6
condo 10 30-00:00:00 150 250
long 2 6-00:00:00 cpu=96 12 18
campus-bigmem 1 node=1 1-00:00:00 node=1 4 8
long-bigmem 2 node=1 6-00:00:00 node=1 2 4
genomics 2 node=1 7-00:00:00 node=1 2 3
short 1 03:00:00 cpu=64 12 18

Beginner resources with practical examples

2024 HPSC Fundamentals Workshop
2024 EPP 622 ISAAC intro
2024 EPP 622 ISAAC cont.
Globus Collections

General questions with bioinformatics tools:

How to decide which settings to use for a job:

  • Check documentation for recommendations on num CPUs, memory, etc
  • Check syntax, run on short qos
  • Estimate start time sbatch --test
  • How to choose a particular qos/partition
  • Parsing common slurmstepd error messages and log files

What resources do I have access to?

This command will list all of your projects and which

sacctmgr -p show assoc

How can I learn more about the resources?

If you want to learn more about a given qos governing a partition, you can use sacctmgr to learn more:

sacctmgr show qos where name=campus

You can use a comma-separated list of qos to return multiple qos summaries and edit the format of the output:

sacctmgr show qos where name=campus,short format=Name,MaxWall

You can also return a summary of the entire partition using scontrol.

scontrol show PartitionName=campus

You can also get an enormous list of partitions on the cluster:

showpartitions

But what if we want to learn a bit more about the actual nodes underlying these partitions?

sinfo

The OIT HPSC ISAAC-NG Running Jobs link has some very helpful additional commands, including some custom to ISAAC.

isaac-sinfo | grep gpu | egrep 'idle|mixed'

Which partition will work the fastest for my job?

Node status overview

If we want to see a summary of the nodes and the CPUs on each that are Allocated, Idle, Other, Total (A/I/O/T).

sinfo --Format statecompact,cpusstate,partition,nodelist --partition=campus,short

Empirical test

Make a new sbatch file called test_short.sbatch with the following content:

#!/bin/bash
#SBATCH -J bwa
#SBATCH --nodes=1
#SBATCH --cpus-per-task=1
#SBATCH -A ACF-UTK0011
#SBATCH -p short
#SBATCH -q short
#SBATCH -t 01:00:00

sleep 600

Note

Don't run this with sbatch just yet!

Now make another file called test_campus.sbatch. It might be helpful to just use cp from the previous file.

#!/bin/bash
#SBATCH -J bwa
#SBATCH --nodes=1
#SBATCH --cpus-per-task=1
#SBATCH -A ACF-UTK0011
#SBATCH -p campus
#SBATCH -q campus
#SBATCH -t 01:00:00

sleep 600

Note

Don't run this with sbatch just yet!

Now, using the --test-only flag, let's see which partition will run first with the exact same resource requests:

sbatch --test-only test_short.sbatch
sbatch --test-only test_campus.sbatch

CHALLENGE

Pick out a few more partitions/qos combinations (remember you can see your partitions using sacctmgr command above). Using the sbatch --test-only approach, which partition is going to take the absolute longest to begin?

How can I learn more about my running or pending job(s)?

The basic way to see jobs in the queue is:

squeue -u <yourusername>

To get a nice summary of all of your currently running jobs including job id (i), partition (P), job name (j), cpus (C), memory (m), and qos (q).

squeue -o "%i %P %j %C %m %q" --me

Get a lot of information about a job in the queue.

scontrol show job <job_id>

Modify the sbatch header to create explicit log files for stdout and stderr. You can also modify --mail-type to include "BEGIN".

#SBATCH --error=job.e%J
#SBATCH --output=job.o%J
#SBATCH --mail-type=END,FAIL
#SBATCH --mail-user=<YOUR EMAIL>@<YOUR EMAIL>

Clone this wiki locally