Monitoring and Managing Slurm Jobs

Who this is for: anyone who has submitted jobs and needs to check on them or stop them. Before you start: Running R Interactively with srun (partitions and resource requests) and Running R Batch Jobs with sbatch.

This page covers what happens after you submit a job: how to check on it, how to see what it uses, and how to cancel it.

Checking on jobs

squeue --me                       # all your jobs
squeue --me -o "%i %j %T %r"      # job ID, name, state, and the reason a job is waiting
scontrol show job 12345678        # full details for one job
sinfo -s                          # how busy each partition is

squeue -u $USER is the same as squeue --me; the myq alias from Setting Up Your HPC Environment runs it.

Common states are PD (pending), R (running), and CG (completing). Common reasons for a pending job are:

• Priority or Resources: waiting its turn.
• Dependency: waiting for another job to finish (see --dependency in the sbatch page).
• PartitionTimeLimit: asked for more time than the partition allows. It will never start; cancel and resubmit with a shorter --time.
• JobHeldUser: held.

A job that is no longer listed has finished, successfully or not. Read its log file to find out how it ended.

There is no job history

Slurm accounting is turned off on this cluster, so sacct and seff do not work (sacct prints "Slurm accounting storage is disabled"). A finished job also disappears from scontrol show job about 5 minutes after it ends. So:

• Give every job a log file with --output and read it to see how the job ended.
• For long runs, ask Slurm to email you when the job ends or fails: #SBATCH --mail-type=END,FAIL and #SBATCH --mail-user=<netid>@emory.edu.

Checking memory use

sstat shows what a running job is using:

sstat -j 12345678.batch --format=JobID,MaxRSS,AveCPU     # a single job
sstat -j 12345678_3.batch --format=JobID,MaxRSS,AveCPU   # task 3 of an array

MaxRSS is the peak memory so far. Use it to size --mem for the next run of the same kind of job; memory requests are not enforced here, so a request that is too small can crash other people's jobs (see Running R Interactively with srun).

To watch a running job directly, you can ssh to the node it is running on (the NODELIST column of squeue --me) and run top -u $USER. Do not start any work in that session; anything run outside a Slurm job may be killed.

Canceling jobs

scancel 12345678                  # one job, or a whole job array
scancel 12345678_3                # one task of an array
scancel --name=myproject_step2    # all of your jobs with this name

Always run squeue --me first and cancel by job ID or job name. Avoid scancel -u <netid>, which cancels every job you have, including runs for other projects.

Project workflows add one rule: cancel the workflow's controller job (named <workflow>_controller) first, or it will launch the next step as soon as the current one ends. See Running Project Workflows on the HPC.

After you cancel an R job that used parallel workers, check with top on that node that no R processes of yours are left behind; leftover workers keep using cores that Slurm has handed to someone else.

Next: Running Project Workflows on the HPC.