Skip to content

Using Slurm to submit jobs on Cannon (Part 2)

Jump to bottom
Bob Yantosca edited this page Feb 2, 2024 · 5 revisions

Submit jobs to the queue

Large jobs (e.g. GEOS-Chem simulations) should be submitted as a batch job. This will run your job in non-interactive mode on one of the computational nodes. using a script.

Set up a run script

Each GEOS-Chem Classic or GCHP run directory that you create will contain sample run scripts that you can modify. Here are some examples:

We recommend specifying multiple partitions by adding this command to the top of your run script:

#SBATCH -p sapphire,huce_cascade,seas_compute,shared,serial_requeue

Your job will start executing in the partition that has availability first.

NOTE: The serial_requeue partition can only be used with jobs of 8 CPU or less. These jobs may be pre-empted if priority demands. This would be an OK choice for short testing runs but not for long production simulations.

Force your job to run on a certain type of CPU

The seas_compute partition consists of nodes with different CPU types. Normally, your job will run on whichever node frees up first. If you wish to restrict your run to a specific CPU type, add one of these commands to the top of your run script:

#SBATCH --constraint=haswell
#SBATCH --constraint=broadwell
#SBATCH --constraint=cascadelake
#SBATCH --constraint=skylake
#SBATCH --constraint=icelake
#SBATCH --constraint=sapphirerapids

This will tell SLURM not to run your job until a node with your desired CPU type frees up. This may result in longer wait times. Also, if there are no nodes of a given CPU type in the seas_compute partition, your job will not run.

  • The former huce_intel partition (now part of seas_compute) contained mostly haswell and broadwell CPUs.
  • The former huce_cascade partition (now part of seas_compute) contained exclusively cascadelake CPUs.

Submit a batch job

You can use the Slurm sbatch command to submit a batch job:

   sbatch geoschem.run

Specify job dependencies

Some jobs may need to be split up into a series of several jobs. To run a set of jobs in a specified order, you will need to make each job dependent on another job so that it won't execute until the earlier job has finished. You can specify job dependencies in SLURM using the --dependency option. For example:

  sbatch geoschem.run.1
  sbatch --dependency=afterok:JOBID geoschem.run.2

The dependency type afterok means that the second job will only begin after the specified JOBID has successfully executed.

Check when your job will start

Adding this tag

#SBATCH --test-only

at the top of your GEOS-Chem run script will invoke SLURM's "dry-run" option. SLURM will not schedule your job, but will instead return its interpretation of the requested resources and an example time for when it would be scheduled to run based on the current queue load. Here is a sample output:

sbatch: Job 40513378 to start at 2018-03-31T15:16:26 using 24 processors on holyjacob01

This option can be very useful in helping to determine your workflow. If your job is not scheduled to start for a long time, you might want to consider starting other jobs that request less memory, cores, and/or time first.

For more information, also see: https://docs.rc.fas.harvard.edu/kb/running-jobs/

Cancel a job

To kill a job that you've submitted, use the scancel command:

   scancel JOBID

Clone this wiki locally