Caltech central cluster

Sample sbatch script

Here is a sbatch script template for setting up simulations using multiple CPUs on Caltech central cluster. Customize the SBATCH instructions if you need more memory/GPUs.

#!/bin/bash
#SBATCH --ntasks=32
#SBATCH --job-name=[YOUR_JOB_NAME]
#SBATCH --time=2:00:00
#SBATCH --partition=expansion  # or gpu

export MODULEPATH="/groups/esm/modules:$MODULEPATH"
module purge
module load climacommon

CA_PATH=[YOUR_CLIMAATMOS_PATH]
CA_EXAMPLE=$CA_PATH'examples/'
DRIVER=$CA_EXAMPLE'hybrid/driver.jl'
CONFIG_FILE=[YOUR_CONFIG_FILE]

export OPENBLAS_NUM_THREADS=1
export JULIA_NVTX_CALLBACKS=gc
export JULIA_MAX_NUM_PRECOMPILE_FILES=100
export JULIA_LOAD_PATH="${JULIA_LOAD_PATH}:${CA_PATH}.buildkite"
export CLIMACORE_DISTRIBUTED="MPI"
export SLURM_KILL_BAD_EXIT=1

julia --project=$CA_EXAMPLE -e 'using Pkg; Pkg.instantiate()'
julia --project=$CA_EXAMPLE -e 'using Pkg; Pkg.precompile()'
julia --project=$CA_EXAMPLE -e 'using Pkg; Pkg.status()'

srun julia --project=$CA_EXAMPLE $DRIVER --config_file $CONFIG_FILE

If you have any LocalPreferences.toml file, remove it.