Running Project Workflows on the HPC

Who this is for: anyone running project simulations on the HPC. Before you start: the HPC setup and Slurm basics pages, and a project that uses renv (Using renv for Reproducible R Environments).

This page explains how we run EpiModel simulations on the RSPH cluster with three of our packages: slurmworkflow chains Slurm jobs into a workflow, EpiModelHPC adds ready-made steps for EpiModel (restore packages, run scenarios, merge results), and swfcalib builds calibration workflows on top of both.

Why workflows

A real analysis is a chain of jobs: restore packages, run a few thousand simulations as an array, merge the outputs, make tables. With plain sbatch you would wait for each stage and submit the next one by hand. A workflow does the chaining for you: you start it once and it runs to the end, sending you an email if something fails and when it is done.

Key ideas

• Built locally, run remotely. You write an R script on your laptop that describes the workflow. Running it creates a folder workflows/<name>/ with shell scripts and small .rds files. You copy that folder to the HPC and start it there.
• Steps. A workflow is an ordered list of steps. Each step is one sbatch submission (a single job or a job array), created with a step template and its own sbatch options.
• Controller. start_workflow.sh submits a tiny "controller" job. The controller submits step 1; when a step ends, it resubmits the controller, which submits the next step. In squeue you will see <name>_controller, <name>_step1, <name>_step2, and so on.
• Logs. Every job writes to workflows/<name>/log/. <name>_controller.out lists what was submitted; each step writes <name>_step<n>_<jobid>_<task>.out (array tasks are numbered from 0 in log names; single jobs end in _4294967294.out, Slurm's code for "not an array").
• Clean environment. Steps run with sbatch --export=NONE, so nothing from your login shell is passed to them. Everything a step needs (loading Spack and R) goes in its setup_lines.
• Working directory. Steps run from the folder where you started the workflow. Always start it from the project root, so relative paths work and .Rprofile activates renv.

Step templates you will use

Template	Package	What the step does
step_tmpl_bash_lines()	slurmworkflow	Runs a few shell lines
step_tmpl_rscript()	slurmworkflow	Runs an R script
step_tmpl_do_call()	slurmworkflow	Calls an R function with a list of arguments
step_tmpl_do_call_script()	slurmworkflow	Runs an R script with some variables predefined
step_tmpl_map(), step_tmpl_map_script()	slurmworkflow	Like Map(): one array task per set of arguments
step_tmpl_renv_restore()	EpiModelHPC	Checks the git branch, runs git pull, then renv::restore()
step_tmpl_netsim_scenarios()	EpiModelHPC	Runs n_rep simulations of each scenario as an array of batches
step_tmpl_merge_netsim_scenarios_tibble()	EpiModelHPC	Merges the simulation files into one tibble per scenario

A function passed to step_tmpl_do_call() or step_tmpl_map() runs in a fresh R session on the node: it can only use its arguments and what it loads itself. A script used with step_tmpl_do_call_script() must not start with rm(list = ls()), because that deletes the variables the step passes in.

A first workflow (toy example)

Three steps: one R call that prints where it runs, an array of two tasks, and a final step that combines their outputs. It takes a few minutes and uses 1 CPU per job.

One-time setup

On your laptop, install slurmworkflow (it is on GitHub, not CRAN):

install.packages("remotes")
remotes::install_github("EpiModel/slurmworkflow")

On the HPC, the toy steps also need slurmworkflow in your personal R library. Install it in a small batch job, not on the login node. Save this as install-swf.sbatch in /projects/epimodel/<netid>/, run sbatch install-swf.sbatch from there, and wait until install-swf_<jobid>.out ends with * DONE (slurmworkflow) (about a minute):

#!/bin/bash
#SBATCH --job-name=install-swf
#SBATCH --partition=short-cpu
#SBATCH --cpus-per-task=2
#SBATCH --mem=4G
#SBATCH --time=00:30:00
#SBATCH --output=install-swf_%j.out

. /projects/epimodel/spack/share/spack/setup-env.sh
spack unload -a
spack load r@4.6.1
Rscript -e '
lib <- path.expand(Sys.getenv("R_LIBS_USER"))
dir.create(lib, recursive = TRUE, showWarnings = FALSE)
.libPaths(c(lib, .libPaths()))
install.packages("remotes", lib = lib, repos = "https://cloud.r-project.org")
remotes::install_github("EpiModel/slurmworkflow", lib = lib)
'

Each R version has its own personal library, so rerun this job if you switch to a different R version. (Real projects skip this: there, slurmworkflow is in the project's renv library.)

Build the workflow on your laptop

Make a folder for the toy project, open R there, and run:

library(slurmworkflow)

setup_lines <- c(
  ". /projects/epimodel/spack/share/spack/setup-env.sh",
  "spack unload -a",
  "spack load r@4.6.1"
)

wf <- create_workflow(
  wf_name = "swf_demo",
  default_sbatch_opts = list(
    "partition" = "epimodel,short-cpu",
    "mail-type" = "FAIL",
    "mail-user" = "<netid>@emory.edu"
  )
)

# Step 1: one R call on a compute node
wf <- add_workflow_step(
  wf_summary = wf,
  step_tmpl = step_tmpl_do_call(
    what = function() {
      cat("Step 1 on node:", Sys.info()[["nodename"]], "\n")
      cat("Working directory:", getwd(), "\n")
      print(sessionInfo())
    },
    args = list(),
    setup_lines = setup_lines
  ),
  sbatch_opts = list("cpus-per-task" = 1, "mem" = "2G", "time" = "00:10:00")
)

# Step 2: an array of 2 tasks (task 1 is slower on purpose)
wf <- add_workflow_step(
  wf_summary = wf,
  step_tmpl = step_tmpl_map(
    FUN = function(task_id, sleep_s, out_dir) {
      Sys.sleep(sleep_s)
      dir.create(out_dir, showWarnings = FALSE, recursive = TRUE)
      res <- data.frame(
        task = task_id,
        node = Sys.info()[["nodename"]],
        finished = format(Sys.time())
      )
      saveRDS(res, file.path(out_dir, paste0("task_", task_id, ".rds")))
    },
    task_id = 1:2,
    sleep_s = c(120, 5),
    MoreArgs = list(out_dir = "data/run/swf_demo"),
    setup_lines = setup_lines
  ),
  sbatch_opts = list("cpus-per-task" = 1, "mem" = "2G", "time" = "00:10:00")
)

# Step 3: combine the results (last step, so also mail at the end)
wf <- add_workflow_step(
  wf_summary = wf,
  step_tmpl = step_tmpl_do_call(
    what = function(out_dir) {
      files <- list.files(out_dir, pattern = "^task_.*rds$", full.names = TRUE)
      res <- do.call(rbind, lapply(files, readRDS))
      print(res)
      write.csv(res, file.path(out_dir, "summary.csv"), row.names = FALSE)
    },
    args = list(out_dir = "data/run/swf_demo"),
    setup_lines = setup_lines
  ),
  sbatch_opts = list(
    "cpus-per-task" = 1, "mem" = "2G", "time" = "00:10:00",
    "mail-type" = "END,FAIL"
  )
)

You now have workflows/swf_demo/. create_workflow() refuses to overwrite an existing folder, so delete workflows/swf_demo/ before rebuilding.

Copy it to the HPC and start it

From a terminal on your laptop, in the toy project folder:

ssh sph "mkdir -p /projects/epimodel/<netid>/swf-demo/workflows"
ssh sph "rm -rf /projects/epimodel/<netid>/swf-demo/workflows/swf_demo"
scp -r workflows/swf_demo sph:/projects/epimodel/<netid>/swf-demo/workflows/
ssh sph "cd /projects/epimodel/<netid>/swf-demo && ./workflows/swf_demo/start_workflow.sh"

start_workflow.sh must be executed from the project root as shown, not sourced. If it came from Windows and is not executable, run chmod +x workflows/swf_demo/start_workflow.sh on the HPC first.

Watch it and read the results

ssh sph 'squeue -u $USER -r -o "%.12i %.28j %.9T %.8M %R"'
ssh sph "cat /projects/epimodel/<netid>/swf-demo/workflows/swf_demo/log/swf_demo_controller.out"
ssh sph "cat /projects/epimodel/<netid>/swf-demo/data/run/swf_demo/summary.csv"

The controller log should end with All steps are done, and summary.csv should have a header and two rows (tasks 1 and 2). Step 3 starts only after both array tasks finish, even though task 2 ends almost two minutes earlier.

How this maps onto a real project (EpiModelHIV-Template)

Projects created from the EpiModelHIV-Template use the same mechanism:

• R/hpc_configs.R defines hpc_node_setup (the three Spack lines, used as setup_lines everywhere) and make_em_workflow(wf_name). That function (1) refreshes renv.lock.hpc from your local library, (2) creates the workflow with the lab's partition list and your HPC_MAIL_USER from the project .Renviron, and (3) adds step 1, step_tmpl_renv_restore(), which checks that the HPC checkout is on your branch, runs git pull, and restores renv.lock.hpc on a compute node. The branch check uses the git branch your local project is on when you build the workflow, so keep the HPC copy of the project on that same branch. (Projects created from the Template before October 2026 set current_git_branch in R/hpc_configs.R by hand; the Template used to ship "dev_v3.3", which stops the restore step with "The git branch is not dev_v3.3" until you change it.)
• workflow-*.R scripts (one per chapter, for example R/D-interventions/workflow-intervention.R) call make_em_workflow() and add the real steps: step_tmpl_netsim_scenarios(), step_tmpl_merge_netsim_scenarios_tibble(), and step_tmpl_do_call_script() for processing. Because the restore is step 1, simulations are usually step 2.
• Two ways files reach the HPC. Code and renv.lock.hpc go through git (you push; step 1 pulls). The workflows/ folder and large inputs such as data/run/estimates/netest-hpc.rds are git-ignored, so they go by scp. A run needs both.

A deploy is the same sequence every time. Many projects keep it in a deploy-<name>.sh file run line by line in the RStudio terminal:

# 1. build the workflow (this also refreshes renv.lock.hpc)
Rscript -e 'source("R/D-interventions/workflow-intervention.R")'

# 2. commit and push the code and lockfile the run needs (list files explicitly)
git add renv.lock.hpc R/hpc_configs.R R/D-interventions/workflow-intervention.R
git commit -m "Update interventions workflow"
git push

# 3. update the HPC copy of the project (first time: git clone over SSH)
ssh sph "cd /projects/epimodel/<netid>/<Project> && git pull --ff-only"

# 4. copy git-ignored inputs, replace the workflow folder, start it
scp data/run/estimates/restart-hpc.rds sph:/projects/epimodel/<netid>/<Project>/data/run/estimates/
ssh sph "rm -rf /projects/epimodel/<netid>/<Project>/workflows/interventions && mkdir -p /projects/epimodel/<netid>/<Project>/workflows"
scp -r workflows/interventions sph:/projects/epimodel/<netid>/<Project>/workflows/
ssh sph "cd /projects/epimodel/<netid>/<Project> && ./workflows/interventions/start_workflow.sh"

For calibration and restart points, see Calibrating a Model and Making a Restart Point.

Monitoring, cancelling, and getting results

• Status: squeue --me (add -r to list each array task) and the logs in workflows/<name>/log/. sacct is not available on this cluster (accounting is disabled), so the logs and the Slurm emails are your only record once a job leaves the queue. See Monitoring and Managing Slurm Jobs.

• Cancel only your workflow, by name, controller first. If you cancel only the running step, the waiting controller wakes up and submits the next step. Cancel both:

  scancel --name=<name>_controller
  scancel --name=<name>_step2

  Never run scancel -u $USER: it cancels every job you have, including other workflows.

• Results: after the END email, copy the small processed outputs (merged tibbles, summary CSVs) to your laptop, for example scp "sph:/projects/epimodel/<netid>/<Project>/data/run/<name>/<name>_summary.csv" data/run/<name>/. Leave raw simulation files on the HPC.

Things that go wrong (read before your first real run)

1. Never run R on the login node. Use workflows, sbatch, or srun. The login node (Rocky 8.9) and compute nodes (Rocky 8.10) also keep separate renv libraries, so packages installed on the login node are not the ones your jobs load (see HPC Rules, Storage, and Getting Help).
2. Push before you launch. Step 1 pulls from GitHub. If you forget to push your code or renv.lock.hpc, the HPC runs the old version.
3. Replace the workflow folder; do not copy on top of it. rm -rf the old workflows/<name>/ on the HPC before scp -r. Leftover step folders from an older version would still be run.
4. GitHub access from the HPC. git pull in step 1 needs an SSH key on the HPC registered with GitHub (it runs in a batch job where nobody can type a passphrase). Installing EpiModelHIV-p and ARTnetData needs GITHUB_PAT in ~/.Renviron on the HPC (chmod 600 ~/.Renviron; see Using Git on the HPC). Without the token the restore builds every other package and then rolls them all back.
5. A failed array task does not always stop the workflow. If any task except the last fails, the next step still runs on incomplete results. If the last task fails, the workflow stops silently (you only get the FAIL email). Check the step logs before trusting merged results.
6. Save only what you need. Scenario and intervention campaigns should use save.run = FALSE in control_msm(); keep save.run = TRUE only for calibration runs that feed a restart point. On a 100k-node model the saved $run state is about 98% of each file.
7. Go easy on the shared storage. Monitor with squeue, not by repeatedly listing output folders with hundreds of large files. If you must look, use find <dir> -maxdepth 1 -name 'sim__*.rds' | wc -l once at the end. Delete raw simulation files you no longer need; /projects has no backups and its 5 TB is shared.
8. Array size. max_array_size can be up to 1000 here (MaxArraySize = 1001); larger arrays are split into chunks automatically.

Further reading

• slurmworkflow vignette: the full API, loops with change_next_workflow_step(), and start_workflow.sh -s <step> / -d <dir>. Its paths and R version are examples, not our setup.
• EpiModelHPC article: a longer walk-through of the same ideas. Where it differs from the Template, follow the Template.
• Running Network Models with Checkpoints: protecting long simulations against interruption.

Next: Running Network Models with Checkpoints.