HPC Rules Storage and Getting Help

Who this is for: everyone who uses the RSPH cluster. Read this before you run anything there.

RSPH IT publishes the official HPC Getting Started Guide (V2.0, revised 11/3/23). By using the cluster you agree to the policies in its "Important Cluster Policies" section (pages 3 and 4). This page summarizes those policies, adds the rules our lab follows on top of them, and covers storage and how to get help. Where the official guide and this wiki differ, follow the wiki; a list of outdated items in the guide is at the bottom of this page.

Cluster rules

1. Never run R or any other computation on the login node. clogin01 is the shared entry point for every RSPH user. Use it only to log in, move files (scp, rsync, git), edit small text files, and run Slurm commands (sbatch, squeue, scancel, sinfo). Everything else goes on a compute node through srun or sbatch, including starting R "just to check something", reading an .rds file to look at a value, and installing R packages. In our setup R on the login node is also unreliable: the login node runs Rocky Linux 8.9 and the compute nodes run 8.10, so renv keeps a separate package library for each, and packages installed from the login node land in a library that your jobs never read. R can appear to work there while doing the wrong thing.
2. All computation runs inside a Slurm job. RSPH staff may kill processes that are not part of a Slurm job, without notice. That includes R started in an ssh session to a compute node and R worker processes left behind after a job ends.
3. Ask for what your job needs. Request the cores, memory, and time your job will use. Ask for a whole node (all 32 cores and --mem=0) only when your code uses all of it, such as full-size network estimation. Memory limits are not enforced on this cluster, so a job that uses more than it asked for can crash other people's jobs. See Running R Interactively with srun.
4. Do not share your account. Each lab member has their own login. Never share passwords, SSH private keys, or GitHub tokens, and keep tokens in files only you can read (see Protecting your tokens).
5. Do not run servers on the cluster, such as web servers, RStudio Server, or Jupyter. This includes remote editors such as VS Code or Positron Remote-SSH: they start a server process on the host they connect to, which would be the login node. We edit code on our own computers and use the cluster only for running jobs.
6. Nothing is backed up. See Backups and snapshots.
7. Use only data you are allowed to use there. The cluster is not certified HIPAA compliant, so do not bring identifiable data onto it. Data under a data use agreement (for example, ARTnetData) may be used only as the agreement allows, and the PI is responsible for compliance, so ask Sam before moving any new data set to the cluster.
8. Cancel only your own jobs, by job ID or job name. Avoid scancel -u <netid>, which cancels every job you own, including unrelated runs, and never use kill or pkill on processes you did not start. See Monitoring and Managing Slurm Jobs.
9. Tell RSPH IT before anything unusual, such as copying hundreds of GB through the login node.

Attempts to get around the cluster's login or security features end cluster access.

Where to put files

Location	Use it for	Limit	Purge	Snapshots
/home/<netid>	Settings files, SSH keys, small configs, your R package library	25 GB per user	none	last 2 nights
/projects/epimodel/<netid>	Project repositories and simulation output	5 TB shared by the whole lab	renewed yearly by the PI	none
/scratch	Large temporary files during a run (rarely needed)	100 GB default, more on request to RSPH IT	files older than 2 weeks may be deleted	none

All three sit on the same Panasas file system and perform the same; only the policies differ. Work in /projects/epimodel/<netid>, and keep simulation output out of your home directory.

Be gentle with shared storage

• Do not watch progress by repeatedly listing large output folders; that puts a heavy load on the shared file system. Ask Slurm instead (squeue --me), and read log files. If you must list or search a big folder, do it rarely and use find <dir> -maxdepth 1.
• Do not save what you will not use. For scenario runs that you only read through merged tibbles, set save.run = FALSE in control_msm() (see Running Project Workflows on the HPC).
• When a project ends, delete large raw output from /projects/epimodel/<netid>; the 5 TB is shared.

Checking your storage quota

pan_quota reads usage totals directly from the storage system. Prefer it over du, which walks every file and is slow and hard on the shared storage for big directories.

pan_quota                               # your home directory (25 GB limit)
pan_quota /projects/epimodel            # your own usage inside the lab space
pan_quota -G /projects/epimodel         # the lab's shared limit in /projects/epimodel

Example output of the last command (10/12/26):

     <GB>  <soft>  <hard> : <files>    <soft>    <hard> :   <path to volume> <pan_identity(name)>
  2078.26 4700.00 5000.00 :  818328 unlimited unlimited : /projects/epimodel gid:3029(epimodel)

The first number is current usage in GB. At <soft> the system starts emailing warnings; at <hard> writes fail, so jobs die when they try to save output, and a full home directory can break your shell and editors. If the lab is close to its limit, tell Sam before you launch a large run.

Backups and snapshots

Nothing on the cluster is backed up. RSPH keeps snapshots of /home from the last two nights, so a file deleted from your home directory can be recovered for about a day. List the snapshots, then copy the file back from the one you want:

ls /home/.snapshot/
cp /home/.snapshot/<snapshot name>/<netid>/path/to/file ~/path/to/file

/projects and /scratch have no snapshots. A file deleted or overwritten in /projects/epimodel is gone. So:

• Keep all code in GitHub. Commit and push from your own computer; the copy on the cluster is only a clone.
• Copy the results you need (merged tibbles, processed CSVs, figures) to your computer with scp or rsync as soon as a run finishes (see Working with Files on the HPC). Keep results worth keeping long term on your computer and in the lab OneDrive. Only small outputs, such as final tables, belong in git.

Protecting your tokens

Home directories on the cluster are readable by other cluster users by default, and new files are created readable by everyone. Your ~/.Renviron holds your GitHub token, so make it private as soon as you create it:

chmod 600 ~/.Renviron
ls -l ~/.Renviron     # should start with -rw-------

Do not put tokens in ~/.bashrc, and do not clone with a token in the URL (https://<token>@github.com/...): the token is then saved in plain text in .git/config, which every lab member can read. Use an SSH key for GitHub instead (see Using Git on the HPC). If you think a token was exposed, revoke it on GitHub and make a new one.

If your connection drops

Jobs you submitted with sbatch keep running if your VPN or ssh connection drops. An interactive srun session ends with the connection. To keep a monitoring window open across disconnects, run tmux on the login node and reattach with tmux attach after you reconnect. Use it only for watching the queue and logs, not for running R.

Getting help

• For our lab workflow (Spack, renv, slurmworkflow, EpiModel, project code), ask in #coding-help on the lab Slack. Someone in the lab has usually seen the problem before.
• For cluster problems that are not about our code (password resets, account or VPN access, a node that seems broken, storage errors, jobs stuck for reasons you do not understand), email RSPH IT at help@sph.emory.edu and start the subject line with RSPH HPC cluster:, for example RSPH HPC cluster: jobs on node67 fail with I/O errors. The prefix routes the ticket to the right team. Include your NetID, the job IDs, the partition, and the exact error message, and copy Sam on anything about the lab's partitions or storage.

Leaving the lab

Your account is sponsored by Sam and stays active while you work with the lab. Before you leave: push all code to GitHub, copy any results you need, tell Sam which of your folders under /projects/epimodel/<netid> can be deleted, and delete any tokens from your cluster files. Sam will ask RSPH IT to close the account.

Learning more

• The official RSPH HPC Getting Started Guide, and the RSPH-HPC/Documentation repository for newer versions
• The Linux Command Line, a free book for learning the shell
• The Slurm quick start guide

Out-of-date items in the official guide (as of 10/12/26)

• There is no preemptable partition, and other groups' partitions are closed to us. Current partitions and limits are in Running R Interactively with srun.
• module load R gives R 4.2.2 by default. We load R with Spack instead (see Using R via Spack on the HPC).
• In the example job scripts, the long dashes must be typed as --, and output folders such as Out/ must exist before you submit.
• /apps/bin/python3 does not exist.
• sacct and seff do not work, because Slurm accounting is turned off on this cluster. See Monitoring and Managing Slurm Jobs for what to use instead.

Next: Working with Files on the HPC.