Skip to content

Harvard ACMG resources on Cannon

Jump to bottom
Bob Yantosca edited this page Apr 6, 2026 · 97 revisions

This page summarizes resources available to Harvard-ACMG group members on the FAS Research Computing Cannon (FASRC) cluster. See also the Cluster Storage page on the FASRC documentation site.

Partitions on Cannon

Harvard-ACMG group members can submit computational jobs to the following partitions:

You can request multiple partitions by adding a command such as these into your run script:

#SBATCH -p sapphire,huce_cascade,seas_compute,shared                     # or
#SBATCH -p sapphire,huce_cascade,seas_compute,shared,serial_requeue

Your job will start in whichever partition first has free resources.

For interactive jobs (i.e. compiling, plotting, etc.), we recommend using the test partition. You can use up to:

  • Up to 5 concurrent jobs per user
  • Up to 12 hours time.
  • Up to 96 total cores per user
  • Up to 384 total GB memory per user.

Storage space

Summary of storage locations

Location Description Quota Suitable for Is data purged?
$HOME Home directory 100 GB / user Editing; Data processing; Storage; Tasks that do not generate large amounts of data no
/n/netscratch/jacob_lab Netscratch 150 TB Running GCClassic or GCHP; Data analysis; Temporary storage after 90 days
/n/holylfs05/LABS/jacob_lab Holylfs05 245 TB Running GCClassic or GCHP; Performing inversions with the IMI; Data analysis; Longer-term storage no
/n/holylfs06/LABS/jacob_lab2 Holylfs06 250 TB Running GCClassic or GCHP; Performing inversions with the IMI; Data analysis; Longer-term storage no
/n/jacob_lab Archival storage 40 TB Long-term (archival) data storage for completed projects no

Home directory

The /n/home* directories are your primary storage location for analysis and data processing. Your home directory is the initial working directory upon login and can always be returned to by typing cd ~ or cd $HOME. Every Cannon user has a 100 GB size limit in their home directory. This directory is backed up daily. You can find the address of another user's home directory by typing

$ file ~username

filling in the username of the person whose directory you are looking for.

You can check the quota for your home directory, including how much space you are using, with the following command from anywhere on Cannon:

$ quota .

Netscratch

ACMG members have access to 150 TB (total) of shared scratch space in the directory:

/n/netscratch/jacob_lab

Which is also reachable via:

$SCRATCH/jacob_lab/

As of December 2024, the system-defined environment variable $SCRATCH points to /n/netscratch/. This could change in the future, so we recommend using $SCRATCH in your data paths for consistency.

The $SCRATCH/jacob_lab disk is subdivided into two folders, Everyone and Lab, as described below:

Directory Contents are visible to Purpose Purged after
$SCRATCH/jacob_lab/Everyone All Cannon users Sharing data with other users on Cannon 90 days
$SCRATCH/jacob_lab/Lab Only to jacob_lab users Generating your own research data 90 days

Create your own folders under Everyone and Lab using these commands:

$ mkdir -p $SCRATCH/jacob_lab/Everyone/$USER
$ mkdir -p $SCRATCH/jacob_lab/Lab/$USER

where the $USER environment variable contains your Cannon username.

We recommend to store research in progress under in your $SCRATCH/jacob_lab/Lab/$USER folder, which is restricted to members of jacob_lab only. This will provide an extra measure of security for your unpublished research.

There is a 90 day time limit for files on the $SCRATCH disk space. However, if the $SCRATCH disk starts to approach 95% of its storage capacity, Research Computing will begin deleting files that are older than 90 days. This space is not backed up.

Currently there is not a user-facing command to check the quota on the $SCRATCH/jacob_lab directory structure. We hope this tool will be available in the future. But you can check the disk usage of an individual directory (and its subdirectories) with this command:

$ du -sh *

You can determine the age of a file or directory with this command:

$ echo $((($(date +%s) - $(date +%s -r "FILENAME")) / 86400)) days

where FILENAME is the file or directory that you wish to test. This may help you to know when your files on scratch are getting close to the 90-day limit.

Holylfs05

ACMG members have access to 145 TB of scratch and/or storage space in the directory:

/n/holylfs05/LABS/jacob_lab

We recommend that you create the subdirectory /n/holylfs05/LABS/jacob_lab/$USER in which to run your simulations and/or do your data analysis.

Unlike Netscratch, data files placed in /n/holylfs05/LABS/jacob_lab WILL NOT be removed if they are older than 90 days. This makes /n/holylfs05/LABS/jacob_lab ideal for running long GEOS-Chem simulations, running inversions with the IMI, or storing data that you need for more than 90 days. But it also means that the disk space can fill up if you are not careful.

Before starting a job, check how much space is available with this command:

$ quota /n/holylfs05

You also can check the disk usage of an individual directory (and its subdirectories) with this command:

$ du -sh *

Holylfs06

ACMG members have access to 250 TB of shared scratch and/or storage space in the directory:

/n/holylfs06/LABS/jacob_lab2/

The /n/holylfs06/LABS/jacob_lab2/ disk is subdivided into two folders, Everyone and Lab, as described below:

Directory Contents are visible to Purpose Is data purged?
/n/holylfs06/LABS/jacob_lab2/Everyone All Cannon users Sharing data with other users on Cannon no
/n/holylfs06/LABS/jacob_lab2/Lab Only to jacob_lab2 users Generating your own research data no

Create your own folders under Everyone and Lab using these commands:

$ mkdir -p /n/holylfs06/LABS/jacob_lab2/Everyone/$USER
$ mkdir -p /n/holylfs06/LABS/jacob_lab2/Lab/$USER

where the $USER environment variable contains your Cannon username.

We recommend to store research in progress under in your /n/holylfs06/LABS/jacob_lab2/Lab/$USER folder, which is restricted to members of jacob_lab2 only. This will provide an extra measure of security for your unpublished research.

Unlike Netscratch, data files placed in /n/holylfs06/LABS/jacob_lab2 WILL NOT be removed if they are older than 90 days. This makes /n/holylfs06/LABS/jacob_lab2 ideal for running long GEOS-Chem simulations, running inversions with the IMI, or storing data that you need for more than 90 days. But it also means that the disk space can fill up if you are not careful.

Before starting a job, check how much space is available with this command:

$ quota /n/holylfs06/LABS/jacob_lab2

You also can check the disk usage of an individual directory (and its subdirectories) with this command:

$ du -sh *

Archival storage

ACMG members have access to 40 TB of long-term storage space in:

/n/jacob_lab/Users     # For your private research files 
/n/jacob_lab/Lab       # For data to be shared w/ other ACMG members
/n/jacob_lab/Everyone  # For data to be shared publicly

which may also be reached by these paths:

/net/holy-nfsisilon/ifs/rc_labs/jacob_lab/Users    
/net/holy-nfsisilon/ifs/rc_labs/jacob_lab/Lab
/net/holy-nfsisilon/ifs/rc_labs/jacob_lab/Everyone

Each Jacob Group user has a folder (or can create a folder) in that location with a quota of 500 GB. There is also a share folder which can be used to share files with other group members. This space is backed up.

NOTE: You should only use the /n/jacob_lab/ directories for archival storage. Because this space is on an Dell/EMC Isilon disk, it is significantly slower to access than your home directory or scratch directories (which are on faster disks).

Shared data directories

The GEOS-Chem data directories are located at:

/n/holylfs06/LABS/jacob_lab/Everyone/GEOS-CHEM/gcgrid/gcdata/ExtData

The first time you create a GEOS-Chem run directory, you will be asked to supply this path. It will be stored in your ~/.geoschem/config file for subsequent use.

GEOS-Chem data directories switch (Dec 2024)

Prior to December 2024, the GEOS-Chem data directories were stored at this path:

/n/holyscratch01/external_repos/GEOS-CHEM/gcgrid/gcdata/ExtData

However, this disk system is being retired. You will still be able to read data from this disk until its removal in February 2025.

The GEOS-Chem Support Team has copied the contents of the GEOS-Chem data directories to a new permanent location:

/n/holylfs06/LABS/jacob_lab/Everyone/GEOS-CHEM/gcgrid/gcdata/ExtData

Follow these steps to begin using the new GEOS-Chem data directories

  1. Open the file ~/.geoschem/config in your favorite editor

  2. Edit the file accordingly

# CHANGE THIS LINE FROM:
export GC_DATA_ROOT=/n/holyscratch01/external_repos/GEOS-CHEM/gcgrid/data/ExtData

# TO:
export GC_DATA_ROOT=/n/holylfs06/LABS/jacob_lab/Everyone/GEOS-CHEM/gcgrid/gcdata/ExtData

  1. Save the ~/.geoschem/config file

  2. Apply the changes by running this command:

$ source ~/.geoschem/config

The new path to the GEOS-Chem data directories will be inserted into configuration files when you create new run directories.

Common software

Spack libraries for RockyLinux

Please see our Loading software packages into your Cannon login enviroment section for more information.

Python environments

We encourage users to create their own Python environments for RockyLinux, following these instructions. You can install Python environments into your /n/jacob_lab/Users/$USER directory tree (where $USER is an environment variable that stores your Cannon username).

Clone this wiki locally