Skip to content

Harvard ACMG resources on Cannon

Jump to bottom
Bob Yantosca edited this page Dec 3, 2024 · 97 revisions

This page summarizes resources available to Harvard-ACMG group members on the FAS Research Computing Cannon (FASRC) cluster. See also the Cluster Storage page on the FASRC documentation site.

Partitions on Cannon

Harvard-ACMG group members can submit computational jobs to the following partitions:

You can request multiple partitions by adding a command such as these into your run script:

#SBATCH -p sapphire,huce_cascade,seas_compute,shared                     # or
#SBATCH -p sapphire,huce_cascade,seas_compute,shared,serial_requeue

Your job will start in whichever partition first has free resources.

For interactive jobs (i.e. compiling, plotting, etc.), we recommend using the test partition. You can use up to:

  • Up to 5 concurrent jobs per user
  • Up to 12 hours time.
  • Up to 96 total cores per user
  • Up to 384 total GB memory per user.

Storage space

Summary of storage locations

Location Description Quota Suitable for Is data purged?
$HOME Home directory 100 GB / user Editing, data processing, storage, tasks that do not generate large amounts of data no
/n/netscratch/jacob_lab Netscratch 50 TB Running GCClassic or GCHP, Data analysis, Temporary storage after 90 days
/n/holylfs05/LABS/jacob_lab Holylfs05 245 TB Running GCClassic or GCHP, Inversions with the IMI, Data analysis, Longer-term storage no
/n/holylfs06/LABS/jacob_lab2 Holylfs05 245 TB Running GCClassic or GCHP, Inversions with the IMI, Data analysis, Longer-term storage no
/n/jacob_lab Archival storage 40 TB Long-term data storage for completed projects no

Home directory

The /n/home* directories are your primary storage location for analysis and data processing. Your home directory is the initial working directory upon login and can always be returned to by typing cd ~. Every Cannon user has a 100 GB size limit in their home directory. This directory is backed up daily. You can find the address of another user's home directory by typing

$ file ~username

filling in the username of the person whose directory you are looking for.

You can check the quota for your home directory, including how much space you are using, with the following command from anywhere on Cannon:

$ df -h ~

The home directory is suitable for

Netscratch

ACMG members have access to 50 TB (total) of shared scratch space in the directory:

/n/netscratch/jacob_lab

Which is also known as

reachable via:

$SCRATCH/jacob_lab/

where as of December 2024 the system-defined environment variable $SCRATCH points to /n/netscratch/. This could change in the future, so we recommend using $SCRATCH in your data paths for consistency.

The $SCRATCh/jacob_lab disk is subdivided into two folders, Everyone and Lab, as described below:

Directory Contents will be visible to Purpose Purged after
$SCRATCH/jacob_lab/Everyone All Cannon users Sharing data with other users on Cannon 90 days
$SCRATCH/jacob_lab/Lab Only to jacob_lab users Generating your own research data 90 days

We recommend to store research in progress under in your $SCRATCH/jacob_lab/Lab/$USER folder, which is restricted to members of jacob_lab only. This will provide an extra measure of security for your unpublished research.

Create your own folders under Everyone and Lab using these commands:

$ mkdir -p $SCRATCH/jacob_lab/Everyone/$USER
$ mkdir -p $SCRATCH/jacob_lab/Lab/$USER

where the $USER environment variable contains your Cannon username.

There is a 90 day time limit for files on the $SCRATCH disk space and a 50 TB limit per group. However, if the $SCRATCH disk starts to approach 95% of its storage capacity, Research Computing will begin deleting files that are older than 90 days. This space is not backed up.

If you plan on storing large amounts of data, even for just a short time period, you should first check how much space is available for the group. Run the following command to view the quota status of the group's scratch space:

If you want to check how much space you are currently using you can use the unix command for checking disk usage du. Below is an example of using this tool to print a human-readable summary for all subdirectories in the directory you are currently in.

$ du -sh *

You can use this command at the Unix prompt to check the age of a file or directory:

$ echo $((($(date +%s) - $(date +%s -r "FILENAME")) / 86400)) days

where FILENAME is the file or directory that you wish to test. This may help you to know when your files on scratch are getting close to the 90-day limit.

Holylfs05

ACMG members have an additional 145 TB of scratch and/or storage space in the directory:

$ /n/holylfs05/LABS/jacob_lab

We recommend that you create the subdirectory /n/holylfs05/LABS/jacob_lab/$USER in which to run your simulations and/or do your data analysis.

Unlike the main scratch disk described above, data files placed in /n/holylfs05/LABS/jacob_lab WILL NOT be removed if they are older than 90 days. Whatever data you place in this directory structure will remain in place until you manually remove it.

The /n/holylfs05/LABS/jacob_lab disk is therefore ideal for running long GEOS-Chem simulations, running inversions with the IMI, or storing data that you need for more than 90 days. However, please be aware that it is possible to fill up this disk, as the contents are not automatically removed. If you plan on storing large amounts of data, even for just a short time period, you should first check how much space is available for the group. Run the following command to view the quota status of the group's scratch space:

$ lfs quota -hg jacob_lab /n/holylfs05

If you want to check how much space you are currently using you can use the unix command for checking disk usage du. Below is an example of using this tool to print a human-readable summary for all subdirectories in the directory you are currently in.

$ du -sh *

Holylfs06

ACMG members have access to 250 TB (total) of shared scratch space in the directory:

/n/holylfs06/LABS/jacob_lab2/

This space is suitable for running inversions with the IMI and storing the data temporarily while analysis is performed.

The $/n/holylfs06/LABS/jacob_lab2/ disk is subdivided into two folders, Everyone and Lab, as described below:

Directory Contents will be visible to Purpose Is data purged?
$SCRATCH/jacob_lab2/Everyone All Cannon users Sharing data with other users on Cannon no
$SCRATCH/jacob_lab2/Lab Only to jacob_lab2 users Generating your own research data no

We recommend to store research in progress under in your $SCRATCH/jacob_lab2/Lab/$USER folder, which is restricted to members of jacob_lab only. This will provide an extra measure of security for your unpublished research.

Create your own folders under Everyone and Lab using these commands:

$ mkdir -p $SCRATCH/jacob_lab/Everyone/$USER
$ mkdir -p $SCRATCH/jacob_lab/Lab/$USER

where the $USER environment variable contains your Cannon username.

If you plan on storing large amounts of data, even for just a short time period, you should first check how much space is available for the group. Run the following command to view the quota status of the group's scratch space:

If you want to check how much space you are currently using you can use the unix command for checking disk usage du. Below is an example of using this tool to print a human-readable summary for all subdirectories in the directory you are currently in.

$ du -sh *

Archival storage

ACMG members have access to 40 TB of long-term storage space in:

/n/jacob_lab/Users     # For your private research files 
/n/jacob_lab/Lab       # For data to be shared w/ other ACMG members
/n/jacob_lab/Everyone  # For data to be shared publicly

which may also be reached by these paths:

/net/holy-nfsisilon/ifs/rc_labs/jacob_lab/Users    
/net/holy-nfsisilon/ifs/rc_labs/jacob_lab/Lab
/net/holy-nfsisilon/ifs/rc_labs/jacob_lab/Everyone

Each Jacob Group user has a folder (or can create a folder) in that location with a quota of 500 GB. There is also a share folder which can be used to share files with other group members. This space is backed up.

NOTE: You should only use the /n/jacob_lab/ directories for storage. Because this space is on an Dell/EMC Isilon disk, it is significantly slower to access than your home directory or scratch directories (which are on faster Lustre disks).

Shared data directories

The GEOS-Chem data directories are located at:

/n/holylfs06/LABS/jacob_lab/Everyone/GEOS-CHEM/gcgrid/gcdata/ExtData

The first time you create a GEOS-Chem run directory, you will be asked to supply this path. It will be stored in your ~/.geoschem/config file for subsequent use.

GEOS-Chem data directories switch (Dec 2024)

Prior to December 2024, the GEOS-Chem data directories were stored at this path:

/n/holyscratch01/external_repos/GEOS-CHEM/gcgrid/gcdata/ExtData

However, this disk system is being retired. You will still be able to read data from this disk until its removal in February 2025.

The GEOS-Chem Support Team has copied the contents of the GEOS-Chem data directories to a new permanent location:

/n/holylfs06/LABS/jacob_lab/Everyone/GEOS-CHEM/gcgrid/gcdata/ExtData

Follow these steps to begin using the new GEOS-Chem data directories

  1. Open the file ~/.geoschem/config in your favorite editor

  2. Edit the file accordingly

# CHANGE THIS LINE FROM:
export GC_DATA_ROOT=/n/holyscratch01/external_repos/GEOS-CHEM/gcgrid/data/ExtData

# TO:
export GC_DATA_ROOT=/n/holylfs06/LABS/jacob_lab/Everyone/GEOS-CHEM/gcgrid/gcdata/ExtData

  1. Save the ~/.geoschem/config file

  2. Apply the changes by running this command:

$ source ~/.geoschem/config

The new path to the GEOS-Chem data directories will be inserted into configuration files when you create new run directories.

Common software

Spack libraries for RockyLinux

Please see our Loading software packages into your Cannon login enviroment section for more information.

Python environments

We encourage users to create their own Python environments for RockyLinux, following these instructions. You can install Python environments into your /n/jacob_lab/Users/$USER directory tree (where $USER is an environment variable that stores your Cannon username).

Clone this wiki locally