Skip to content

Commit

Permalink
Merge pull request #23 from flindersuni/revert-22-feature/hpc-upgrade…
Browse files Browse the repository at this point in the history 
…-docs

Revert "Feature/hpc upgrade docs" due to incorrect branch target
  • Loading branch information
@The-Scott-Flinders
The-Scott-Flinders committed Aug 16, 2021
2 parents 2bb6203 + 9bc97c3 commit e5455e2
Show file tree
Hide file tree
Showing 10 changed files with 68 additions and 217 deletions.
2 changes: 1 addition & 1 deletion docs/source/Access/GettingAccess.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ This page will guide you on the steps to get access and then connect to the HPC

## Undergraduates

Currently, the HPC is not open to Undergraduates on a general basis. Exceptions are made on a case-by-case basis - please talk to your project supervisor or Topic Coordinator first and then contact the HPC Support Team via [email](mailto:deepthought@flinders.edu.au).
Currently, the HPC is not open to Undergraduates on a general basis. Exceptions are made on a case-by-case basis - please talk to your project supervisor first and then contact the HPC Support Team via [email](mailto:deepthought@flinders.edu.au).

## Unix/Linux/MacOS

Expand Down
132 changes: 32 additions & 100 deletions docs/source/SLURM/SLURMIntro.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -153,7 +153,7 @@ To requeue (cancel and rerun) a particular job:

### Job Arrays

Job arrays is a popular strategy to process large numbers of the same workflow repetitively in one go, often reducing analytical time significantly. Job arrays are also often refereed as embarrassingly/pleasingly parallel processes. For more information, see [SLURM Job Arrays](https://slurm.schedmd.com/job_array.html).
Job arrays is a popular strategy to process large numbers of a same workflow repetitively in one go, often reduce analytical time significantly. Job arrays are also often refereed as embarrassingly/pleasingly parallel processes. For more information, see SLURM job arrays.

To cancel an indexed job in a job array:

Expand Down Expand Up @@ -183,11 +183,7 @@ The following is useful if your group has its own queue and you want to quickly

### Environmental Variables

There are environment variables set by both SLURM and the HPC to manipulate jobs and their execution.

#### SLURM Specific Environment Variables

The following variables are set per job, and can be access from your SLURM Scripts if needed.
The following varaibles are set per job, and can be access from your SLURM Scripts if needed.

|Variable Name | Description|
|-----------------------------|----------------|
Expand All @@ -203,61 +199,10 @@ The following variables are set per job, and can be access from your SLURM Scrip
|$SLURM_SUBMIT_HOST | Host on which job was submitted.|
|$SLURM_PROC_ID | The process (task) ID within the job. This will start from zero and go up to $SLURM_NTASKS-1.|

#### DeepThought Set Environment Variables

The DeepThought HPC will set some additional environment variables to manipulate some of the Operating system functions. These directories are set at job creation time and then are removed when a job completes, crashes or otherwise exists.

This means that if you leave anything in $TMP or $SHM directories it will be *removed when your job finishes*.

To make that abundantly clear. If the Job creates `/local/jobs/$SLURM_USER/$SLURM_JOB_ID/` it will also **delete that entire directory when the job completes**. Ensure that your last step in any job creation is to _move any data you want to keep to /scratch or /home_.
|Variable Name | Description | Value |
|-----------------------------|-----------------------------------------------|----------------------------|
| $TMP | The Linux default 'Temp' file path. | /local/$SLURM_USER/$SLURM_JOB_ID |
| $TMPDIR | An alias for $TMP| /local/$SLURM_USER/$SLURM_JOB_ID/ |
| $TMP_DIR | An alias for $TMP| /local/$SLURM_USER/$SLURM_JOB_ID/ |
| $TEMP | An alias for $TMP| /local/$SLURM_USER/$SLURM_JOB_ID/ |
| $TEMPDIR | An alias for $TMP| /local/$SLURM_USER/$SLURM_JOB_ID/ |
| $TEMP_DIR | An alias for $TMP| /local/$SLURM_USER/$SLURM_JOB_ID/ |
| $SCRATCH_DIR | A Per-Job Folder on the HPC /scratch mount | /scratch/users/$SLURM_USER/$SLURM_JOB_ID/ |
| $SHM_DIR | A Per-Job Folder on the Compute Node Shared-Memory / Tempfs Mount | /dev/shm/jobs/$USER/ |
| $OMP_NUM_THREADS | The OpenMPI CPU Count Environment Variable | $SLURM_CPUS_PER_TASK |

#### $TMPDIR and SLURM Job-Arrays

[ande0548@hpcdev-head01 slurm]$ sbatch --array=0-5 tmpdir_increment.sh
Submitted batch job 502

[ande0548@hpcdev-head01 slurm]$ squeue
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
502_0 general tmpdir_.s ande0548 R 0:00 1 hpcdev-node001
502_1 general tmpdir_.s ande0548 R 0:00 1 hpcdev-node001
502_2 general tmpdir_.s ande0548 R 0:00 1 hpcdev-node001
502_3 general tmpdir_.s ande0548 R 0:00 1 hpcdev-node001
502_4 general tmpdir_.s ande0548 R 0:00 1 hpcdev-node002
502_5 general tmpdir_.s ande0548 R 0:00 1 hpcdev-node002
[ande0548@hpcdev-head01 slurm]$ ls
slurm-502_0.out slurm-502_1.out slurm-502_2.out slurm-502_3.out slurm-502_4.out slurm-502_5.out tmpdir_increment.sh

[ande0548@hpcdev-head01 slurm]$ cat slurm-502_0.out
TMP: /local/jobs/ande0548/503
TMPDIR: /local/jobs/ande0548/503
JOB ID: 503
TASK ID: 0

[ande0548@hpcdev-head01 slurm]$ cat slurm-502_1.out
TMP: /local/jobs/ande0548/504
TMPDIR: /local/jobs/ande0548/504
JOB ID: 504
TASK ID: 1

Notice that the $TMP directories are different for every step in the array? This ensures that each job will never collide with another jobs $TMP, even if they are running on the same node.

To reiterate the warning above - if you leave anything in the $TMP or $SHM Directories, SLURM will delete it at the end of the job, so make sure you move any results out to /scratch or /home.

### Filename Patterns

Some commands will take a filename. The following modifiers will allow you to generate files that are substituted with different variables controlled by SLURM.

### Filename Patterns

Some commands will take a filename. THe following modifiers will allow you to generate files that are substituted with different variables controlled by SLURM.

| Symbol | Substitution |
|-|-|
Expand All @@ -268,15 +213,16 @@ Some commands will take a filename. The following modifiers will allow you to g
|%J|Jobid.stepid of the running job. (e.g. "128.0") |
|%j|Jobid of the running job. |
|%N| Short hostname. This will create a separate IO file per node. |
|%n|Node identifier relative to current job (e.g. "0" is the first node of the running job) This will create a separate output file per node. |
|%n|Node identifier relative to current job (e.g. "0" is the first node of the running job) This will create a separate IO file per node. |
|%s|Stepid of the running job. |
|%t|Task identifier (rank) relative to current job. This will create a separate output file per task. |
|%t|Task identifier (rank) relative to current job. This will create a separate IO file per task. |
|%u|User name. |
|%x|Job name. |


## SLURM: Extras

Here is an assortment of resources that have been passed on to the Support Team as 'Useful to me'. Your mileage may vary on how useful you find them.
Here is an assortment of resources that have been passed on to the Support Team as 'Useful to me'. Your mileage may vary on how useful you find

[Slurm batch scripting](https://hpc-uit.readthedocs.io/en/latest/jobs/examples.html)

Expand All @@ -286,15 +232,15 @@ Besides useful commands and ideas, this [FAQ](http://www.ceci-hpc.be/slurm_faq.h

An excellent guide to [submitting jobs](https://support.ceci-hpc.be/doc/_contents/QuickStart/SubmittingJobs/SlurmTutorial.html).

## SLURM: Script Template

## SLURM: Script Template

#!/bin/bash
# Please note that you need to adapt this script to your job
# Submitting as is will fail cause the job to fail
# The keyword command for SLURM is #SBATCH --option
# Anything starting with a # is a comment and will be ignored
# ##SBATCH is a commented-out #SBATCH command
# SBATCH and sbatch are identical, SLURM is not case-sensitive
##################################################################
# Change FAN to your fan account name
# Change JOBNAME to what you want to call the job
Expand All @@ -319,19 +265,21 @@ An excellent guide to [submitting jobs](https://support.ceci-hpc.be/doc/_content
# %j will append the 'Job ID' from SLURM.
# %x will append the 'Job Name' from SLURM
# %
#SBATCH --output=/home/<FAN>/%x-%j.out.txt
#SBATCH --error=/home/<FAN>/%x-%j.err.txt
#SBATCH --output=/home/$FAN/%x-%j.out.txt
#SBATCH --error=/home/$FAN/%x-%j.err.txt
##################################################################
# The default partition is 'general'.
# Valid partitions are general, gpu and melfu
# You can leave this commented out, or submit to hpc_general
# Valid partitions are hpc_general and hpc_melfeu
##SBATCH --partition=PARTITIONNAME
#
##################################################################
# Tell SLURM how long your job should run for as a hard limit.
# My setting a shorter time limit, it is more likely that your
# job will be scheduled when attempting to backfill jobs.
# Tell SLURM how long your job should run for, at most.
# SLURM will kill/stop the job if it goes over this amount of time.
# Currently, this is unlimited - however, but the longer your job
# runs, the worse your Fairshare score becomes!
#
# The current cluster-wide limit is 14 Days from Start of Execution.
# In the future this will have a limit, so best to get used to
# setting it now.
#
# The command format is as follows: #SBATCH --time=DAYS-HOURS
#SBATCH --time=14=0
Expand All @@ -358,17 +306,13 @@ An excellent guide to [submitting jobs](https://support.ceci-hpc.be/doc/_content
# jobs. Pick ONE of the below options. They are Mutually Exclusive.
# You can ask for X Amount of RAM per CPU (MB by default)
#SBATCH --mem-per-cpu=4000
# Or, you can ask for a 'total amount of RAM'. If you have multiple
# tasks and ask for a 'total amount' like below, then SLURM will
# split the total amount to each task evenly for you.
# Or, you can ask for a 'total amount of RAM'
##SBATCH --mem=12G
##################################################################
# Change the number of GPU's required and the most GPU's that can be
# requested is 2 per node. As there are limited GPU slots, they are heavily
# weighted against for Fairshare Score calculations.
# You can request either a 'tesla:X' or a 'gpu:x'
# You can either request 0, or omit this line entirely if you
# a GPU is not needed.
# This line requests 0 GPU's by default.
#
#SBATCH --gres="gpu:0"
##################################################################
Expand All @@ -383,24 +327,13 @@ An excellent guide to [submitting jobs](https://support.ceci-hpc.be/doc/_content
##################################################################
# This example script assumes that you have already moved your
# dataset to /scratch as part of your HPC Pre-Job preparations.
# Its best to use the $TMP/$TMPDIR setup for you here
# to allow for the HPC to auto-clean anything you
# leave behind by accident.
# If you have a job-array and need a shared directory for
# data on /local, you will need to manually cleanup that
# directory as a part of your job script.

# Example using the HPC Set $TMPDIR Variable
cd $TMPDIR

# Move your dataset to /local
cd /local
mkdir -p /local/$SLURM_JOBID/$SLURM_ARRAY_TASK_ID/
cd /local/$SLURM_JOBID/$SLURM_ARRAY_TASK_ID/
cp /scratch/user/<FAN>/dataset ./

# A Manual 'Shared' Data-Set Directory
# DATADIR=/local/$SLURM_USER/dataset/
# mkdir -p $DATADIR
# cd $DATADIR
# cp -r /scratch/users/$USER/dataset/ ./


##################################################################
# Enter the command-line arguments that you job needs to run.

Expand All @@ -410,10 +343,9 @@ An excellent guide to [submitting jobs](https://support.ceci-hpc.be/doc/_content
# and ONLY the results to /scratch, then cleanup the temporary
# working directory

cp -r /$TMPDIR/<OUTPUT_FOLDER> /scratch/user/<FAN>/<JOB_RESULT_FOLDER>
cp /local/$SLURM_JOBID/$SLURM_ARRAY_TASK_ID/results/ /scratch/user/<FAN>/???

# Using the example above with a shared dataset directory, your final step
# in the script should remove the directory folder
# rm -rf $DATADIR
rm -rf /local/$SLURM_JOBID/

##################################################################

65 changes: 25 additions & 40 deletions docs/source/Storage/storageusage.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,30 +6,30 @@ The HPC is a little different that your desktop at home when it comes to storage
So, before we start putting files onto the HPC, its best you know where to put them in the first place.

On DeepThought, are two main storage tiers. Firstly our bulk storage is the 'Scratch' area - and is slower, spinning Hard-Disk Drives (HDD's).
The hyper-fast NVMe Solid-State Drives (SSD's) are located at /local and are much smaller. For the exact specifications and capacities, see the `System Specifications`_.
The smaller, hyper-fast NVMe Solid-State Drives (SSD's) are located at /local and is much smaller. For the exact specifications and capacities, see the `System Specifications`_.

There is a critical difference between these two locations. The /scratch area is a common storage area. You can access it from all of the login, management and compute nodes on the HPC. This is not the same as /local, which is only available on each compute node. That is - if your job is running on Node001, the /local only exists on that particular node - you cannot access it anywhere else on the HPC.
There is a critical difference between these two locations. The /scratch area is a common storage area. You can access it from all of the login, management and compute nodes on the HPC. This is not the same as /local, which is only available on each compute node. That is - if you job is running on Node001, the /local only exists on that particular node - you cannot access it anywhere else on the HPC.

.. attention:: The HPC Job & Data Workflow, along with links to the new Data-Workflow Management Portal are under construction and will be linked here when completed.

################################
Storage Accessibility Overview
################################
As general guide, the following table presents the overall storage for the HPC.

+---------------------+--------------------------+-------------------------------------+
| Filesystem Location | Accessible From | Capacity |
+=====================+==========================+=====================================+
| /scratch | All Nodes | ~250TB |
+---------------------+--------------------------+-------------------------------------+
| /home | All Nodes | ~12TB |
+---------------------+--------------------------+-------------------------------------+
| /local | Individual Compute Nodes | ~400GB or ~1.5TB |
+---------------------+--------------------------+-------------------------------------+
| /r_drive/\<folder> | Head Nodes | N/A |
+---------------------+--------------------------+-------------------------------------+
| /RDrive | Head Nodes | Variable, Size of R-Drive Allocation|
+---------------------+--------------------------+-------------------------------------+

.. attention:: /The r_drive/ location is NOT the University R:\\ Drive. It is a remnant from eRSA that is being phased out to the University R:\\ Drive.
+---------------------+--------------------------+-----------------------+
| Filesystem Location | Accessible From | Capacity |
+=====================+==========================+=======================+
| /scratch | All Nodes | ~250TB |
+---------------------+--------------------------+-----------------------+
| /home | All Nodes | ~12TB |
+---------------------+--------------------------+-----------------------+
| /local | Individual Compute Nodes | ~400GB or ~1.5TB |
+---------------------+--------------------------+-----------------------+
| /r_drive/\<folder> | Head Nodes | N/A |
+---------------------+--------------------------+-----------------------+

.. attention:: /The r_drive/ location is NOT the University R:\\ Drive - the ability to move data between R:\\ and the HPC is currently undergoing testing.

The /r_drive/ locations are data mount points from the now defunct eRSA Project and are slowly being phased out. Any point under /r_drive/ will *auto mount on access*. Just attempt to touch or change to the correct directory under the /r_drive/ path and the HPC will handle this automatically for you. Until you do this, the directory **will be invisible**.

Expand All @@ -47,12 +47,11 @@ Your 'home' directories. This is a small amount of storage to store your small b
^^^^^^^^^^^^^^^^^^^^^^^^
What to store in /home
^^^^^^^^^^^^^^^^^^^^^^^^
Here is a rough guide as to what should live in your /home/$FAN directory. In general, you want to store small miscellaneous files here.
Here is a rough guide as to what should live in your /home/$FAN directory. In general, you want small, little things is here.

1. SLURM Scripts
2. Results from Jobs.
3. 'Small' Data-Sets (<5GB)
4. Self-Installed Programs or Libraries
* SLURM Scripts
* Results from Jobs.
* 'Small' Data-Sets (<5GB)

==========
/Scratch
Expand All @@ -65,32 +64,18 @@ What to store in /scratch
^^^^^^^^^^^^^^^^^^^^^^^^^^

Here is a rough guide as to what should live in your /scratch/$FAN directory. In general, anything large, bulky and only needed for a little while should go here.
1. Job Working Data-sets
2. Large Intermediate Files

* Job Working Data-sets
* Large Intermediate Files

=========
/Local
=========

.. _SLURM Temporary Directories: ../SLURM/SLURMIntro.html#tmpdir-and-slurm-job-arrays
Local is the per-node, high speed flash storage that is specific to each node. When running a job, you want to run your data-sets on /local if at all possible - its the quickest storage location on the HPC. You MUST cleanup /local once you are done.

^^^^^^^^^^^^^^^^^^^^^^^^^
What to Store in /local
^^^^^^^^^^^^^^^^^^^^^^^^^

Only *transient files* should live on /local. Anything that your job is currently working on should be on /local. Once your job has finished with these files, they should be copied (or moved) to /scratch. The directory you were working in on /local should then cleaned, removing all files.

The HPC creates a /local directory for you per job that can be used in your SLURM scripts. This is covered in more detail in `SLURM Temporary Directories`_.

===========
/RDrive
===========

/RDrive is the location for all RDrive allocations. The HPC will discover and automatically display any RDrive Folders you have access to.

All /RDrive mount points are only surfaced on the Head-Node. The /RDrive is not present on the compute nodes and you cannot use it as a part of your SLURM scripts.

The /RDrive is not a location to perform any computation on, and is limited in access speed. All data that forms part of dataset for calculations
must be copied to a HPC local mount before you commence work.
Only *transient files* should live on /local. Anything that your job is currently working on should be on /local. Once your job has finished with these files, they should be copied (or moved) to /scratch. The directory you were working in on /local should then cleaned, removing all files from your job.

0 comments on commit e5455e2

Please sign in to comment.