Merge pull request #20 from flindersuni/develop

Usability Pass Release

.editorconfig

@@ -0,0 +1,10 @@
+root = true
+
+[*]
+charset = utf-8
+end_of_line = lf
+indent_size = 4
+indent_style = space
+insert_final_newline = false
+max_line_length = 120
+tab_width = 4

docs/source/Access/GettingAccess.md

@@ -4,19 +4,23 @@ This page will guide you on the steps to get access and then connect to the HPC
 
 ## Flinders Staff / HDR Students
 
-1. Fire off a ServiceOne Ticket asking for [Access to the HPC](https://flindersuni.service-now.com/csp?id=sc_cat_item&sys_id=d95a633cdbf87810062472f5f39619f5)
+1. Create a [ServiceOne](https://flindersuni.service-now.com/csp?id=sc_cat_item&sys_id=d95a633cdbf87810062472f5f39619f5) Ticket asking for Access to the HPC. Currently the request is under the following path: Research Services -> Undertake Research -> Access to DeepThought HPC.
 
 2. Pick your Operating System ([Unix/Linux/MacOS](#unix-linux-macos) or [Windows](#windows) )
 
 3. Read up on some basic [SLURM](../SLURM/SLURMIntro.md)
 
+## Undergraduates
+
+Currently, the HPC is not open to Undergraduates on a general basis. Exceptions are made on a case-by-case basis - please talk to your project supervisor first and then contact the HPC Support Team via [email](mailto:deepthought@flinders.edu.au).
+
 ### Unix/Linux/MacOS
 
 MacOS / MacOSX shares a similar procedure to Unix/BSD Based system. Unix/Linux & MacOS systems have native support for the SSH Protocol, used to connect to the HPC.
 
 #### The Windows Sub-System for Linux
 
-The windows Subsystem for Linux (WSL) allows you to run a Linux Distribution as a sub-system in windows. When following these instructions, a 'terminal' is the same as starting your WSL Distribution.
+The windows Subsystem for Linux (WSL) allows you to run a Linux Distribution as a sub-system in windows. When following these instructions, a 'terminal' is the same as starting your WSL Distribution. Generally, if you are using the WSL, then following the Unix/Linux instructions.
 
 #### Getting Connected
 
@@ -44,21 +48,18 @@ If you wish to setup password-less login via SSH Keys, you may do so.
 
 To connect to Deep Thought a SSH application such as PuTTy is required. Below is a short list of the possible programs you can use as a client to connect to the HPC. This guide will focus on Putty - but will be equally applicable to the other programs.
 
-#### The Windows Sub-System for Linux (WSL)
-
-If you are using the Windows SubSystem for Linux (WSL), then go ahead and read the [Unix/Linux](#unix-linux-macos) instructions instead.
-
 #### Client Options
 
 - [Putty](https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html)
 - [KiTTY](http://www.9bis.net/kitty/#!pages/download.md)
 - [BitVise Client](https://www.bitvise.com/download-area)
+- [MobaXterm](https://mobaxterm.mobatek.net/)
 
 #### Getting Connected on Windows
 
 Open PuTTy, and you are presented with this screen:
 
-![](../../_static/puttyAccessImage.png)
+![](../_static/puttyAccessImage.png)
 
 - Fill in the hostname to be: deepthought.flinders.edu.au,
 - Change the Connection Type to SSH
@@ -69,7 +70,7 @@ Open PuTTy, and you are presented with this screen:
 
 If all has gone well, you will be presented with this screen:
 
-![](../../_static/puttyLoginImage.png)
+![](../_static/puttyLoginImage.png)
 
 - Your Username is your FAN
 - Your Password is your FAN Password.
@@ -80,10 +81,10 @@ These are the same credentials you use to login to OKTA.
 
 Upon a successful login, you should get a screen similar to this:
 
-![](../../_static/loginOKImage.png)
+![](../_static/loginOKImage.png)
 
-If so, you are now connected and ready to start using the HPC!
+If so, you are now connected and ready to start using the HPC.
 
-#### SSH Keys on Windows
+#### SSH Keys
 
-As with the Unix/Linux/MacOS system, you may also setup SSH Keys for password-less logins. Be sure to follow the specific instructions for your client, as they will differ!
+As with the Unix/Linux/MacOS system, you may also setup SSH Keys for password-less logins. Be sure to follow the specific instructions for your client, as they will differ.

docs/source/FAQ/faq.rst

@@ -118,23 +118,3 @@ The given bx-python is a problematic module that appears in many of the BioScien
 These steps are the same as the installation for IsoSeq3, but given how often this particular python package gives the support team issues, it gets its own section!
 
 * conda install -c conda-forge -c bioconda bx-python
-
-
-What can I do on the Head Node? 
-================================
-The head nodes are for small, 'Less than 10 minutes' (as a rough guide), small jobs. 
-Things like: 
-
-* Compiling software
-* Copying / Decompressing Files 
-* Preparing Scripts
-
-As a good rule, if it takes more than 10-15 minutes or > 2GB RAM, it should be run a SLURM Job, not on the head nodes. 
-Anything that uses too many resources on the head nodes will be *Terminated* **WITHOUT WARNING**.
-
-
-My Mount Point at /r_drive or /mnt is missing! 
-===============================================
-Anything under those directories above are set to transparent auto-mount. 
-Try and move to a directory or touch a file, and the operating system will mount and surface the locations for you 
-automatically. 

docs/source/FileTransfers/FileTransfersIntro.md

@@ -1,12 +1,20 @@
-# Transferring Files to Deepthought
+# Transferring Files to DeepThought
 
-Transferring files to the HPC will change depending upon your OS. Thankfully, there are some excellent tools that take this from 'potentially-scary' to 'click a button or two'.
+Transferring files to the HPC will change depending upon your OS. Thankfully, there are some excellent tools that take this from 'potentially-scary' to 'click a button or two'.  
+
+Before we start, ensure that you have read the [Storage Overview & Usage Guidelines](../Storage/storageusage.html).
+
+## Transferring Files
+
+All file-transfers are done via Secure File Transfer Protocol (SFTP), or Secure Copy Protocol (SCP). Other options, like the tool RSync are also usable. This guide will focus upon the GUI based tools, using SFTP.
 
 ## Before we get started
 
-The HPC is a little different that your desktop at home when it comes to storage (not just computing power!). It's a shared resource, so we cant store everybody's data for all time - there just isn't enough space!
+The HPC is a little different that your desktop at home when it comes to storage, not just computing power. It's a shared resource, so we cant store everybody's data for all time - there just isn't enough space. 
 
-So, we have two main storage locations we use which are:
+On DeepThought, are two main storage tiers, with a smaller pool for your documents and scripts. Firstly our bulk storage (approx 250TB) is the 'Scratch' area (located at /scratch/user/<fan>) - and is slower, spinning Hard-Disk Drives (HDD's). The smaller, hyper-fast NVMe Solid-State Drives (located at /local) are roughly 400GB on the 'standard' nodes (1-16) and 1.5TB on the 'high-capacity' nodes (19-21).
+
+There is a critical difference between these two locations. The /scratch area is a common storage area. You can access it from all of the login, management and compute nodes on the HPC. This is not the same as /local, which is only available on each compute node.  That is - if you job is running on Node001, the /local only exists on that particular node - you cannot access it anywhere else on the HPC. 
 
 - /home/$FAN
 - /scratch/$FAN
@@ -16,27 +24,25 @@ There are also two location that treated specially:
 - /mnt/
 - /r_drive/ 
 
-Both of these locations will surface research specific mount points on request. If you have access to these locations, they auto-mount **on usage**. That is, simply try and touch or access the correct location that you have access to, and it will be handled automatically for you.
+Both of these locations will surface research specific mount points on request. If you have access to these locations, they auto-mount **on usage**. That is, simply try and touch or access the correct location that you have access to, and it will be handled automatically for you. For example, if you have access to a mount point called 'molecular_data', then the following command with surface that mount point to you - `cd /r_drive/molecular_data/`. 
 
 ### /Home
 
-Your 'home' directories. This is a small amount of storage to store your small bits and pieces. This is the analogous to the Windows 'Documents' folder.
+Your 'home' directories. This is a small amount of storage (~11TB total) to store your small bits and pieces. This is the analogous to the Windows 'Documents' folder.
 
-At a command promp, your home directory usually gets shortened to ~/ - we will cover this more in the Linux Guide.
+At a command promp, your home directory usually gets shortened to ~/. 
 
 #### What to store in /home
 
 Here is a rough guide as to what should live in your /home/$FAN directory. In general, you want small, little things is here.
 
 - SLURM Scripts
 - Results from Jobs.
-- 'Small' Data-Sets (<1GB)
+- 'Small' Data-Sets (<5GB)
 
 ### /Scratch
 
-Scratch is your working space. Whenever you are running a job, it should be running in the the /scratch area. This storage area is quicker and isolated from the others - meaning that even under high usage you are not going to slow down much (if at all) waiting to read data to or from the disks.
-
-It's also much, much larger than your /home area. Some of the working data-sets we have seen on Deep Thought have gotten out to Multiple Terabytes! Thankfully, /scratch can handle this, however /home would not.
+Scratch is your working space. Depending upon your dataset, you may need to run your job here - this is not optimal and will be much slower than running it from /local. Scratch is still not an area to store your data permanently - there are no backups in place for the HPC, so ensure you follow the [HPC Research Data Flow ]() and the [HPC Job Data Flow]().
 
 #### What to store in /scratch
 
@@ -47,7 +53,7 @@ Here is a rough guide as to what should live in your /scratch/$FAN directory. In
 
 ## Transferring Files
 
-All file-transfers are done via Secure File Transfer Protocol (SFTP). As was the same with the 'Getting Access', pick your platform:
+All file-transfers are done via Secure File Transfer Protocol (SFTP) or Secure Copy Protocol (SCP). As was the same with the 'Getting Access', pick your platform:
 
 ### Linux/Unix File Transfers
 
@@ -59,7 +65,7 @@ Since Windows 10 and Windows Server 2019, the windows Subsystem for Linux (WSL)
 
 #### Transferring Files to the HPC
 
-To upload files to DeepThought, the simplest method is to open a Terminal window.
+When using a *NIX based system, using the terminal is the fastest way to upload files to the HPC.
 
 #### The Quick Version
 
@@ -111,7 +117,7 @@ This is not an exhaustive list - feel free to use whatever you wish that support
 
 This guide will focus on WinSCP.
 
-#### Getting Connected
+#### Getting Connected with WinSCP
 
 Open WinSCP, enter deepthought.flinders.edu.au as the host to connect to, and click Login. You should have a screen that looks like this.
 
@@ -121,6 +127,8 @@ The first time you connect up you will get a warning - this is fine, just click
 
 ![](../_static/WinSCPSSHKeyNotice.png)
 
-A connection to Deep Thought will then be created - login using your FAN and password. If all goes well, you will be treated to this screen:
+A connection to Deep Thought will then be created. If all goes well, you will be treated to this screen:
 
 ![](../_static/WinSCPConnected.png)
+
+You can now drag and drop files between your computer (Left-hand side) and DeepThought (Right-hand side).

docs/source/LinuxCommands/LinuxIntro.md

@@ -1,11 +1,11 @@
 # Linux Commands: HPC Style
 
-We try not to re-invent the wheel, and there is already an excellent guide for basic linux commands, written by the excellent people at Harvard. Head on over to [here](https://informatics.fas.harvard.edu/basic-unix-workshop.html) and have a read!
+We try not to re-invent the wheel, and there is already a wonderful guide for basic linux commands, written by the excellent people at Harvard. Head on over to [here](https://informatics.fas.harvard.edu/basic-unix-workshop.html) and have a read!
 
 ## Tips and Tricks
 
 If you want more from your command line, they have a [useful tips and tricks](https://informatics.fas.harvard.edu/unix-command-line-tips-and-tricks.html) guide that should keep you satisfied.
 
 ## Help! I have no idea
 
-Stuck? No idea what Unix is? Linux? BSD? This all complete gibbersh-and-gobbldegook to you? If so, head on over to a [Introduction to UNIX and the Command Line](https://software.rc.fas.harvard.edu/training/intro_unix/latest/#(1)) that should help get you up to speed.
+Stuck? No idea what Unix is? Linux? BSD? This all complete gibbersh-and-gobbldegook to you? If so, head on over to a [Introduction to UNIX and the Command Line](https://software.rc.fas.harvard.edu/training/intro_unix/latest/#(1)) that will help get you up to speed on the interface of choice for High-Performance Computing.