Merge pull request #19 from flindersuni/usability-pass

Initial Usability Pass Release

docs/source/Access/GettingAccess.md

@@ -4,19 +4,23 @@ This page will guide you on the steps to get access and then connect to the HPC
 
 ## Flinders Staff / HDR Students
 
-1. Create a ServiceOne Ticket asking for Access to the HPC.
+1. Create a [ServiceOne](https://flindersuni.service-now.com/csp?id=sc_cat_item&sys_id=d95a633cdbf87810062472f5f39619f5) Ticket asking for Access to the HPC. Currently the request is under the following path: Research Services -> Undertake Research -> Access to DeepThought HPC.
 
 2. Pick your Operating System ([Unix/Linux/MacOS](#unix-linux-macos) or [Windows](#windows) )
 
 3. Read up on some basic [SLURM](../SLURM/SLURMIntro.md)
 
+## Undergraduates
+
+Currently, the HPC is not open to Undergraduates on a general basis. Exceptions are made on a case-by-case basis - please talk to your project supervisor first and then contact the HPC Support Team via [email](mailto:deepthought@flinders.edu.au).
+
 ### Unix/Linux/MacOS
 
 MacOS / MacOSX shares a similar procedure to Unix/BSD Based system. Unix/Linux & MacOS systems have native support for the SSH Protocol, used to connect to the HPC.
 
 #### The Windows Sub-System for Linux
 
-The windows Subsystem for Linux (WSL) allows you to run a Linux Distribution as a sub-system in windows. When following these instructions, a 'terminal' is the same as starting your WSL Distribution.
+The windows Subsystem for Linux (WSL) allows you to run a Linux Distribution as a sub-system in windows. When following these instructions, a 'terminal' is the same as starting your WSL Distribution. Generally, if you are using the WSL, then following the Unix/Linux instructions.
 
 #### Getting Connected
 
@@ -44,10 +48,6 @@ If you wish to setup password-less login via SSH Keys, you may do so.
 
 To connect to Deep Thought a SSH application such as PuTTy is required. Below is a short list of the possible programs you can use as a client to connect to the HPC. This guide will focus on Putty - but will be equally applicable to the other programs.
 
-#### The Windows Sub-System for Linux (WSL)
-
-If you are using the Windows SubSystem for Linux (WSL), then go ahead and read the [Unix/Linux](#unix-linux-macos) instructions instead.
-
 #### Client Options
 
 - [Putty](https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html)
@@ -85,6 +85,6 @@ Upon a successful login, you should get a screen similar to this:
 
 If so, you are now connected and ready to start using the HPC.
 
-#### SSH Keys on Windows
+#### SSH Keys
 
 As with the Unix/Linux/MacOS system, you may also setup SSH Keys for password-less logins. Be sure to follow the specific instructions for your client, as they will differ.

docs/source/FileTransfers/FileTransfersIntro.md

@@ -8,6 +8,53 @@ Before we start, ensure that you have read the [Storage Overview & Usage Guideli
 
 All file-transfers are done via Secure File Transfer Protocol (SFTP), or Secure Copy Protocol (SCP). Other options, like the tool RSync are also usable. This guide will focus upon the GUI based tools, using SFTP.
 
+## Before we get started
+
+The HPC is a little different that your desktop at home when it comes to storage, not just computing power. It's a shared resource, so we cant store everybody's data for all time - there just isn't enough space. 
+
+On DeepThought, are two main storage tiers, with a smaller pool for your documents and scripts. Firstly our bulk storage (approx 250TB) is the 'Scratch' area (located at /scratch/user/<fan>) - and is slower, spinning Hard-Disk Drives (HDD's). The smaller, hyper-fast NVMe Solid-State Drives (located at /local) are roughly 400GB on the 'standard' nodes (1-16) and 1.5TB on the 'high-capacity' nodes (19-21).
+
+There is a critical difference between these two locations. The /scratch area is a common storage area. You can access it from all of the login, management and compute nodes on the HPC. This is not the same as /local, which is only available on each compute node.  That is - if you job is running on Node001, the /local only exists on that particular node - you cannot access it anywhere else on the HPC. 
+
+- /home/$FAN
+- /scratch/$FAN
+
+There are also two location that treated specially: 
+
+- /mnt/
+- /r_drive/ 
+
+Both of these locations will surface research specific mount points on request. If you have access to these locations, they auto-mount **on usage**. That is, simply try and touch or access the correct location that you have access to, and it will be handled automatically for you. For example, if you have access to a mount point called 'molecular_data', then the following command with surface that mount point to you - `cd /r_drive/molecular_data/`. 
+
+### /Home
+
+Your 'home' directories. This is a small amount of storage (~11TB total) to store your small bits and pieces. This is the analogous to the Windows 'Documents' folder.
+
+At a command promp, your home directory usually gets shortened to ~/. 
+
+#### What to store in /home
+
+Here is a rough guide as to what should live in your /home/$FAN directory. In general, you want small, little things is here.
+
+- SLURM Scripts
+- Results from Jobs.
+- 'Small' Data-Sets (<5GB)
+
+### /Scratch
+
+Scratch is your working space. Depending upon your dataset, you may need to run your job here - this is not optimal and will be much slower than running it from /local. Scratch is still not an area to store your data permanently - there are no backups in place for the HPC, so ensure you follow the [HPC Research Data Flow ]() and the [HPC Job Data Flow]().
+
+#### What to store in /scratch
+
+Here is a rough guide as to what should live in your /scratch/$FAN directory. In general, anything large, bulky and only needed for a little while should go here.
+
+- Job Working Data-sets
+- Intermediate files
+
+## Transferring Files
+
+All file-transfers are done via Secure File Transfer Protocol (SFTP) or Secure Copy Protocol (SCP). As was the same with the 'Getting Access', pick your platform:
+
 ### Linux/Unix File Transfers
 
 Linux / Unix based systems share native support for the SFTP Protocol. The Secure Copy Protocol (SCP) is also widely accepted, which can sometimes offer an edge in transfer speed.
@@ -70,7 +117,7 @@ This is not an exhaustive list - feel free to use whatever you wish that support
 
 This guide will focus on WinSCP.
 
-#### Getting Connected
+#### Getting Connected with WinSCP
 
 Open WinSCP, enter deepthought.flinders.edu.au as the host to connect to, and click Login. You should have a screen that looks like this.
 

docs/source/ModuleSystem/LMod.md

@@ -4,46 +4,63 @@ One of such challenges of running a HPC is effectively managing a complex suite
 
 Best way to think of Module is a singular program version and all of its associated dependencies to run correctly.
 
+## Additional Software & Modules
+
+Generally speaking, we can install almost all Linux/CentOS bounded software/applications on HPC, but we don't always need to go thorough the effort to install things 'globally' for everybody.
+
+1. Are people other than just me going to use this software?
+2. If yes, create a [ServiceOne](https://flindersuni.service-now.com) Ticket, and the HPC Support Team will assess the request.
+
+Otherwise, there is nothing stopping you installing the program locally for yourself! If you run into issues installing software then open [ServiceOne](https://flindersuni.service-now.com) ticket, or contact the HPC Support team at their [email](mailto:deepthought@flinders.edu.au).
+
 ## How Do I Install Software?
 
-There are multiple ways to install software on the HPC. Below is an expansion on some of the common ones. The short and sweet version is that, if you compile/install it yourself to your /home or a Virtual Environment of some kind, you are free to do whatever you want!
+There are multiple ways to install software on the HPC. Below is an expansion on some of the common ones. The short and sweet version is that, if you compile/install it yourself to your /home or a Virtual Environment of some kind, you are free to do whatever you want! Just be mindful of disk space. If its a common tool that your whole research lab will be using, consider putting in a support request so the HPC Team can make it a global module instead of everybody having their own copy.
 
 You may also utilise the same tooling as the HPC Support Team - EasyBuild.  EasyBuild is a management tool allowing for repeatable installation of a specific piece of software to aid in reproducibility of results. You can load this the same way as any other module on the HPC. By default, this tool will install to your home directory, and more information can be [read here](https://docs.easybuild.io/en/latest/).
 
 The HPC support team will need to action your request if you need something big and complicated like ANSYS, GNU Octave, a new version of R or other similar large and complicated programs.
 
 ### Python / Conda
 
-The HPC Team allows you to install your own packages by using the inbuilt package manager tools, like Pythons 'pip', or Conda. 
+The HPC Team allows you to install your own packages by using the inbuilt package manager tools, like Pythons 'pip', or Conda when using a virtual environment - you cannot install modules globally on the HPC using these tools.
 
 As an example, you can create a Conda Virtual Environment - this is under your complete control and you may install, remove or alter it as you wish. This is also the same for Pythons 'venv', which functions in much the same way.  
 
 The Conda Guide is located at: [Conda Guide](https://docs.conda.io/projects/conda/en/latest/user-guide/getting-started.html#managing-envs)
 
 The Python Guide is located at: [Python Guide](https://packaging.python.org/guides/installing-using-pip-and-virtual-environments/#creating-a-virtual-environment)
 
+### EasyBuild
+
+The HPC Support Team use [EasyBuild](https://easybuild.io/) to manage most of the software on the HPC (Not all, things like ANSYS and MATLAB are too complicated for such a tool). It is also open for users to self install software using the same tooling. As with all things HPC, it can get complicated - the [documentation is situated here](https://docs.easybuild.io/en/latest/).
+
 ### Compile Your Own
 
-The HPC uses the FOSS Toolchain, as detailed in the [Fair Usage](../policies/fairuse.html#toolchains) Policy.  Should you wish to compile and use your own software, simply load the associated module (eg, foss-2020a) which will load up the associated GCC Toolchains and libraries. 
+The HPC uses the FOSS Toolchain, as detailed in the [Fair Usage](../policies/fairuse.html#toolchains) Policy.  Should you wish to compile and use your own software, simply load the associated module (eg, foss-2020a) which will load up the associated tools and libraries.
 
 #### My Toolchain isn't Listed
 
-Should you require a different Toolchain, like LLVM or Go and it is not listed under the `module avail` list, you can either: 
+Should you require a different Toolchain, like LLVM or Go and it is not listed under the `module avail` list, you can either:
 
-1.) Bootstrap the compiler + libraries yourself in your /home directory, managing it and any programs yourself
+1.) Bootstrap the compiler + libraries yourself in your /home directory
 
-2.) Contact the HPC Support Team, either via [Email](mailto:deepthought@flinders.edu.au) or ServiceOne
+2.) Contact the HPC Support Team, either via [Email](mailto:deepthought@flinders.edu.au) or [ServiceOne](https://flindersuni.service-now.com/csp)
 
 ## Module Format
 
 As Software requirements for research can be very specific, the modules follow suit as well. Newer modules managed with the EasyBuild Management tooling will have the following syntax:
 
 - Program/Version-Toolchain-Version-Interpreter-Version
 
+An example EasyBuild module is: `BioPerl/1.7.2-GCCcore-8.2.0-Perl-5.28.1`.
+
 Manually installed software will always be a shorter format of:
 
 - Program/Version
 
+An example of a manual module is: matlab/r2020b
+
 The following sections will break down the longer string of `BioPerl/1.7.2-GCCcore-8.2.0-Perl-5.28.1`.
 
 ### Program-Version
@@ -74,7 +91,7 @@ Will get you a list of something similar to this - a list of every single availa
 
     module list
 
-Will get you a list of your current modules.
+Will get you a list of your currently loaded modules.
 
 ![](../_static/moduleListExample.png)
 
@@ -87,21 +104,12 @@ There are three main ways to load a module. For most of the time, they are funct
 #### Module Load
 
     module load BioPerl/1.7.2-GCCcore-8.2.0-Perl-5.28.1.
-
+s
 There is also a nice shortcut, that you can use:
 
     ml BioPerl/1.7.2-GCCcore-8.2.0-Perl-5.28.1.
 
-'ml' is just short for 'module load'. You can also use it as shorthand for `module` commands.
-
-## Additional Software & Modules
-
-Generally speaking, we can install almost all Linux/CentOS bounded software/applications on HPC, but we don't always need to go thorough the effort to install things 'globally' for everybody.
-
-1. Are people other than just me going to use this software?
-2. If yes, create an ServiceOne Ticket, and the HPC Support Team will assess your request.
-
-Otherwise, there is nothing stopping you installing the program locally for yourself! If you run into issues installing software then open an ServiceOne ticket and again, the HPC Support Team will do our best to help you.
+'ml' is just short for 'module load'. You can also use it as shorthand for `module` commands, like `ml spider bioperl`.
 
 ___
 
@@ -116,4 +124,4 @@ The latest list is always available by running the `module avail` command on the
 
 ### Writing Your Own Modules
 
-You can [write your own module files](https://lmod.readthedocs.io/en/latest/015_writing_modules.html#) if you want, although this should be a last resort. Its far less error prone to use a Python/Conda environment to bypass the need for modules entirely or the EasyBuild tool to handle this for you.
+You can [write your own module files](https://lmod.readthedocs.io/en/latest/015_writing_modules.html#) if you want, although this should be a last resort. It is far less error prone to use a Python/Conda environment to bypass the need for modules entirely or the EasyBuild tool to handle this for you.