Merge pull request #26 from flindersuni/feature/hpc-upgrade-docs

Feature/hpc upgrade docs

docs/source/index.rst

@@ -1,8 +1,3 @@
-.. DeepThoughtHPC-docs documentation master file, created by
-   sphinx-quickstart on Thu Mar 12 09:21:20 2020.
-   You can adapt this file completely to your liking, but it should at least
-   contain the root `toctree` directive.
-
 Welcome to the DeepThought Documentation
 =========================================
 
@@ -11,7 +6,7 @@ The new Flinder HPC is called DeepThought. This new HPC comprises of AMD EPYC ba
 .. _Upgrade Migration Information: migration/upgrademigration.html
 
 .. warning:: 
-    DeepThought has recently undergone a series of upgrades that require some user intervention when utlising the upgraded cluster
+    DeepThought has recently undergone a series of upgrades that require some user intervention when utlising the upgraded cluster.
     Please see `Upgrade Migration Information`_ for actions required.
 
 .. attention::
@@ -77,6 +72,7 @@ Table of Contentss
     software/ansys.rst
     software/jupyter.rst
     software/singularity.rst
+    software/matlab.rst
 
 
 .. toctree::
@@ -102,24 +98,13 @@ Table of Contentss
    migration/upgrademigration.rst
    upgrades/updatelog.rst
 
-Citing DeepThoughtHPC
-----------------------
-
-To cite DeepThoughtHPC service, please use following or download `BibTex`_/`EndNote`_ file. Your citation will demonstrate the research impact that DeepThoughtHPC service brings and support ongoing funding efforts for this service.
-
-Flinders University (2021). DeepThought (HPC). Retrieved from https://doi.org/10.25957/FLINDERS.HPC.DEEPTHOUGHT
-
-.. _BibTex: https://raw.githubusercontent.com/flindersuni/DeepThoughtHPC-docs/master/docs/source/flindershpc2021-bibtex.bib
-.. _EndNote: https://raw.githubusercontent.com/flindersuni/DeepThoughtHPC-docs/master/docs/source/flindershpc2021-endnote.xml
-
-
 
 Acknowledgements
 ----------------
 
 We recognise the respect the trademarks of all third-party providers referenced in this documentation. Please see the respective EULAs for software packages used in configuring your own environment based on this knowledgebase.
 
 License
-=========
+----------
 
 This documentation is released under the `Creative-Commons: Attribution-ShareAlike 4.0 International <http://creativecommons.org/licenses/by-sa/4.0/>`_ license.

docs/source/migration/upgrademigration.rst

@@ -2,13 +2,26 @@
 DeepThought Upgrade Migration 
 ===============================
 
-DeepThought has undergone major upgrades. To make it as seamless for our users as possible, we made sure to keep 
-your /home and /scratch directories and import them across to the Upgraded Cluster. 
+The DeepThought HPC has undergone major upgrades. To make it as seamless for our users as possible, we made sure to keep 
+your /home and /scratch directories the same in the new cluster. 
+
+This means that all your files and folders are untouched when you migrate to the new, rebuilt cluster.
 
 The following steps are required for a user to undertake when they migrate across to the new 
 cluster. 
 
 
+Installed Software 
+++++++++++++++++++++++
+The HPC Support Team has removed all software not loaded via the module system in the last 6 months. Every effort was made 
+to ensure we did not remove an actively used piece of software. 
+
+
+Action Needed
+****************
+If your software package is no longer visible in the ``module avail`` list, then please either submit a ServiceOne request 
+for HPC Software, or notify the HPC Support Team via email. 
+
 Module System
 +++++++++++++++
 The old module system was partially configured, and while it worked just fine, has quite a few legacy options 

docs/source/software/ansys.rst

@@ -1,4 +1,15 @@
 -------------------------
 ANSYS Engineering Suite 
 -------------------------
-Currently under development, with the end goal of presenting the HPC as a remote solver target for all possible ANSYS Suites. 
+=======
+Status
+=======
+Currently Under Development. Contact the HPC Support Team for more information. 
+
+==========
+Overview 
+========== 
+The ANSYS Engineering Suite is a comprehensive software suite for engineering simulation.
+The new HPC ANSYS integration aims to bring the Remote Solver Manager to DeepThought - allowing 
+users to leverage the HPC from their desktops and familiar ANSYS interface. 
+

docs/source/software/jupyter.rst

@@ -1,13 +1,22 @@
 ------------
 Jupyter Hub
 ------------
+=======
+Status
+=======
+Released and accessible to all HPC USers at the correct URLs. 
+
 .. _Jupyter Enterprise Gateway: https://jupyter.org/hub
 .. _Jupyter URL: https://deepweb.flinders.edu.au/jupyter
 
+=========
+Overview
+=========
 The `Jupyter Enterprise Gateway`_ is a multi-user environment for JupyterNotebooks. DeepThought has integrated 
 the Jupyter Gateway to allow users to run jobs on the cluster via the native Web Interface.  
 
 If you have access to the HPC, you automatically have access to the Jupyter Lab. You can the JupyterLab Instance 
 via the following `Jupyter URL`_ or manually via https://deepweb.flinders.edu.au/jupyter. Your credentials are the
 the same as the HPC, your FAN and password.
 
+If you are a student with access to the HPC, the above URLs may work - the URL http://deepteachweb.flinders.edu.au/jupyter is guaranteed to work correctly. 

docs/source/software/matlab.rst

@@ -0,0 +1,18 @@
+------------------
+Mathworks MATLAB 
+------------------
+=======
+Status
+=======
+Currently Under Development. Contact the HPC Support Team for more information. 
+
+.. MathWorks MATLAB: https://au.mathworks.com/
+
+
+=================
+Overview 
+=================
+`Mathworks MATLAB`_ is a leading mathematical analysis suite.  Coupled with their 
+Parallel Server, the new MATLAB Integration will enable the full power of DeepThought 
+and a greatly enhanced user experience. 
+

docs/source/software/singularity.rst

@@ -1,9 +1,15 @@
 ========================
 Singularity Containers
 ========================
+=======
+Status
+=======
+Released and available as a module on the HPC.  
+
+==========
+Overview
+==========
 .. _here: https://singularity.hpcng.org/user-docs/master/
 
 Singularity is the most common container engine used for HPC.  With the capability to run most container formats, including Docker, 
-DeepThought now allows for containerised workflows via the singularity container engine. More information and a user guide is available `here`_, or at https://singularity.hpcng.org/user-docs/master/. 
-
-Singularity is available as a Software Module on DeepThought.
+DeepThought now allows for containerised workflows via the singularity container engine. More information and a user guide is available `here`_, or at https://singularity.hpcng.org/user-docs/master/. 

docs/source/software/softwaresuitesoverview.rst

@@ -14,8 +14,10 @@ List of Enterprise Software Suites
 .. _Jupyter Hub: jupyter.html
 .. _ANSYS: ansys.html
 .. _Singularity Containers: singularity.html 
+.. _MATLAB: matlab.html
 
 1. `Jupyter Hub`_
 2. `ANSYS`_ 
 3. `Singularity Containers`_ 
+4. `MATLAB`_
 

docs/source/userguides/FAME/fame-tutorials.rst

@@ -0,0 +1,111 @@
+===================================================
+FAME: Bioinformatics Series
+===================================================
+----------------
+Activate Python
+----------------
+We are going to use conda to install and update software. Before we begin, we need to activate the python module that is not initiated by default.
+
+We are going to use vi again to add the python module activation so that we don’t need to do it every time. Copy this line and press Enter.
+
+``vim ~/.bashrc``
+
+Use the arrow keys to move to the line after the line that says module load slurm, and press i to switch to --INSERT-- mode.
+
+Paste this text:
+
+``module load python/3.9.6``
+
+Press the Escape key, and then type :wq to save the changes and exit. (Note that here we combine the :w and :q commands!)
+
+Now run this command to activate Python:
+
+``source ~/.bashrc``
+
+You should be able to type this command and get a response now:
+
+``python3.9 --version``
+
+------------------------
+Download Conda  / Mamba 
+------------------------
+
+``From the HPC Team: No need to download conda - there is a module for you! module load Miniconda3 will get you up and running for conda. 
+You can also follow these instruction if you would like you own copy of conda.``
+
+Head to the miniconda download page and download the appropriate installer.
+
+You want the one from Latest Miniconda Installer Links. Get:
+
+    Linux Installer
+    Python with the biggest number
+    Miniconda3 Linux 64-bit
+
+Here is a quick way to download that script!
+
+Right click on the appropriate link, and choose Copy link address. Go back to your terminal window (probably Putty) and type wget and a space, and then paste the URL that you just copied. Press return and it should download the file for you!
+
+The file should be called Miniconda3-latest-Linux-x86_64.sh but in case it is not, just substitute the appropriate file name below. Remember that you can check with ls -ltr to see the newest file downloaded.
+
+Run the miniconda installer:
+
+bash Miniconda3-latest-Linux-x86_64.sh
+
+This will ask you some questions, and you can pretty much accept the default answer to all the questions.
+
+Once the installer has finished the best way to continue is to log out, and then log back in. This will reset your account and you will have conda activated. At the bottom left of your screen you should see it say (base) which means that you are in the base conda installation.
+Conda is great, but Mamba is better!Permalink
+
+Mamba is a drop in replacement for conda, and it is the first thing we will install.
+
+conda install -c conda-forge mamba
+
+This will take a moment to set everything up and figure out what needs to be installed, and then ask you if you are sure.
+
+Type yes and press return and mamba will be installed. From here on out, forget conda and think mamba!
+Install your first bioinformatics packagePermalink
+
+We are going to install prinseq++ and test to see if it works. This will demonstrate how to install a conda package. We make a new environment named bioinformatics and use the conda-forge and bioconda channels to install prinseq++
+
+mamba install -c conda-forge -c bioconda prinseq-plus-plus 
+
+Again, this will work on resolution of the packages for you and ask if you are sure. Once it is complete, you should be able to issue the command:
+
+prinseq++ -v
+
+to see the version of prinseq++ that has been installed.
+Install snakemakePermalink
+
+For the next steps of this tutorial, we are going to use snakemake to run some things on the cluster. So we are going to use mamba to install that:
+
+mamba install -c conda-forge -c bioconda snakemake=5.22.0
+
+    NOTE: At the time of writing, snakemake was introducing a new package called PuLP which makes the pipelining a lot faster, but is a bit glitchy. So we install version 5.22.0 which predates those changes. For more information see this issue, or this issue or this suggestion
+
+Once that has completed, snakemake -v should show you the current version.
+Install other bioinformatics packagesPermalink
+
+You can install pretty much any bioinformatics package using conda. The anaconda website has a complete list and you can visit the bioconda page for more information about bioconda.
+
+but to get started, you might want to install:
+
+mamba install -c conda-forge -c bioconda blast focus diamond
+
+Adding channelsPermalink
+
+It is a pain to keep typing -c conda-forge -c bioconda so we can just add those two channels to our configuration
+
+conda config --add channels conda-forge
+conda config --add channels bioconda
+
+Updating a packagePermalink
+
+With conda (or mamba) you can easily update a package if there is a newer version. For example, to update snakemake you would use:
+
+mamba update snakemake
+
+What this means is the you are responsible for ensuring your software is up-to-date. Or not. If you are working on a set of data analyses you may want to keep all the software at the same version so that each time you do an analysis you get comparable answers. With conda, you have control over the update cycles, but don’t forget from time-to-time you might want to update the software!
+EnvironmentsPermalink
+
+If you want to keep different versions of software or run different pipelines you can do that with conda, in what are called environments. Each one can have different software. conda is clever, because if you have the same software in two different environments you don’t need an entire copy of the software. At this stage, you don’t need to worry about that, and you can just install everything in the base environment. But if you start to run into installation issues, then remember you can separate things into different environments.
+Up nextPermalink

docs/source/userguides/userguideoverview.rst

@@ -0,0 +1,14 @@
+========================
+User Contributed Guides 
+========================
+The HPC Team does its best to write as much as possible on how to use the HPC, we welcome anything that has been written by out users.  
+This is the landing page where we keep track of who has contributed guide directly, or graciously allowed us to tweak existing content. 
+
+.. _FAME: https://fame.flinders.edu.au/
+.. _FAME Tutorials: fame-tutorials.html
+
+-----------------------------
+Centre Contributors
+-----------------------------
+The Flinders Accelerator for Microbiome Exploration has graciously allowed to the HPC Team to mirror their excellent tutorial series on 
+conda and microbiology python packages. The original content can be found on the `FAME`_ homepage.