Merge pull request #59 from uit-no/gaussian_guide

Update of Gaussian documentation on Stallo. Newest module, change of …

applications/chemistry/Gaussian/Gaussian.rst

@@ -15,7 +15,6 @@ Related information
  firsttime_gaussian.rst
  GaussView.rst
  gaussian_on_stallo.rst
- advanced.rst
 
 
 General Information
@@ -38,11 +37,11 @@ License information
 
 The license of GAUSSIAN is commercial/proprietary.
 
-The license of Gaussian constitutes of 4 site licenses for the 4 current host institutions of NOTUR installations; NTNU, UiB, UiO, UiT. In principle, only person from one of these institutions have access to the Gaussian Software system installed on Stallo.
+The license of Gaussian constitutes of 4 site licenses for the 4 current host institutions of NOTUR installations; NTNU, UiB, UiO, UiT. In principle, only person from one of these institutions have access to the Gaussian Software system installed on Stallo. Note that users that do not come from one of the abovementioned institutions still may be granted access, but they need to document access to a valid license for the verion in question first.
 
 * To get access to the code, you need to be in the gaussian group of users.
-* To be in the gaussian group of users, you need either to be a member of the abovementioned
-  institutions or provide proof of holding a license on your own.
+* To be in the gaussian group of users, you need be qualified for it - see above.
+
 
 Citation
 --------
@@ -65,15 +64,15 @@ Use
 
 .. code-block:: bash
 
-    $ module avail Gaussian
+    $ module avail Gaussian/
 
 to see which versions of Gaussian are available. Use
 
 .. code-block:: bash
 
- $ module load Gaussian/<version> # i.e 09.d01
+ $ module load Gaussian/<version> # i.e g16_B.01
 
-to get access to any given version of Gaussian.
+to load the version of Gaussian of your desire.
 
 
 About the Gaussian version(s) installed on Stallo

applications/chemistry/Gaussian/firsttime_gaussian.rst

@@ -28,14 +28,13 @@ You can also download the input file here: :download:`Caffeine-input<../files/ca
 Gaussian runscrip example
 -------------------------
 
-.. include:: ../files/job_g09.sh
+.. include:: ../files/job_gaussian.sh
    :literal:
 
 
-**NB: Note that we for Gaussian advice to specify both nodes and core/node. This is due to the special way Gaussian sets up parallel jobs!**
-
-You can also download the runscript here: :download:`Gaussian run script<../files/job_g09.sh>`
+**NB: Note that there has been a change in how the local IB adaption wrapper has been set up. Read the comments in the run script example!**
 
+You can also download the runscript here: :download:`Gaussian run script<../files/job_gaussian.sh>`
 
 
 The runscript example and the input file are also on Stallo
@@ -45,7 +44,7 @@ Type:
 
 .. code-block:: bash
 
-	module load Gaussian/9.d01
+	module load Gaussian/g16_B.01
 	cd <whereevertestfolderyouhave> # For instance testgau
 	cp -R /global/hds/software/notur/apprunex/Gaussian/* .
 
@@ -54,9 +53,7 @@ When you have all the necessary files in the correct folders, submit the job by
 
 .. code-block:: bash
 
-	sbatch job_g09.sh
+	sbatch job_gaussian.sh
 
-To verify that nothing has gone wrong, compare the energy of g09_caffeine.out with the correct energy of HF=-680.4153661 for this calculation.
-The energy should ideally be identical or close to identical. After that, you may alter the varia\
-bles in the shell script as much as you like to adapt to your own jobs. Good luck.
+To verify that nothing has gone wrong, check the output of the calculations. In Gaussian output, you allways get an entertaining quote at the end. Good luck. 
 

applications/chemistry/Gaussian/gaussian_on_stallo.rst

@@ -6,12 +6,12 @@ About the Gaussian install on Stallo
 
 This page contains info about special features related to
 the Gaussian install made on Stallo, but also generall issues
-vaguely documented elsewhere.
+related to Gaussian only vaguely documented elsewhere.
 
 Gaussian on Stallo
 ------------------
 
-First note is that the Gaussian install on Stallo is the Linda parallel version, so it
+First note that the Gaussian install on Stallo is the Linda parallel version, so it
 scales somewhat initially. On top of this, Gaussian on Stallo is installed with a little trick, where the executables are intercepted before launced, and an
 alternative socket library is loaded. This enables the possibility of running Gaussian natively on the infiniband
 network giving us two advantages:
@@ -21,7 +21,7 @@ network giving us two advantages:
 
 But since we do this trick, we are greatly depending on altering the specific nodal adress into the
 input file: To run gaussian in parallel requires the additional keywords %LindaWorkers and %NProcshared
-in the Link 0 part of the input file. This is taken care of by a wrapper, and is commented in jobscript example file.
+in the Link 0 part of the input file. This is taken care of by a wrapper we have chosen to put around the original g<VER> binary (g09, g16 etc) in each individual version folder. This is also commented in the job script example for first-time runs. 
 Please do use this script or similar when submitting jobs; it will benefit all of us.
 
 We have also taken care of the rsh/ssh setup in our installation procedure, to avoid .tsnet.config dependency for users.
@@ -32,27 +32,29 @@ About memory allocation for Gaussian
 * In general, Gaussian takes care of memory allocation internally.
 
 That means that if the submitted job needs more memory per core than what is in average available on the node, it will automaticall scale down the number o\
-f cores to mirror the need. This also means that you allways should ask for full nodes when submitting Gaussian 09 jobs!
+f cores to mirror the need. This also means that you allways should ask for full nodes when submitting Gaussian jobs on Stallo! This is taken care of by the exclusive flag and commented in the job script example.
 
 The %mem allocation of memory in the Gaussian input file means two things:
 
 * In general it means memory/node – for share between nprocshared, and aditional to the memory allocated pr. process. This is also documented by Gaussian.
-* For the mother superior (mother process/node) it also represents the network buffer allocated by Linda. So basically G09 takes a part and Linda takes a part the same
-  size – thus you should never ask for more than half of the physical memory on the nodes, unless they have swap space available - which you never should assume.
-  Basically, the general %mem limit will allways be half of the physical memory pool given in MB instead of GB - 16000MB instead of 16GB since this leaves a small part for the system.
+* For the mother superior (mother process/node) it also represents the network buffer allocated by Linda. So basically the main Gaussian process takes a part and Linda communication process takes a part equally sized – thus you should never ask for more than half of the physical memory on the nodes, unless they have swap space available - which you never should assume.
+  Basically, the general %mem limit will allways be half of the physical memory pool given in MB instead of GB - 16000MB instead of 16GB since this leaves a small part for the system. (That is why we would actually advise to use 15GB as maximum %mem size.
 
 For core-count, node-count and amounts of memory on Stallo, see :doc:`/stallo/stallo`
 
-
 Scalability and parallel performance of Gaussian on Stallo
 ----------------------------------------------------------
 
 Due to the nifty trick mentioned above, we have a somewhat more generous policy when it comes to
 allocating cores/nodes to Gaussian jobs:
 
-#. We do advice people to use up to 16 nodes, at least 12 full nodes, with the current install on Stallo. We have proved acceptale scaling of the current
-   Gaussian install beyond 16 nodes with 16 cores on each. So in essence; at least up to 16 nodes. Enjoy.
-#. Linda networking overhead seems to hit hard around this amount of cores; causing us to be somewhat reluctant to advice going to more than this. On the
-   other hand, this will give you a maximum of 320 cores with the most recent hardware available on Stallo.
+#. We do advice people to use up to 256 cores (slurm syntax tasks). We have proved acceptale scaling of the current
+   Gaussian install beyond 16 nodes for the jobs that do scale outside of one node (i.e. the binaries in the $gXXroot/linda-exe folder). So in essence; enjoy!
+#. Linda networking overhead seems to hit hard around this amount of cores; causing us to be somewhat reluctant to advice going to more than this. Meddling with this requires a bit more insight, which we are happy to help out with but reluctant to advise beginners do at an early stage. Though; this will give a potential of up to 320 cores in parallell on Stallo, which is more than most other places anyway. 
+#. Since we have two different architectures with two different core counts on Stallo, the ----exclusive flag mentioned	above is also important to ensure that the distribution of jobs accross the whole system are done in a rather flexible (and painless) way. 
+Large temporary outputs on Stallo
+----------------------------------
+
+As commented here :doc:`/storage/storage` there is an issue related to very large temporary files on Stallo. Please read up on it at act accordingly. This issue is also commented in the job script example. 
 
-We have been looking into issues related to this, and expect this situation to change when the new Gaussian Major release comes out late 2016/ early 2017.
+ 

applications/chemistry/files/job_gaussian.sh

@@ -0,0 +1,80 @@
+#!/bin/bash -l
+
+################### Gaussian Job Batch Script Example ###################
+# SLURM-section
+#SBATCH --job-name=gaussian_runex
+#SBATCH --ntasks=80 
+## Stallo has got 16 or 20 cores/node, thus the optimal distribution of cores will be a number that is a full number sum of both subsets. 
+#SBATCH --time=00:59:00 # Syntax is DD-HH:MM:SS. 
+#SBATCH --mem-per-cpu=1500MB # Giving a total of 30 000MB/node and leaving a bit over 2GB for the system to survive.
+#SBATCH --output=gaussina_runex.log
+#SBATCH --mail-type=ALL
+#SBATCH --exclusive
+######################################
+
+# Defining job other job variable and settings:
+
+input=caffeine # Name of input without extention
+ext=com # We use the same naming scheme as the software default extention
+
+# We load all the default program system settings with module load:
+# But first we flush the environment for unwanted settings
+
+module --quiet purge
+module load Gaussian/g16_B.01
+
+# This one is important; setting the heap-size for the job to 20GB:                                                                         
+export GAUSS_LFLAGS2="--LindaOptions -s 20000000"
+
+# Now we create working directory and temporary scratch for the job(s):
+# We assume no predefined variables except defaults from used software.
+
+export GAUSS_SCRDIR=/global/work/$USER/$SLURM_JOB_ID
+mkdir -p $GAUSS_SCRDIR
+
+# Preparing and moving inputfiles to tmp:
+
+submitdir=$SLURM_SUBMIT_DIR # Assuming slurm batch resource allocation system
+tempdir=$GAUSS_SCRDIR
+
+cp $submitdir/${input}.${ext} $tempdir
+cd $tempdir
+
+# Typically Gaussian creates rather large temp-files, and they will clog physical drives easilly of not split into smaller parts:
+
+lfs setstripe –c 8 .
+
+# NOTE: Preparation of inputfile is done by the wrapper. For convenience we have moved the wrapper to work as an intermediate step around the binary, the old setup
+# does exist but requires that you uncomment the lines below and comment the lines in the  "Running the program"-paragraph.
+
+#Gaussian.prep $input
+#time g16 < ${input}.${ext} > gaussian_$input.out # This is for Gaussian 16, for Gaussian 09 you need g09.
+
+# Running the program and cleaning up:
+
+# NOTE: We have now placed a wrapper around the binary so that the startup command for Gaussian has changed.
+
+time g16.ib ${input}.${ext} > gaussian_$input.out
+
+cp gaussian_$input.out $submitdir
+cp $input.chk $submitdir
+
+# Make sure to move all essential files specific for the given job/software.
+# To zip some of the output might be a good idea if the resultsfiles are large.
+
+# ALWAYS clean up after yourself. Please do uncomment the following line
+#cd $submitdir
+#if [ -z $tempdir ]
+#then
+#    echo "tempdir not set, please clean up manually!"
+#else
+#    cd $tempdir
+#    rm *
+#    cd $submitdir
+#    rmdir $temdir
+#fi
+#
+echo "Job finished at"
+date
+#
+exit 0