Added more Singularity def files

Added Singularity image index to docs

sharc/groupnodes/dgx-1.rst

@@ -47,3 +47,8 @@ Run your script with the ``qsub`` command ::
 You can use ``qstat`` command to check the status of your current job. An output file is created in your home directory that captures your script's outputs.
 
 See :ref:`sge-queue` for more information on job submission and the Sun Grid Engine scheduler.
+
+Deep Learning on the DGX-1
+--------------------------
+
+Many popular Deep Learning packages are available to use on the DGX-1 and the ShARC cluster. Please see :ref:`DeepLearning_sharc` for more information.

sharc/index.rst

@@ -20,4 +20,4 @@ This is the documentation for ShARC (Sheffield Advanced Research Computer), The
 * :ref:`sharc-differences`
 * :ref:`GPUComputing_sharc`
 * :ref:`DeepLearning_sharc`
-* :ref:`group_nodes_sharc`
+* :ref:`groupnodes_sharc`

sharc/software/apps/caffe.rst

@@ -9,32 +9,57 @@ Caffe
 
 `Caffe <http://caffe.berkeleyvision.org/>`_ is a deep learning framework made with expression, speed, and modularity in mind. It is developed by the Berkeley Vision and Learning Center (`BVLC <http://bvlc.eecs.berkeley.edu/>`_) and by community contributors.
 
+About Caffe on ShARC
+--------------------
+
+Caffe is available on ShARC as both Singularity images and as a module.
+
+This software and documentation is maintained by the `RSES group <http://rse.shef.ac.uk/>`_ and `GPUComputing@Sheffield <http://gpucomputing.shef.ac.uk/>`_. For feature requests or if you encounter any problems, please raise an issue on the `GPU Computing repository <https://github.com/RSE-Sheffield/GPUComputing/issues>`_.
 
 
 Caffe Singularity Images
 ------------------------
 
 Singularity images are self-contained virtual machines similar to Docker. For more information on Singularity and how to use the images, see :ref:`singularity_sharc`.
 
-The following Singularity images are available on ShARC and can also be downloaded for use on your local machine:
+A symlinked file is provided that always point to the latest image: ::
 
-* CPU Caffe rc5, Ubuntu 16.04, GCC 5.4.0
-    * Image path: ``/usr/local/packages/singularity/images/caffe/rc5-CPU-Ubuntu16.04.img``
-    * Def file: `Caffe CPU </sharc/software/apps/singularity/caffe_cpu.def>`
-* GPU Caffe rc5, Ubuntu 16.04, CUDA 8, cuDNN 5.0, GCC 5.4.0
-    * Image path: ``/usr/local/packages/singularity/images/caffe/rc5-GPU-Ubuntu16.04-CUDA8-cudNN5.0.img``
-    * Def file: `Caffe GPU </sharc/software/apps/singularity/caffe_gpu.def>`
+  #CPU Caffe
+  /usr/local/packages/singularity/images/caffe/cpu.img
+
+  #GPU Caffe
+  /usr/local/packages/singularity/images/caffe/gpu.img
 
 To get a bash terminal in to an image for example, use the command: ::
 
-  singularity exec /usr/local/packages/singularity/images/caffe/rc5-CPU-Ubuntu16.04.img /bin/bash
+  singularity exec /usr/local/packages/singularity/images/caffe/gpu.img /bin/bash
 
 The ``exec`` command can also be used to call any command/script inside the image e.g. ::
 
-  singularity exec /usr/local/packages/singularity/images/caffe/rc5-CPU-Ubuntu16.04.img "caffe train your_solver.prototxt"
+  singularity exec /usr/local/packages/singularity/images/caffe/gpu.img "caffe train your_solver.prototxt"
 
 The paths ``/fastdata``, ``/data``, ``/home``, ``/scratch``, ``/shared`` are automatically mounted to your ShARC filestore directories. For GPU-enabled images the ``/nvlib`` and ``/nvbin`` is mounted to the correct Nvidia driver version for the node that you're using.
 
+Image Index
+^^^^^^^^^^^
+
+Paths to the actual images and definition files are provided below for downloading and building of custom images.
+
+* Shortcut to Latest Image
+    * CPU
+        * ``/usr/local/packages/singularity/images/caffe/cpu.img``
+    * GPU
+        * ``/usr/local/packages/singularity/images/caffe/gpu.img``
+* Images list
+    * CPU
+        * Latest: rc5-CPU-Ubuntu16.04 (Python 2.7)
+            * Path: ``/usr/local/packages/singularity/images/caffe/rc5-CPU-Ubuntu16.04.img``
+            * Def file: `/sharc/software/apps/singularity/caffe_cpu.def </sharc/software/apps/singularity/caffe_cpu.def>`
+    * GPU
+      * Latest: rc5-GPU-Ubuntu16.04-CUDA8-cudNN5.0 (Python 2.7)
+          * Path: ``/usr/local/packages/singularity/images/caffe/rc5-GPU-Ubuntu16.04-CUDA8-cudNN5.0.img``
+          * Def file: `/sharc/software/apps/singularity/caffe_gpu.def </sharc/software/apps/singularity/caffe_gpu.def>`
+
 Using the Caffe Module
 ----------------------
 

sharc/software/apps/singularity/caffe_cpu.def

@@ -9,6 +9,13 @@ From: bvlc/caffe:cpu
 %post
 	#Post setup script
 
+	#Use bash as default shell
+	echo "\n #Using bash as default shell \n" >> /environment
+	echo 'SHELL=/bin/bash' >> /environment
+
+	#Make environment file executable
+	chmod +x /environment
+
 	#Default mount paths
 	mkdir /scratch /data /shared /fastdata
 

sharc/software/apps/singularity/caffe_gpu.def

@@ -9,6 +9,13 @@ From: bvlc/caffe:gpu
 %post
 	#Post setup script
 
+	#Use bash as default shell
+	echo "\n #Using bash as default shell \n" >> /environment
+	echo 'SHELL=/bin/bash' >> /environment
+
+	#Make environment file executable
+	chmod +x /environment
+
   #Default mount paths
 	mkdir /scratch /data /shared /fastdata
 

sharc/software/apps/singularity/tensorflow_cpu.def

@@ -0,0 +1,60 @@
+Bootstrap: docker
+From: nvidia/cuda:8.0-cudnn5-devel-ubuntu16.04
+
+%setup
+	#Runs on host
+	#The path to the image is $SINGULARITY_ROOTFS
+
+
+
+%post
+	#Post setup script
+
+	#Use bash as default shell
+	echo "\n #Using bash as default shell \n" >> /environment
+	echo 'SHELL=/bin/bash' >> /environment
+
+	#Make environment file executable
+	chmod +x /environment
+
+	#Default mount paths
+	mkdir /scratch /data /shared /fastdata
+
+	#Nvidia Library mount paths
+	mkdir /nvlib /nvbin
+
+
+  #Updating and getting required packages
+  apt-get update
+  apt-get install -y wget git vim
+
+  #Creates a build directory
+  mkdir build
+  cd build
+
+  #Download and install Anaconda
+  CONDA_INSTALL_PATH="/usr/local/anaconda3-4.2.0"
+  wget https://repo.continuum.io/archive/Anaconda3-4.2.0-Linux-x86_64.sh
+  chmod +x Anaconda3-4.2.0-Linux-x86_64.sh
+  ./Anaconda3-4.2.0-Linux-x86_64.sh -b -p $CONDA_INSTALL_PATH
+
+  #Add Anaconda path
+  echo "\n #Anaconda paths \n" >> /environment
+	echo 'export PATH="'$CONDA_INSTALL_PATH'/bin:$PATH"' >> /environment
+
+	#Loads the environment file
+  . /environment
+
+  #Install Tensorflow
+  TF_PYTHON_URL="https://storage.googleapis.com/tensorflow/linux/cpu/tensorflow-1.0.1-cp35-cp35m-linux_x86_64.whl"
+  pip install --ignore-installed --upgrade $TF_PYTHON_URL
+
+	#Install Keras
+	pip install keras
+
+%runscript
+	#Run every time image starts
+
+
+%test
+	#Test that script is a success

sharc/software/apps/singularity/tensorflow_gpu.def

@@ -0,0 +1,71 @@
+Bootstrap: docker
+From: nvidia/cuda:8.0-cudnn5-devel-ubuntu16.04
+
+%setup
+	#Runs on host
+	#The path to the image is $SINGULARITY_ROOTFS
+
+
+
+%post
+	#Post setup script
+
+	#Use bash as default shell
+	echo "\n #Using bash as default shell \n" >> /environment
+	echo 'SHELL=/bin/bash' >> /environment
+
+	#Make environment file executable
+	chmod +x /environment
+
+	#Default mount paths
+	mkdir /scratch /data /shared /fastdata
+
+	#Nvidia Library mount paths
+	mkdir /nvlib /nvbin
+
+	#Add nvidia driver paths
+	echo "\n #Nvidia driver paths \n" >> /environment
+	echo 'export PATH="/nvbin:$PATH"' >> /environment
+	echo 'export LD_LIBRARY_PATH="/nvlib:$LD_LIBRARY_PATH"' >> /environment
+
+	#Add CUDA paths
+	echo "\n #Cuda paths \n" >> /environment
+	echo 'export CPATH="/usr/local/cuda/include:$CPATH"' >> /environment
+	echo 'export PATH="/usr/local/cuda/bin:$PATH"' >> /environment
+	echo 'export LD_LIBRARY_PATH="/usr/local/cuda/lib64:$LD_LIBRARY_PATH"' >> /environment
+	echo 'export CUDA_HOME="/usr/local/cuda"' >> /environment
+
+  #Updating and getting required packages
+  apt-get update
+  apt-get install -y wget git vim
+
+  #Creates a build directory
+  mkdir build
+  cd build
+
+  #Download and install Anaconda
+  CONDA_INSTALL_PATH="/usr/local/anaconda3-4.2.0"
+  wget https://repo.continuum.io/archive/Anaconda3-4.2.0-Linux-x86_64.sh
+  chmod +x Anaconda3-4.2.0-Linux-x86_64.sh
+  ./Anaconda3-4.2.0-Linux-x86_64.sh -b -p $CONDA_INSTALL_PATH
+
+  #Add Anaconda path
+  echo "\n #Anaconda paths \n" >> /environment
+	echo 'export PATH="'$CONDA_INSTALL_PATH'/bin:$PATH"' >> /environment
+
+  #Loads the environment file
+  . /environment
+
+  #Install Tensorflow
+  TF_PYTHON_URL="https://storage.googleapis.com/tensorflow/linux/gpu/tensorflow_gpu-1.0.1-cp35-cp35m-linux_x86_64.whl"
+  pip install --ignore-installed --upgrade $TF_PYTHON_URL
+
+	#Install Keras
+	pip install keras
+
+%runscript
+	#Run every time image starts
+
+
+%test
+	#Test that script is a success

sharc/software/apps/singularity/theano.def

@@ -0,0 +1,70 @@
+Bootstrap: docker
+From: nvidia/cuda:8.0-cudnn5-devel-ubuntu16.04
+
+%setup
+	#Runs on host
+	#The path to the image is $SINGULARITY_ROOTFS
+
+%post
+	#Post setup script
+
+	#Use bash as default shell
+	echo "\n #Using bash as default shell \n" >> /environment
+	echo 'SHELL=/bin/bash' >> /environment
+
+	#Make environment file executable
+	chmod +x /environment
+
+	#Default mount paths
+	mkdir /scratch /data /shared /fastdata
+
+
+	#Nvidia Library mount paths
+	mkdir /nvlib /nvbin
+
+	#Add nvidia driver paths
+	echo "\n #Nvidia driver paths \n" >> /environment
+	echo 'export PATH="/nvbin:$PATH"' >> /environment
+	echo 'export LD_LIBRARY_PATH="/nvlib:$LD_LIBRARY_PATH"' >> /environment
+
+
+	#Add CUDA paths
+	echo "\n #Cuda paths \n" >> /environment
+	echo 'export CPATH="/usr/local/cuda/include:$CPATH"' >> /environment
+	echo 'export PATH="/usr/local/cuda/bin:$PATH"' >> /environment
+	echo 'export LD_LIBRARY_PATH="/usr/local/cuda/lib64:$LD_LIBRARY_PATH"' >> /environment
+	echo 'export CUDA_HOME="/usr/local/cuda"' >> /environment
+
+  #Updating and getting required packages
+  apt-get update
+  apt-get install -y wget git vim
+
+  #Creates a build directory
+  mkdir build
+  cd build
+
+  #Download and install Anaconda
+  CONDA_INSTALL_PATH="/usr/local/anaconda3-4.2.0"
+  wget https://repo.continuum.io/archive/Anaconda3-4.2.0-Linux-x86_64.sh
+  chmod +x Anaconda3-4.2.0-Linux-x86_64.sh
+  ./Anaconda3-4.2.0-Linux-x86_64.sh -b -p $CONDA_INSTALL_PATH
+
+  #Add Anaconda path
+  echo "\n #Anaconda paths \n" >> /environment
+	echo 'export PATH="'$CONDA_INSTALL_PATH'/bin:$PATH"' >> /environment
+
+	#Loads the environment file
+  . /environment
+
+  #Install Theano
+	conda install -y scipy nose pydot-ng theano pygpu
+
+	#Install Keras
+	pip install keras
+
+%runscript
+	#Run every time image starts
+
+
+%test
+	#Test that script is a success