MPI

We currently have OpenMPI 1.8 installed with a prefix of /nfsopt/openmpi-1.8.

Configuration

To use this installation you have to set the PATH and LD_LIBRARY_PATH environment variables using one of the two approaches:

Using .bashrc

First add the following lines to your .bashrc file:

export PATH=/nfsopt/openmpi-1.8/bin:$PATH
export LD_LIBRARY_PATH=/nfsopt/openmpi-1.8/lib:$LD_LIBRARY_PATH

But you must make sure to comment out the following line in the default Ubuntu .bashrc file to make .bashrc run in non-interactive mode:

[ -z "$PS1" ] && return

Using a --prefix

Or the user can add --prefix /nfsopt/openmpi-1.8 to their mpiexec runs:

mpiexec --prefix /nfsopt/openmpi-1.8 -np 80 ./a.out

Passwordless SSH

Next, you need to setup passwordless SSH between the hosts. You also need to make sure all of the hosts are listed in $HOME/.ssh/known_hosts. If you are having problems with MPI, try removing all of the entries from the knows_hosts file.

Usage

Then you can use one of the MPI compilers (mpicc, mpif90, etc.) to build your program. To run your program use mpiexec:

mpiexec -n 80 ./a.out

We have a total of 412+216=80 cores with 4 GB RAM per core and have setup a default hostfile. If you want to change the order you can specify your own hostfile. Here is our default one:

192.168.193.1 slots=12 max-slots=12
192.168.193.3 slots=12 max-slots=12
192.168.193.5 slots=12 max-slots=12
192.168.193.7 slots=12 max-slots=12
192.168.193.9 slots=16 max-slots=12
192.168.193.11 slots=16 max-slots=12

To run a job with a custom hostfile do the following:

mpiexec --hostfile hosts.txt -n 80 ./a.out

Finally, you should use screen to background long running parallel jobs!