MR MPI Setup

Instructions on how to compile/run MR-MPI on rush.ccr.buffalo.edu

http://mapreduce.sandia.gov/

http://mapreduce.sandia.gov/doc/Interface_c++.html

Compile mr-mpi shared lib

$ module load intel
$ module load intel-mpi
$ tar xf mrmpi.tar.gz
$ cd mrmpi-22Nov13/src
$ make -f Makefile.shlib mpicc

NOTE: There are specific makefiles in the MAKE directory. I modified one of them to use the intel mpi compilers instead of the gnu ones.

Compile mr-mpi c/c++ examples

$ module load intel
$ module load intel-mpi
$ cd mrmpi-22Nov13/examples
$ make -f Makefile.mpicc

NOTE: you may need to edit the makefile a little b/c of shared lib name changes, I made a custom one for mpiicpc

Run mr-mpi c/c++ examples

$ module load intel
$ module load intel-mpi
$ cd mrmpi-22Nov13/examples
$ wget -r -P /panasas/scratch/kmarcus2/textfiles -A txt http://www.textfiles.com/etext/NONFICTION/
$ mpirun -np 8 ./wordfreq /panasas/scratch/kmarcus2/textfiles/*
$ mpirun -np 8 ./cwordfreq /panasas/scratch/kmarcus2/textfiles/*

NOTE: these examples work on multiple text files so I just downloaded a bunch of text file books from textfile.com

To run the python examples you first need to install PyPar

NOTE: Use version 2.1.5 at http://code.google.com/p/pypar/

$ git clone https://github.com/daleroberts/pypar.git
$ cd pypar
$ module load intel-mpi
$ module load python-epd/7.1.2
$ python setup.py build
$ python setup.py install --user

NOTE: this will install the module in ~/.local

Run the python example

$ module load python-epd/7.1.2
$ module load intel
$ module load intel-mpi
$ module load use.own
$ module load mrmpi-python
$ cd mrmpi-22Nov13/examples
$ mpirun -np 8 python wordfreq.py /panasas/scratch/kmarcus2/textfiles/*

NOTE: the mrmpi-python module is a custom built module that appends to the PYTHONPATH and the LD_LIBRARY_PATH env vars. This module is inside the privatemodules/mrmpi-python directory in this repo. Copy this file to your local private module directoy and then load the module. Note that you will have to change some of the paths in side this file to point to your own install path of MP-MPI.