Running MPICH2 on Yarn
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.



MPICH-yarn is an application running on Hadoop YARN that enables MPI programs running on Hadoop YARN clusters.


As a prerequisite, you need to

  1. The cluster has been deployed Hadoop YARN and HDFS.
  2. Each node in the cluster has installed mpich-3.1.2 and its ./bin folder has been added to PATH.

This version of mpich-yarn uses MPICH-3.1.2 as implementation of MPI and uses ssh as communication daemon.

##Recommended Configuation

  1. Ubuntu 12.04 LTS
  2. hadoop 2.4.1
  3. gcc 4.6.3
  4. jdk 1.7.0_25
  5. Apache Maven 3.2.3


To compile MPICH-yarn, first you need to have maven installed. Then type command at source folder:

mvn clean package -Dmaven.test.skip=true

You need to ensure Internet connected as maven needs to download plugins on the maven repository, this may take minutes.

After this command, you will get mpich2-yarn-1.0-SNAPSHOT.jar at ./target folder. This is the application running at YARN to execute MPI programs.


There are many tutorials on the Internet about configuring Hadoop. However, there are many troubles in configuring YARN to make it work well with mpich2- yarn. To save your time, here is a sample configuration that has successfully run in our cluster for your reference.




			The HDFS address that stores temporary file like:
			MPICH-YARN will create a temporary RSA key pair for
			password-less login and automatically configure it. 
			All of your hosts should enable public_key login.

#Submit Jobs


On the client nodes:

mpicc -o cpi cpi.c
hadoop jar mpich2-yarn-1.0-SNAPSHOT.jar -a cpi -M 1024 -m 1024 -n 2

Hello world

hadoop jar mpich2-yarn-1.0-SNAPSHOT.jar -a hellow -M 1024 -m 1024 -n 2


svn checkout plda  # Prepare source code
cd plda
make  # call mpicc to compile
cd ..

Put the input data to the hdfs (P.S. there is a testdata in the PLDA source code dir):

hadoop fs -mkdir /group/dc/zhuoluo.yzl/plda\_input
hadoop fs -put plda/testdata/test\_data.txt /group/dc/zhuoluo.yzl/plda\_input/
hadoop jar mpich2-yarn-1.0-SNAPSHOT.jar -a plda/mpi\_lda -M 1024 -m 1024 -n 2\
 -o "--num_topics 2 --alpha 0.1 --beta 0.01 --training_data_file MPIFILE1 --model_file MPIOUTFILE1 --total_iterations 150"\
 -DMPIFILE1=/group/dc/zhuoluo.yzl/plda_input -SMPIFILE1=true -OMPIOUTFILE1=/group/dc/zhuoluo.yzl/lda_model_output.txt -ppc 2