Skip to content

ORTE

Jump to bottom
Andre Merzky edited this page Mar 7, 2017 · 29 revisions

Open RunTime Environment from OpenMPI

This page documents the installation and configuration of OpenMPI in order to use the ORTE layer as a execution component for RADICAL-Pilot.

General

  • Generally we would like to provide a general installation per system that is used by all users, which will be configured in the main resource config for that resource, and thus the installation needs to be on a place in the filesystem that is readable by all target users.
  • Currently we are working from a git checkout, at some point in time we should be able to move to release tarballs.
  • Because of the use of module files, its relatively easy to have multiple installations next to each other and switch the default easily as well.
  • There will be differences on a system per system basis, we try to capture that as much as possible.
  • One might want to throw some chmod -R a+rX $OMPI_INSTALLED into the mix too to make the files from the installation readable for all.

Toolchain installation

To build OpenMPI, we need the proper versions of automake, autoconf, m4 and libtool. This step generally only need to be done once and the versions are pretty static.

Make sure you use the GNU compiler quite, i.e. on Cray's some variant of:

module switch PrgEnv-pgi PrgEnv-gnu # e.g. Titan
# or
module swap PrgEnv-cray PrgEnv-gnu # i.e. Blue Waters

Setup paths and environments:

# The environments below are only important during build time
# and can generally point anywhere on the filesystem.
# Your homedir is a logical choice.
export OMPI_DIR=$HOME/openmpi
export OMPI_DOWNLOAD=$OMPI_DIR/download
export OMPI_SOURCE=$OMPI_DIR/src
export OMPI_TOOLS_PREFIX=$OMPI_DIR/tools
export PATH=$OMPI_TOOLS_PREFIX/bin:$PATH
# The file system locations of the variables below need a bit more care,
# as this path needs to be accessible during job run time.
# E.g. on Blue Waters, a good location for 
# OMPI_INSTALLED=/projects/sciteam/gk4/openmpi/installed,
# and OMPI_ALL_MODULES=/projects/sciteam/gk4/openmpi/modules.
export OMPI_ALL_MODULES=$OMPI_DIR/modules
export OMPI_MODULE=$OMPI_ALL_MODULES/openmpi
export OMPI_INSTALLED=$OMPI_DIR/installed

Create download directory:

mkdir -p $OMPI_DOWNLOAD
cd $OMPI_DOWNLOAD

Download following files into $OMPI_DOWNLOAD:

  • autoconf-2.69.tar.gz (wget http://ftp.nluug.nl/gnu/autoconf/autoconf-2.69.tar.gz)
  • automake-1.13.3.tar.gz (wget http://ftp.nluug.nl/gnu/automake/automake-1.13.3.tar.gz)
  • libtool-2.4.2.tar.gz (wget http://nl.mirror.babylon.network/gnu/libtool/libtool-2.4.2.tar.gz)
  • m4-1.4.17.tar.gz (wget https://ftp.gnu.org/gnu/m4/m4-1.4.17.tar.gz)

Create source directory:

mkdir -p $OMPI_SOURCE

Install m4:

cd $OMPI_SOURCE
tar -xvzf $OMPI_DOWNLOAD/m4-1.4.17.tar.gz
cd m4-1.4.17
./configure --prefix=$OMPI_TOOLS_PREFIX
make
make install

Install autoconf:

cd $OMPI_SOURCE
tar -xvzf $OMPI_DOWNLOAD/autoconf-2.69.tar.gz
cd autoconf-2.69
./configure --prefix=$OMPI_TOOLS_PREFIX
make
make install

Install automake:

cd $OMPI_SOURCE
tar -xvzf $OMPI_DOWNLOAD/automake-1.13.3.tar.gz
cd automake-1.13.3
./configure --prefix=$OMPI_TOOLS_PREFIX
make
make install

Install libtool:

cd $OMPI_SOURCE
tar -xvzf $OMPI_DOWNLOAD/libtool-2.4.2.tar.gz
cd libtool-2.4.2
./configure --prefix=$OMPI_TOOLS_PREFIX
make
make install

OpenMPI installation

Last know good git commit: 4c9f7af.

Prepare OMPI source code:

cd $OMPI_SOURCE
git clone https://github.com/open-mpi/ompi.git
cd ompi
git checkout 4c9f7af
./autogen.pl

Build and install:

OMPI_LABEL=$(date '+%Y_%m_%d')
export OMPI_BUILD=$OMPI_DIR/build/$OMPI_LABEL
mkdir -p $OMPI_BUILD
cd $OMPI_BUILD
CFLAGS=-O3 CXXFLAGS=-O3 $OMPI_SOURCE/ompi/configure --enable-orterun-prefix-by-default --with-devel-headers --disable-debug --enable-static --disable-dlopen --prefix=$OMPI_INSTALLED/$OMPI_LABEL
make
make install

Module files

A module file allows us to capture all the paths with a system module wrapper.

Setup module structure:

mkdir -p $OMPI_MODULE

For example, to create a module file for the version we have just installed, we can create a file ...

cat > $OMPI_MODULE/nov18

... and add the following content. Make sure that the $mpiroot path matches the with the $OMPI_INSTALLED path used in the environment variables used above and that the nov18 label matches too.

#%Module########################################################################

##
## Open MPI from git
##
proc ModulesHelp { } {
        global version

        puts stderr "Sets up a dynamic build of Open MPI HEAD from git."
        puts stderr "Version $version"
}

module-whatis	"Dynamic build of Open MPI from git."

set 		version		nov18

set mpiroot /projects/sciteam/gki/openmpi/installed/$version
prepend-path	PATH            $mpiroot/bin
prepend-path	LD_LIBRARY_PATH $mpiroot/lib
prepend-path	MANPATH 	$mpiroot/share/man

Test

Find module

Add the created module file to the module search path:

module use --append $OMPI_ALL_MODULES

Show the just created module file:

module avail openmpi

Should give an output like:

------------------- /u/sciteam/marksant/openmpi/modules -------------------
openmpi/nov18

Load module

Load the just created module:

module load openmpi/$OMPI_LABEL

Most basic test to see if installation is reachable in the path:

orte-info | head

Interactive job

To launch an interactive session on Blue Waters with two nodes:

qsub -I -l nodes=2:ppn=32 -A gk4 -q debug

Within that session setup the things as configured above:

# 
# Set environment variables as above
#

#
# Add our own modules to the modules search path
#
module use --append $OMPI_ALL_MODULES

#
# Load openmpi
#
module load openmpi/nov18

#
# test path
#
orte-info | head

#
# Launch persistent ORTE DVM and write uri to file and move to the background
#
orte-dvm --report-uri dvm_uri &

#
# After the message "DVM ready" appears, one can start to submit tasks to the dvm
# (in this particular case one task per node)
#
orterun --hnp file:dvm_uri -N 1 /bin/hostname

#
# Should give the following output if we had requested two nodes from torque
#
[ORTE] Task: 0 is launched! (Job ID: [53225,1])
nid03578
nid03579
[ORTE] Task: 0 returned: 0 (Job ID: [53225,1])

Hooray!!

RADICAL-Pilot resource configuration

Now the right module needs to be added to the resource configuration, e.g. https://github.com/radical-cybertools/radical.pilot/blob/devel/src/radical/pilot/configs/resource_ncsa.json:

        "pre_bootstrap_1"             : [
            "module switch PrgEnv-cray PrgEnv-gnu",
            "module load bwpy",
            "module use --append /projects/sciteam/gki/openmpi/modules",
            "module load openmpi/nov18"
        ],

HOWTO update orte-cffi in the agent VE on Titan:

  source ~/sandbox/ve_titan
  module swap PrgEnv-pgi PrgEnv-gnu
  module load python
  module load python_pip
  module use --append /lustre/atlas/world-shared/bip103/modules
  module load openmpi/DEVEL-STATIC
  export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:/lustre/atlas/world-shared/bip103/ffi/pkgconfig
  export OMPI_MCA_timer_require_monotonic=false
  pip install --upgrade orte-cffi

Note: https://bitbucket.org/cffi/cffi/issues/226/setuptools-installs-cffi-and-pycparser

TODO

  • Compare and contrast static vs dynamic linked builds
  • Construct list/table with proper install locations for the various systems
Clone this wiki locally