Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MPI support #172

Open
wants to merge 109 commits into
base: master
from

Conversation

@rukkal
Copy link
Contributor

commented Apr 28, 2017

This PR adds MPI support to Shifter.

The user can activate the MPI support either through the --mpi CLI option of Shifter:

srun -N 2 shifter --mpi --image=ethcscs/dockerfiles:osu5.3.2-mpich3.1.4 /usr/local/libexec/osu-micro-benchmarks/mpi/pt2pt/osu_latency

or through the --shifter-mpi CLI option of the Slurm plugin:

salloc -N 2 --shifter-mpi --image=ethcscs/dockerfiles:osu5.3.2-mpich3.1.4
srun shifter /usr/local/libexec/osu-micro-benchmarks/mpi/pt2pt/osu_latency

How the MPI support works

In order to activate MPI support the compute node must provide an MPI implementation that is MPICH ABI compatible (MPICH ABI compatibility initiative) and the container's application must dynamically link to an MPI library that is also MPICH ABI compatible. We could successfully run MPI applications in containers with MPICH 3.1.4, MVAPICH2 2.2, and Intel MPI Library 2017 Update 1.

The system administrator must specify the location of the compute nodes' MPI libraries through the siteMPISharedLibs parameter in udiRoot.conf. The MPI dependency libraries, i.e. those libraries that the MPI libraries need to access, are configured through the siteMPIDependencyLibs parameter.

Shifter performs the following steps when the user requires the activation of the MPI support:

  1. Searches the MPI shared libraries in the container through the container's ldconfig.
  2. Compares the version numbers in the library filenames to check for ABI compatibility between compute node and container.
  3. Bind mounts the compute node's MPI shared libraries on top of the containers' ones.
  4. Bind mounts the compute node's dependency libraries into the container and updates the container's LD_LIBRARY_PATH accordingly.

Other addition in the PR

We also implemented the Shifter's --version CLI option:

$shifter --version
Shifter 17.04.00
rukkal added 30 commits Oct 19, 2016
mpi support: added inspection of container's ld.so.cache
The script activate_mpi_support.sh searches the ls.so.cache of
the container for references to MPI libraries. When a reference
is found, the corresponding site MPI library is bind mounted into
the container to override the container's library. If no
ABI-compatible site library is found, an error is generated.
mpi_support: added check for dependencies in container image
now we check that the container image containes the "sed" and
"realpath" command line tools. If that is not the case, shifter
issues an error message and aborts.
mpi support: implemented used interface
Now the user can enable the MPI support by specifying the command
line option --mpi or -m.
mpi support: added checks for siteResources config
The siteResources configuration option might be empty,
i.e. unspecified in the configuration file. I added
checks for such case.
added function shifter_setupenv_site_resources
This function recursively searches the site resources
folder. Note: the location of the site resources folder
is specified by the system administrator in the configuration file.

If an executable file is found in the site resources folder (or any
subfolder), the parent directory of such file is added to the
environment variable PATH.

If a shared library is found in the site resources folder (or any
subfolder), the parent directory of such library is added to the
environment variable LD_LIBRARY_PATH.
mpi support: got rid of hard-coded values in activate_mpi_support.sh
The location of the container's root directory and site-resources
directory is no longer hard-coded in activate_mpi_support.sh.
Such values are passed as arguments to the script.
fixed GPU support integration test
During set up time, ldconfig didn't scan the /lib64 folder.
mpi support: check that container has MPI libraries
Now when the user requires MPI support (--mpi command line option),
activate_mpi_support.sh checks that the container's dynamic linker
is configured to load MPI libraries (check output of 'ldconfig -p').
If the dynamic linker is not configured as expected an error message
is issued and the program aborts. Thus, at least in some cases,
we are able to enforce the user to properly configure the dynamic linker
instead of doing LD_LIBRARY_PATH hacks in the container.
mpi support: changed overriding of container's MPI libraries
Now the container's MPI libraries are overridden by bind mounting
the corresponding site MPI libraries on top of the container's ones.
Thus we further enforce the container to use the site libraries,
since there is no way the the container will still be able to access
the overridden libraries.
mpi support: enabled verbose output
Now, if the user specifies the command line option --verbose,
the script activate_mpi_support.sh generates log messages
with information about all the files (libraries, binaries and
configuration files) that are being bind mounted into the container.
gpu support integration test: minor refactoring
Renamed the dummy libraries used for the GPU support's integration
tests, since they will be reused for the MPI support's integration
tests.
mpi support: added checks in integration test
Now the value of the MPI-related environment variables inside the
container is checked.
rukkal and others added 22 commits Apr 11, 2017
Revert "Version bump"
This reverts commit 6647113.
mpi support: encapsulated configurations in mpi struct
Now the values of MPI-specific configurations in udiRoot.conf are
stored in the struct mpi_support_config.
Merge branch 'refactor_mpi_support_configurations' into 'mpi_support'
mpi support: encapsulated configurations in mpi struct

See merge request !35
@dmjacobsen

This comment has been minimized.

Copy link
Contributor

commented Apr 28, 2017

@coveralls

This comment has been minimized.

Copy link

commented Apr 28, 2017

Coverage Status

Coverage increased (+0.09%) to 68.104% when pulling a58461a on rukkal:mpi_support into 4dc2134 on NERSC:master.

mpi support: fixed security hole
The script activate_mpi_support.sh used to execute container's
binaries with root privileges, i.e. the container developer
could forge his own binaries that would have been executed with root
privileges.
@coveralls

This comment has been minimized.

Copy link

commented Jun 19, 2017

Coverage Status

Coverage increased (+0.09%) to 68.104% when pulling f68f974 on rukkal:mpi_support into 4dc2134 on NERSC:master.

@rukkal

This comment has been minimized.

Copy link
Contributor Author

commented Jul 14, 2017

Hi,
with this commit f68f974 Shifter uses only the host's ldconfig binary and doesn't touch the container's one anymore.

@scanon

This comment has been minimized.

Copy link
Member

commented Jul 18, 2017

I don't think that is the behavior we want. There will likely be cases where the host and container glibc versions are too far apart.

@fcruzcscs

This comment has been minimized.

Copy link
Contributor

commented Jul 19, 2017

@scanon Can you elaborate on the problem? it seems to me that glibc between host and container is not an issue.

If you, look at line 204 of src/activate_mpi_support.sh on f68f974 what is being done is that the host uses its own ldconfig, sed, and readlink binaries this is still prior to the chroot so there is no glibc mix.

Note that ldconfig is using the -r option so as that ldconfig does the search on the standard locations but on the container filesystem.

@dmjacobsen

This comment has been minimized.

Copy link
Contributor

commented Mar 25, 2018

this mechanism is not going to be integrated in based on other discussions. I've added on the dev branch the initial modules support, which should allow for functionality-specific, site-definable, user-selectable "modules" to be implemented that can integrate the type of features you are looking for here. The dev branch should be promoted to master within a day or so

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
6 participants
You can’t perform that action at this time.