We highly encourage to use external (either custom-built or system-provided) MPI installations in production scenarious to get ultimate performance.
In Building MPI from source section of mpi4py
documentation you can find executive instructions for building some of the open-source MPI implementations out there with support for shared/dynamic libraries on POSIX environments.
Once you have a working MPI implementation, you will have to adapt your PATH
environment variable to use the installed MPI version.
export PATH=path/to/mpi/bin:$PATH
Then, you can install unidist
and the required dependencies for the MPI backend.
pip install unidist[mpi]
Now you can use unidist on MPI backend using the installed MPI implementaion.
The following instructions will help you install Intel MPI from Intel oneAPI HPC Toolkit to use it as the unidist's backend. We will use an offline installer an an example but you are free to use other installation options.
- Create a directory for installing Intel MPI and go into it. You can do this in a terminal by typing
mkdir local
cd local
- Download a toolkit installer from Intel oneAPI HPC Toolkit, e.g., using
wget
command
wget https://registrationcenter-download.intel.com/akdlm/IRC_NAS/1ff1b38a-8218-4c53-9956-f0b264de35a4/l_HPCKit_p_2023.1.0.46346_offline.sh
Note that we use the specific version of the toolkit as an example. You can install any version you want.
- Launch the installer
sh ./l_HPCKit_p_2023.1.0.46346_offline.sh
During installation process you can choose a directory in which the toolkit should be installed (e.g., path/to/local/<toolkit>
).
- Source the
setvars.sh
(global to the toolkit) or thevars.sh
(local to the Intel MPI)
# source path/to/local/<toolkit>/oneapi/setvars.sh
source path/to/local/<toolkit>/oneapi/mpi/latest/env/vars.sh
- Install
unidist
and the required dependencies for the MPI backend
pip install unidist[mpi]
Now you can use unidist on MPI backend using Intel MPI implementaion.
- Remove the installer (optional):
rm l_HPCKit_p_2023.1.0.46346_offline.sh