UCX depends on the following system libraries being present:
- For MOFED 4.x support:
libibcm
,libibverbs
andlibrdmacm
. Ideally installed from Mellanox OFED Drivers - For MOFED 5.x support: Mellanox OFED Drivers
- For system topology identification:
libnuma
(numactl
on Enterprise Linux)
Please install the packages above with your Linux system's package manager.
When building from source you will also need the *-dev
(*-devel
on
Enterprise Linux) packages as well.
Some preliminary Conda packages can be installed as so. Replace
<CUDA version>
with either 11.0
or 11.2
. These are
available both on rapidsai
and rapidsai-nightly
.
With GPU support:
conda create -n ucx -c conda-forge -c rapidsai \ cudatoolkit=<CUDA version> ucx-proc=*=gpu ucx ucx-py python=3.7
Without GPU support:
conda create -n ucx -c conda-forge -c rapidsai \ ucx-proc=*=cpu ucx ucx-py python=3.7
The following instructions assume you'll be using ucx-py on a CUDA enabled system and is in a Conda environment.
Note
As of version 0.15, the UCX conda package build will no longer include IB/RDMA support. This is largely due to compatibility issues between OFED versions. We do however provide instructions below for how to build UCX with IB/RDMA support in the UCX + OFED section.
conda create -n ucx -c conda-forge \ automake make libtool pkg-config \ psutil \ "python=3.7" setuptools "cython>=0.29.14,<3.0.0a0"
conda install -n ucx -c rapidsai -c nvidia -c conda-forge \ pytest pytest-asyncio \ cupy "numba>=0.46" rmm \ distributed
Instructions for building UCX >= 1.11.1 (minimum version supported by UCX-Py), make sure to change git checkout v1.11.1 to a newer version if desired:
conda activate ucx git clone https://github.com/openucx/ucx cd ucx git checkout v1.11.1 ./autogen.sh mkdir build cd build # Performance build ../contrib/configure-release --prefix=$CONDA_PREFIX --with-cuda=$CUDA_HOME --enable-mt # Debug build ../contrib/configure-devel --prefix=$CONDA_PREFIX --with-cuda=$CUDA_HOME --enable-mt make -j install
As noted above, the UCX conda package no longer builds support for IB/RDMA. To build UCX with IB/RDMA support first confirm OFED is installed properly:
(ucx) user@dgx:~$ ofed_info -s OFED-internal-4.7-3.2.9
If OFED drivers are not installed on the machine, you can download drivers at directly from Mellanox. For versions older than 5.1 click on, archive versions.
Building UCX >= 1.11.1 as shown previously should automatically include IB/RDMA support if available in the system. It is possible to explicitly activate those, ensuring the system satisfies all dependencies or fail otherwise, by including the --with-rdmacm
and --with-verbs
build flags. For example:
../contrib/configure-release \ --enable-mt \ --prefix="$CONDA_PREFIX" \ --with-cuda="$CUDA_HOME" \ --enable-mt \ --with-rdmacm \ --with-verbs
conda activate ucx git clone https://github.com/rapidsai/ucx-py.git cd ucx-py pip install -v . # or for develop build pip install -v -e .