If you are building on a cluster, first check if there are specialized scripts available for your cluster at nv-legate/quickstart. Even if your specific cluster is not covered, you may be able to adapt an existing workflow.
The primary method of retrieving dependencies for Legate Core and downstream libraries is through conda. You will need an installation of conda to follow the instructions below.
Please use the scripts/generate-conda-envs.py script to create a conda
environment file listing all the packages that are required to build, run and
test Legate Core and all downstream libraries. For example:
$ ./scripts/generate-conda-envs.py --python 3.10 --ctk 12.2.2 --os linux --ucx
--- generating: environment-test-linux-py310-cuda-12.2.2-ucx.yamlRun this script with -h to see all available configuration options for the
generated environment file (e.g. all the supported Python versions). See the
Dependencies section for more details.
Once you have this environment file, you can install the required packages by creating a new conda environment:
conda env create -n legate -f <env-file>.yamlor by updating an existing environment:
conda env update -f <env-file>.yamlThe Legate Core repository comes with a helper install.py script in the
top-level directory, that will build the C++ parts of the library and install
the C++ and Python components under the currently active Python environment.
To add GPU support, use the --cuda flag:
./install.py --cudaYou can specify the CUDA toolkit directory and the CUDA architecture you want to
target using the --with-cuda and --arch flags, e.g.:
./install.py --cuda --with-cuda /usr/local/cuda/ --arch ampereBy default the script relies on CMake's auto-detection for these settings.
CMake will first search the currently active Python/conda environment
for dependencies, then any common system-wide installation directories (e.g.
/usr/lib). If a dependency cannot be found but is publicly available in source
form (e.g. OpenBLAS), cmake will fetch and build it automatically. You can
override this search by providing an install location for any dependency
explicitly, using a --with-<dep> flag, e.g. --with-nccl and
--with-openblas.
For multi-node execution Legate can use GASNet (use
--network gasnet1 or --network gasnetex, see below for
more details) or UCX (use --network ucx, see
below for more details).
Compiling with networking support requires MPI.
For example this would be an installation for a DGX SuperPOD:
./install.py --network gasnet1 --conduit ibv --cuda --arch ampereAlternatively, here is an install line for the Piz-Daint supercomputer:
./install.py --network gasnet1 --conduit aries --cuda --arch pascalTo see all available configuration options, run with the --help flag:
./install.py --helpThe following table lists Legate's minimum supported versions of major dependencies.
"Full support" means that the corresponding versions (and all later ones) are being tested with some regularity, and are expected to work. Please report any incompatibility you find against a fully-supported version by opening a bug.
"Best-effort support" means that the corresponding versions are not actively tested, but Legate should be compatible with them. We will not actively work to fix any incompatibilities discovered under these versions, but we accept contributions that fix such incompatibilities.
| Dependency | Full support (min version) | Best-effort support (min version) |
|---|---|---|
| CPU architecture | x86-64 (Haswell), aarch64 | ppc64le, older x86-64, Apple Silicon |
| OS | RHEL 8, Ubuntu 20.04, MacOS 12 | other Linux |
| C++ compiler | gcc 8, clang 7, nvc++ 19.1 | any compiler with C++17 support |
| GPU architecture | Volta | Pascal |
| CUDA toolkit | 11.4 | 10.0 |
| Python | 3.9 | |
| NumPy | 1.22 |
In this section we comment further on our major dependencies. Please consult an
environment file created by generate-conda-envs.py for a full listing of
dependencies, e.g. building and testing tools, and for exact version
requirements.
Legate has been tested on Linux and MacOS, although only a few flavors of Linux such as Ubuntu have been thoroughly tested. There is currently no support for Windows.
Specify your OS when creating a conda environment file through the --os flag
of generate-conda-envs.py.
In terms of Python compatibility, Legate roughly follows the timeline outlined in NEP 29.
Specify your desired Python version when creating a conda environment file
through the --python flag of generate-conda-envs.py.
We suggest that you avoid using the compiler packages available on conda-forge. These compilers are configured with the specific goal of building redistributable conda packages (e.g. they explicitly avoid linking to system directories), which tends to cause issues for development builds. Instead prefer the compilers available from your distribution's package manager (e.g. apt/yum) or your HPC vendor.
If you want to pull the compilers from conda, use an environment file created
by generate-conda-envs.py using the --compilers flag. An appropriate
compiler for the target OS will be chosen automatically.
Only necessary if you wish to run with Nvidia GPUs.
Some CUDA components necessary for building, e.g. the nvcc compiler and driver
stubs, are not distributed through conda. These must instead be installed using
system-level packages. If these
are not installed under a standard system location, you will need to inform
install.py of their location using --with-cuda.
If you intend to pull any CUDA libraries from conda (see below), conda will need
to install an environment-local copy of the CUDA toolkit, even if you have it
installed system-wide. To avoid versioning conflicts it is safest to match the
version of CUDA installed system-wide, by specifying it to
generate-conda-envs.py through the --ctk flag.
Legate is tested and guaranteed to be compatible with Volta and later GPU architectures. You can use Legate with Pascal GPUs as well, but there could be issues due to lack of independent thread scheduling. Please report any such issues on GitHub.
Only necessary if you wish to run with Nvidia GPUs.
The following additional CUDA libraries are required, for use by legate.core or downstream libraries. Unless noted otherwise, these are included in the conda environment file.
cublascufftcurand(can optionally be used for its host fallback implementations even when building without CUDA support)cusolvercutensorncclnvmlnvtxthrust(pulled from github)
If you wish to provide alternative installations for these, then you can remove
them from the environment file (or invoke generate-conda-envs.py with --ctk none, which will skip them all), and pass the corresponding --with-<dep> flag
to install.py (or let the build process attempt to locate them automatically).
Used by cuNumeric for implementing linear algebra routines on CPUs.
This library is automatically pulled from conda. If you wish to provide an
alternative installation, then you can manually remove openblas from the
generated environment file and pass --with-openblas to cuNumeric's
install.py.
Note that if you want to build OpenBLAS from source you will need to get a
Fortran compiler, e.g. by pulling fortran-compiler from conda-forge.
If you wish to compile Legate with OpenMP support, then you need a build of OpenBLAS configured with the following options:
USE_THREAD=1USE_OPENMP=1NUM_PARALLEL=32(or at least as many as the NUMA domains on the target machine) -- TheNUM_PARALLELflag defines how many instances of OpenBLAS's calculation API can run in parallel. Legate will typically instantiate a separate OpenMP group per NUMA domain, and each group can launch independent BLAS work. IfNUM_PARALLELis not high enough, some of this parallel work will be serialized.
Used by cuNumeric for implementing tensor contraction routines on CPUs.
This library will be automatically downloaded and built during cuNumeric
installation. If you wish to provide an alternative installation, pass
--with-tblis to cuNumeric's install.py.
cuNumeric requires a build of TBLIS configured as follows:
--with-label-type=int32_t --with-length-type=int64_t --with-stride-type=int64_t
and additionally --enable-thread-model=openmp if cuNumeric is compiled
with OpenMP support.
Required to support CPU and memory binding in the Legate launcher.
Not available on conda; typically available through the system-level package manager.
Only necessary if you wish to run on multiple nodes.
We suggest that you avoid using the generic build of OpenMPI available on conda-forge. Instead prefer an MPI installation provided by your HPC vendor, or from system-wide distribution channels like apt/yum and MOFED, since these will likely be more compatible with (and tuned for) your particular system.
If you want to use the OpenMPI distributed on conda-forge, use an environment
file created by generate-conda-envs.py using the --openmpi flag.
Legate requires a build of MPI that supports MPI_THREAD_MULTIPLE.
Only necessary if you wish to run on multiple nodes, using the corresponding networking hardware.
Not available on conda; typically available through MOFED or the system-level package manager.
Depending on your hardware, you may need to use a particular Realm networking backend, e.g. as of October 2023 HPE Slingshot is only compatible with GASNet.
Only necessary if you wish to run on multiple nodes, using the GASNet1 or GASNetEx Realm networking backend.
This library will be automatically downloaded and built during Legate
installation. If you wish to provide an alternative installation, pass
--with-gasnet to install.py.
When using GASNet, you also need to specify the interconnect network of the
target machine using the --conduit flag.
Only necessary if you wish to run on multiple nodes, using the UCX Realm networking backend.
You can use the version of UCX available on conda-forge by using an environment
file created by generate-conda-envs.py using the --ucx flag. Note that this
build of UCX might not include support for the particular networking hardware on
your machine (or may not be optimally tuned for such). In that case you may want
to use an environment file generated with --no-ucx (default), get UCX from
another source (e.g. MOFED, the system-level package manager, or compiled
manually from source), and pass the location
of your UCX installation to install.py (if necessary) using --with-ucx.
Legate requires a build of UCX configured with --enable-mt.
If you do not wish to use conda for some (or all) of the dependencies, you can
remove the corresponding entries from the environment file before passing it to
conda. See the install.py section for
instructions on how to provide alternative locations for these dependencies to
the build process.
Note that this is likely to result in conflicts between conda-provided and system-provided libraries.
Conda distributes its own version of certain common libraries (in particular the
C++ standard library), which are also typically available system-wide. Any
system package you include will typically link to the system version, while
conda packages link to the conda version. Often these two different versions,
although incompatible, carry the same version number (SONAME), and are
therefore indistinguishable to the dynamic linker. Then, the first component to
specify a link location for this library will cause it to be loaded from there,
and any subsequent link requests for the same library, even if suggesting a
different link location, will get served using the previously linked version.
This can cause link failures at runtime, e.g. when a system-level library happens to be the first to load GLIBC, causing any conda library that comes after to trip GLIBC's internal version checks, since the conda library expects to find symbols with more recent version numbers than what is available on the system-wide GLIBC:
/lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.30' not found (required by /opt/conda/envs/legate/lib/libarrow.so)
You can usually work around this issue by putting the conda library directory first in the dynamic library resolution path:
LD_LIBRARY_PATH="$CONDA_PREFIX/lib:$LD_LIBRARY_PATH"This way you can make sure that the (typically more recent) conda version of any common library will be preferred over the system-wide one, no matter which component requests it first.
Legate Core is not yet registered in a standard pip repository. However, users can still use the pip installer to build and install Legate Core. The following command will trigger a single-node, CPU-only build of Legate Core, then install it into the currently active Python environment:
$ pip install .or
$ python3 -m pip install .Legate relies on CMake to select its toolchain and build flags. Users can set
the environment variables CXX or CXXFLAGS prior to building to override the
CMake defaults.
Alternatively, CMake and build tool arguments can be passed via the
CMAKE_ARGS/SKBUILD_CONFIGURE_OPTIONS and SKBUILD_BUILD_OPTIONS
environment variables:
$ CMAKE_ARGS="${CMAKE_ARGS:-} -D Legion_USE_CUDA:BOOL=ON" \
pip install .An alternative syntax using setup.py with scikit-build is
$ python setup.py install -- -DLegion_USE_CUDA:BOOL=ONpip uses scikit-build
in setup.py to drive the build and installation. A pip install will trigger three general actions:
- CMake build and installation of C++ libraries
- CMake generation of configuration files and build-dependent Python files
- pip installation of Python files
The CMake build can be configured independently of pip, allowing incremental C++ builds directly through CMake.
This simplifies rebuilding the C++ shared libraries either via command-line or via IDE.
After building the C++ libraries, the pip install can be done in "editable" mode using the -e flag.
This configures the Python site packages to import the Python source tree directly.
The Python source can then be edited and used directly for testing without requiring another pip install.
There are several examples in the scripts folder. We walk through the steps in
build-separately-no-install.sh here.
First, the CMake build needs to be configured:
$ cmake -S . -B build -GNinja -D Legion_USE_CUDA=ONOnce configured, we can build the C++ libraries:
$ cmake --build buildThis will invoke Ninja (or make) to execute the build. Once the C++ libraries are available, we can do an editable (development) pip installation.
$ SKBUILD_BUILD_OPTIONS="-D FIND_LEGATE_CORE_CPP=ON -D legate_core_ROOT=$(pwd)/build" \
python3 -m pip install \
--root / --no-deps --no-build-isolation
--editable .The Python source tree and CMake build tree are now available with the environment Python for running Legate programs. The diagram below illustrates the complete workflow for building both Legate core and a downstream package, cuNumeric
