Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add changes to libtorch installation instructions. #27689

Open
wants to merge 5 commits into
base: next
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
106 changes: 88 additions & 18 deletions modules/doc/content/getting_started/installation/install_libtorch.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,11 @@ The way one enables LibTorch [!cite](paszke2019pytorch) capabilities in MOOSE de
the operating system (Linux or Mac) and if we use HPC or just a local workstation.

!alert! note
Before we review the main approaches, it is important to emphasize that
Before trying to install MOOSE with libtorch make sure to check out the
[compatibility matrix](https://github.com/pytorch/pytorch/blob/main/RELEASE.md#release-compatibility-matrix)
which will tell you if the required packages are compatible with the ones required by MOOSE.

Furthermore, it is important to emphasize that
linking MOOSE with libtorch on +Linux machines+ is not supported if the compiler stack has been built
using a `libc` version below 2.27 (for `libtorch v 1.8+`)
or 2.23 (for `libtorch v1.4-1.8`). Furthermore, we do not support `libtorch` versions below
Expand All @@ -16,22 +20,37 @@ ldd --version

!alert-end!

## Setting up the environment
## Setup Environment

For both Mac and Linux workstations, the user needs to follow the
[Conda MOOSE Environment](installation/conda.md) instructions.

## Install Libtorch

For Mac workstations, the user needs to create a conda environment using the
instructions [here](installation/conda.md). On Linux machines, however,
we cannot use the conda packages due to the mismatch between `libc` versions.
For this reason, given that the system `libc` version allows the linking between
the two libraries, we need to install `PETSc` and `libMesh` manually. For instructions
on the installation of these, see [installation/hpc_install_moose.md].
The user can choose from three alternatives when it comes to installing `libtorch`:

In situations when the `libc` version allows the linking but the compiler stack
has been compiled with an older `libc` version (HPC machines potentially), we need to build the
compiler from scratch. For instructions in this process, visit [installation/manual_installation_gcc.md]
- +Install using conda:+

## Installing Libtorch
For ARM and Intel Mac workstations the user can install libtorch using conda within the `moose` environment:

```bash
conda install pytorch
```

The user can choose from two alternatives when it comes to installing `libtorch`:
This will provide the headers and libraries needed to use libtorch.

!alert! note

The same process works for Linux workstations if the user requires CPU support only.
To ensure this, add the `cpuonly` package to the installation line:

```bash
conda install pytorch cpuonly -c pytorch
```

For notes on the GPU support, visit the [notes below](install_libtorch.md#gpu-notes).

!alert-end!

- +Install using the script provided in MOOSE:+

Expand All @@ -41,19 +60,27 @@ The user can choose from two alternatives when it comes to installing `libtorch`
./scripts/setup_libtorch.sh
```

which downloads `libtorch` from the official site and sets it up in the `framework\contrib`
directory. The script checks for operating system and `libc` version and throws errors
if the system is not suitable for the coupling.
which downloads `libtorch` from the official site and sets it up in the `framework/contrib`
directory.

!alert! note

The script checks for operating system and `libc` version (on Linux workstations)
and throws errors if the system is not suitable for the coupling. If there is a mismatch
between the detected `GLIBC` version and the one used to compile libtorch, the user needs
to install `libMesh` and `PETSc` manually with a compatible compiler.
!alert-end!

!alert! note

The desired version of libtorch can be set by the following argument:

```bash
grmnptr marked this conversation as resolved.
Show resolved Hide resolved
./scripts/setup_libtorch.sh --version=1.8
```

Note that we do not support `libtorch` below a version of 1.4. The default
version downloaded by the script is 1.10.
version downloaded by the script is 2.1.

!alert-end!

Expand All @@ -62,9 +89,10 @@ The user can choose from two alternatives when it comes to installing `libtorch`
[official website](https://github.com/pytorch/pytorch/blob/master/docs/libtorch.rst).


## Configure and compile MOOSE with libtorch
## Configure MOOSE

To achieve this, first configure MOOSE with `libtorch` support (along with any other desired configure options)
from within the `moose` folder:

```bash
./configure --with-libtorch
Expand All @@ -85,6 +113,29 @@ by the `setup_libtorch.sh` script.

!alert-end!

For conda-based installations the user can link to the conda-based libtorch libraries
using the approach above (using a typical installation path within conda):


```bash
./configure --with-libtorch=${CONDA_PREFIX}/lib/python3.10/site-packages/torch
```
grmnptr marked this conversation as resolved.
Show resolved Hide resolved


!alert! note

The python version can be different depending on the distribution, so make sure you double-check if
the directory you point to actually exists!
An easy way to find if the library exists within the conda package is running the following command in the terminal:

```bash
find ${CONDA_PREFIX} -type d -name torch
```

!alert-end!

## Build MOOSE

The last step is to compile MOOSE with libtorch support:

```bash
Expand All @@ -100,3 +151,22 @@ The makefile of the [Stochastic Tools Module](stochastic_tools/stochastic_tools.
serves as a good example on how to achieve this for applications.

!alert-end!

## GPU Support id=gpu-notes

When using ARM Macs the conda-based installation supports both CPU and GPU devices.
For GPU acceleration through Metal, users need to select MPS as a device when writing source code.

!alert! warning

GPU devices on Linux machines are not +officially+ supported yet.

However, on Linux machines with existing CUDA and CUDA-enabled compiler support, one can
compile MOOSE with a GPU-based Libtorch library without using the MOOSE conda environment.
This means that the dependencies (PETSc, libMesh, WASP) need to be compiled
manually. Following this, one can download an officially supported, CUDA-based
precompiled library from the Libtorch website and supply that to the configure script
before compiling MOOSE. If you have questions regarding this path,
feel free to ask them on out Discussion forum!

!alert-end!
60 changes: 48 additions & 12 deletions scripts/setup_libtorch.sh
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
#* Licensed under LGPL 2.1, please see LICENSE for details
#* https://www.gnu.org/licenses/lgpl-2.1.html

# For now, 2.1.0 is the default version of torchlib
# For now, 2.1.0 is the default version of libtorch
VERSION=2.1.0
IGNORE_CERT=""

Expand Down Expand Up @@ -40,7 +40,7 @@ done

# Show how to use the script
if [[ -n "$HELP" ]]; then
echo "Usage: $0 [-h | --help | -k | --version=VERSION_NUMBER ]"
echo "Usage: $0 [-h | --help | -k | --version=VERSION_NUMBER | --cleanup ]"
echo
echo "-h | --help Display this message and list of available setup options"
echo "-k Ignore certifications while downloading packages with curl"
Expand All @@ -57,6 +57,12 @@ TORCH_DIR=$TORCH_DEST/libtorch
# We need to do this because 1.12 is lower than 1.9 if parsed as a single number
MAINVERSION=$(echo $VERSION | cut -d. -f1)
SUBVERSION=$(echo $VERSION | cut -d. -f2)
REVISION=$(echo $VERSION | cut -d. -f3)

if [[ -z "${REVISION}" ]]; then
REVISION="0"
VERSION=$VERSION.0
fi

if (( $MAINVERSION < 1 || ( $MAINVERSION == 1 && $SUBVERSION < 4 ) )); then
echo "The current implementation does not support libtorch versions below 1.4!"
Expand All @@ -68,25 +74,55 @@ if [ -d $TORCH_DIR ]; then
rm -rf $TORCH_DIR
fi

# Checking the operating system
UNAME_OUT="$(uname -s)"
case "${UNAME_OUT}" in
# Checking the operating system and architecture type
UNAME_SYS="$(uname -s)"
UNAME_ARCH="-$(uname -m)"
case "${UNAME_SYS}" in
Linux*) OP_SYS=linux;;
Darwin*) OP_SYS=mac;;
esac

# We do some sanity checking on the version number. They started distributing
# precompiled libraries for ARM machines afer version 2.2. Before that, it was x86 without the
# tag
if (( $OP_SYS == mac )); then
if (( $MAINVERSION < 2 || ( $MAINVERSION == 2 && $SUBVERSION < 2 ) )); then
if (( UNAME_ARCH == "-arm64" )); then
echo "Precompiled libraries below version 2.2 are not available for ARM architecture!"
exit 1
else
UNAME_ARCH=""
fi
fi
else
UNAME_ARCH="" # We don't need this for linux machines
fi

# Checking if the available GLIBC version is sufficient for proper linkig. Only
# causes issues on linux distributions. Considering that most Macs use the
# moose compiler stack.
function version_greater {
if { echo "$1"; echo "$2"; } | sort --version-sort --check=quiet; then
false
else
true
fi
}
function error_message {
echo "ERROR! The current version of GLIBC is not sufficient for proper linking!"
echo "Upgrade it to at least $1! Current version: $2"
}

if [[ $OP_SYS == linux ]]; then
GLIBC_VERSION=`ldd --version | awk '/ldd/{print $NF}'`
if (( $SUBVERSION < 8 && $(echo "$GLIBC_VERSION < 2.23" | bc -l) )); then
echo "ERROR! The current version of GLIBC is not sufficient for proper linking!"
echo "Upgrade it to at least 2.23! Current version: $GLIBC_VERSION"
if not version_greater $VERSION 1.8 && (( $(echo "$GLIBC_VERSION < 2.23" | bc -l) )); then
error_message 2.23 $GLIBC_VERSION
exit 1
elif not version_greater $VERSION 2.1 && (( $(echo "$GLIBC_VERSION < 2.27" | bc -l) )); then
error_message 2.27 $GLIBC_VERSION
exit 1
elif (( $SUBVERSION > 8 && $(echo "$GLIBC_VERSION < 2.27" | bc -l) )); then
echo "ERROR! The current version of GLIBC is not sufficient for proper linking!"
echo "Upgrade it to at least 2.27! Current version: $GLIBC_VERSION"
elif version_greater $VERSION 2.1 && (( $(echo "$GLIBC_VERSION < 2.29" | bc -l) )); then
error_message 2.29 $GLIBC_VERSION
exit 1
fi
fi
Expand All @@ -97,7 +133,7 @@ FILENAME=""
if [[ $OP_SYS == linux ]]; then
FILENAME=libtorch-cxx11-abi-shared-with-deps-$VERSION%2B$TORCH_DISTRIBUTION.zip
elif [[ $OP_SYS == mac ]]; then
FILENAME=libtorch-macos-$VERSION.zip
FILENAME=libtorch-macos$UNAME_ARCH-$VERSION.zip
else
echo "Unknown operating system! We only support Linux/Mac machines!"
exit 1
Expand Down