TTK-MPI-at-example

This github repository contains the exact code used to reproduce the AT example in the reference below.

This repository is a standalone example created to reproduce the results of the reference below. It is not maintained. For a maintained version of this example, please see the TTK documentation.

Reference

If you plan to use this code to generate results for a scientific document, thanks for referencing the following publication:

"TTK is Getting MPI-Ready" Eve Le Guillou, Michael Will, Pierre Guillou, Jonas Lukasczyk, Pierre Fortin, Chistoph Garth, Julien Tierny

Installation Notes

Tested on Ubuntu 22.04.2 LTS.

Install the dependencies

To install the dependencies, run the following commands:

sudo apt update
sudo apt install g++ git cmake-qt-gui libboost-system-dev libopengl-dev qttools5-dev libqt5x11extras5-dev libqt5svg5-dev qtxmlpatterns5-dev-tools python3-dev libopenmpi-dev

Install Paraview

In the following command, replace the 4 in make -j4 install by the number of cores available.

cd ~
git clone https://github.com/topology-tool-kit/ttk-paraview.git
cd ttk-paraview
git checkout 5.11.0
mkdir build && cd build
cmake -DCMAKE_BUILD_TYPE=Release -DPARAVIEW_USE_PYTHON=ON -DPARAVIEW_USE_MPI=ON -DPARAVIEW_INSTALL_DEVELOPMENT_FILES=ON -DCMAKE_INSTALL_PREFIX=../install ..
make -j4 install

Please note that ParaView will use an X server by default to produce the image below. If there are no X server on your machine and that you wish to produce the image, please refer to this ParaView documentation to build the software using off-screen rendering.

Install TTK using this repository

We will now install TTK using the repository on Github. Again, replace the 4 in make -j4 install by the number of cores available.

cd ~
git clone https://github.com/topology-tool-kit/ttk.git
cd ttk
git checkout c701cd60b5432d5efe1e63442c3db2998ff383d8
mkdir build && cd build
PARAVIEW_PATH=~/ttk-paraview/install/lib/cmake/paraview-5.11
cmake -DParaView_DIR=$PARAVIEW_PATH -DTTK_ENABLE_MPI=ON -DTTK_ENABLE_MPI_TIME=ON 
-DTTK_ENABLE_64BITS_IDS=ON -DCMAKE_INSTALL_PREFIX=../install .. 
make -j4 install

Update environment variables

TTK is now installed, but needs an update of the environment variables to be called easily in the command line.

export PATH=$PATH:~/ttk-paraview/install/bin/
TTK_PREFIX=~/ttk/install
export PV_PLUGIN_PATH=$TTK_PREFIX/bin/plugins/TopologyToolKit
export LD_LIBRARY_PATH=$TTK_PREFIX/lib:$LD_LIBRARY_PATH
export PYTHONPATH=$TTK_PREFIX/lib/python3.10/site-packages

Run the example

First, retrieve this repository.

cd ~ 
git clone git@github.com:eve-le-guillou/TTK-MPI-at-example.git
cd TTK-MPI-at-example/

By default, the example is resampled to $256^3$. To execute it using 2 threads and 4 processes, use the following command:

OMP_NUM_THREADS=2 mpirun -n 4 pvbatch pipeline.py

Regarding performance, the optimal configuration depends on the architecture and on the OpenMP and OpenMPI versions. The performance tests presented in the paper have been performed on 24-core nodes (without SMT -- simultaneous multithreading) using 64 nodes:

OMPI_MPI_THREAD_LEVEL=1 OMP_PLACES=cores OMP_PROC_BIND=close OMP_NUM_THREADS=24 mpirun --bind-to none --map-by node --mca btl_openib_btls_per_lid 8 -n 64 pvbatch pipeline.py

If you want to resample to a higher dimension, for example $2048^3$ as in the reference paper, it can simply be done by executing the following command:

OMP_NUM_THREADS=2 mpirun -n 4 pvbatch pipeline.py 2048

Be aware that this will require a lot of memory to execute and will most likely not be possible on a regular laptop.

The command will create the following image and output the execution time of data loading and resampling, the pipeline and the creation of the output image. Depending on the dimension of the resample for the execution, the result may contain some resampling artifacts.