This project develops an MPI-based parallel Preconditioned Conjugate Gradient (PCG) solver. The solver has been tested on the Gadi supercomputer using approximately 12,000 cores for solving sparse linear systems with over 1 billion unknowns (dofs).
The PCG is an algorithm for the numerical solution of systems of linear equations (Ax=b) with positive-semidefinite system matrix (A). The method can also be used to solve unconstrained optimization problems such as energy minimization.
The Message Passing Interface (MPI) is a standardized and portable message-passing system that enables the development of efficient parallel codes on distributed computing architectures.
In this project, the PCG-MPI solver has been applied to solve a fundamental problem in computational solid mechanics - Linear Elastostatics - showcasing its robust capabilities in practical applications.
The following steps outline how to install the necessary libraries and dependencies on the Gadi supercomputer at NCI, Canberra:
cd ~
module load python3/3.11.0
module load openmpi/4.0.5
python3 -m venv env_MPI --system-site-package
source ~/env_MPI/bin/activate
python3 -m pip install mpi4py mgmetis jupyter
To run the code on a local workstation, ensure you have the following installed:
- Python 3
- Open-MPI
- Python libraries: numpy, scipy, matplotlib, Cython, mpi4py, mgmetis, jupyter
You can install the required Python libraries using pip:
pip install numpy scipy matplotlib Cython mpi4py mgmetis jupyter
- Save the repository "PCG_MPI_Solver" in your home directory and set it as the working directory:
cd ~/PCG_MPI_Solver
- Execute the bash file to set up the environment and test run the solver:
bash examples/run_basic_script.bash
- Alternatively, setup the environment as follows and run the jupyter notebook.
module load python3/3.11.0
source ~/env_MPI/bin/activate
export PYTHONPATH=~/env_MPI/lib/python3.11/site-packages:$PYTHONPATH
jupyter nbconvert --execute --clear-output notebooks/solver_demo.ipynb
A working demonstration of this code for an example problem is provided in the jupyter notebook. Refer to notebooks/solver_demo.ipynb
Below is the folder structure of the project, outlining the main directories and their contents:
PCG-MPI-solver/
├── .gitignore
├── LICENSE # Contains the licensing agreement for the project
├── README.md # Provides a detailed description of the project
├── data/
│ └── concrete.zip # Model data used by the project for demonstration
├── docs/
│ └── references.txt # References related to the project
├── examples/
│ └── run_basic_script.bash # Bash script for running a basic example of the solver
├── notebooks/
│ ├── setup_env.sh # Shell script for setting up the environment
│ ├── solver_demo.ipynb # Jupyter notebook demonstrating the solver's usage
│ └── images/ # Directory containing images used in notebooks
├── scripts/
│ └── install.bash # Script to automate the installation of dependencies
└── src/
├── solver/ # Core directory for the solver's algorithms
│ ├── partition_mesh.py # Script for mesh partitioning
│ ├── pcg_solver.py # Script for the preconditioned conjugate gradient solver
│ └── run_metis.py # Script to run METIS for mesh partitioning
├── utils/ # Utility scripts for the project
│ └── file_operations.py # Script to handle file operations
└── data/
├── export_vtk.py # Script to export data in VTK format
├── read_input_model.py # Script to read the input model from files
└── evtk/ # EVTK library to convert data in VTK format
- For detailed information on the numerical techniques and solver algorithms employed in this project, refer to the article:
- Title: "An octree pattern-based massively parallel PCG solver for elasto-static and dynamic problems"
- Link: Article - Computer Methods in Applied Mechanics and Engineering
- Additional details can be found in the PhD thesis:
- Title: "High-Performance Computing for Impact-Induced Fracture Analysis exploiting Octree Mesh Patterns"
- Link: PhD Thesis - UNSW Sydney
If you use this code in a scientific publication, please cite the following paper:
Ankit A, Zhang J, Eisenträger S, Song C. An octree pattern-based massively parallel PCG solver for elasto-static and dynamic problems. Computer Methods in Applied Mechanics and Engineering. 2023 Feb 1;404:115779.