pertool - Perturbo Multi-Tool

The pertool program provides multiple helpful operations for managing Perturbo input and data files when conducting large simulations and other complex tests.

This program solves two main problems:

Manipulation of eph_g2 pool data files

To support scaling the calculations, Perturbo is able to generate distinct pools of k-points and their corresponding q-points and other values, one pool per MPI task. The problem is, once these files have been generated for a given number of pools (i.e. MPI tasks), the files cannot be reused for a different number of pools. This can be very frustrating when trying to find the right number of MPI tasks for a given simulation. You generate the data files, and then you try running Perturbo - but it crashes. So you try generating a different number of data files. You keep doing this until you find a pool size that seems to work well enough.

No longer! The pertool program can take an existing set of pool data files and regenerate them for a different number of pools. This tends to run much faster than computing them from scratch.

Generation of experiments from a template

Coming soon...

Overview of Commands

• The analyze command reports details of the eph_g2 pool data files generated by Perturbo into the ./tmp directory.

• The reshape command allows the eph_g2 pool data files to be "reshaped" to a different number of pools.

Installing pertool

Currently the installation process is very basic. Clone the repository, and run:

pip install -r requirements.txt

The pertool program has a pretty small number of dependencies:

• h5py for HDF5 access from Python (which has numpy as a dependency)
• progressbar2 for providing a helpful progress bar at the command prompt

Running pertool

To run preshape to regenerate pool data files, run it like this:

python -m pertool reshape \
        -f <path to source tmp directory> \
        -t <path to target tmp directory> \
        -p <number of pools to generate>
        [--mp]

The source tmp directory is scanned, and preshape will determine the number of pools in the source directory automatically.

If the target tmp directory is not empty, the program will report an error.

The --mp flag can be used to enable multi-process parallelism, which often yields a substantial performance improvement.

Parallel Analyze/Reshape Operations

The HDF5 library is not designed to be used in highly concurrent settings. However, a certain amount of performance improvement is possible through the use of multi-process programming and the multiprocessing Python standard library. The pertool program supports using multiprocessing with the --mp flag, which parallelizes analysis/reshape operation in these ways:

• The source HDF5 files are scanned in parallel, with one subprocess being started for each source file. This yields significant performance improvement in the typical case, as the source files are completely independent. Since the scan operation will typically be IO-bound, having multiple concurrent scans will be a big win.

• The target HDF5 files are generated in parallel, with one subprocess generating each target file. This also yields significant performance improvement, but each subprocess must access all source HDF5 files, and the HDF5 library may use file locking to ensure exclusive access, even in read-only cases. (See above link for details.)

You should test your operations to see if the multi-process code will be faster than serial code, but in the limited tests done on NERSC Perlmutter, an order of magnitude performance improvement is typical.

NOTE: For large simulations with many pool files, running a reshape operation that involves spawning many subprocesses will massively degrade the machine's performance while the operation is running. Therefore, pertool also includes a --max-processes argument that limits the maximum number of subprocesses spawned. The default value is 20, but it can be changed as appropriate.