Hydra Slurm example

This is a collection of examples of how to use Hydra to launch jobs locally and on a Slurm cluster. For more background, read the Hydra documentation.

Local setup

Install conda and create a new environment:

conda env update -f env.yaml --prune

add .local/bin to your PATH

export PATH=$PATH:$HOME/.local/bin

HPC setup

Depending on the infrastructure and cluster, you maybe need to change some modules

module swap cluster/donphan
source modules.sh

You can always reset this setup using module purge.

Examples

• src/sleep_pbs/README.md is an example used to explain interactive and job-based scheduling with PBS and SLURM. The example sleep script is benchmarked for runtime and memory usage with timeit and memray.
• src/sleep_hydra/README.md is the same sleep example and benchmarking, but executed with the Hydra framework. More powerful and flexible, but also more complex.
• src/dask_jobqueue/README.md is an example of how to use Hydra and submitit to launch a Dask jobqueue through the SLURM scheduler.
• src/frequencies_hydra/README.md is the counting frequencies example with benchmarking. It uses a Python-only configuration, based on hydra-zen.

Common patterns

There are various usage patterns in Hydra to make your life easier. For more information, see the Hydra documentation on common patterns.

Possible improvements

• Allow benchmarking using platform instrumentation and diagnostics (Slurm, Dask, Prefect...)
• Optuna sweeper
• Usage with Prefect

References

• hydra_submitit_launcher