This module is a simple tool to automatically explore a parameter space.
You can install parspace with pip:
$ python3 -m pip install -U --user parspace
This package provides a class that can be used to automatically run a function over all possible combinations of a given parameter space.
Say that you have a problem controlled by two parameters: an aspect ratio
asp and a density density. The following example shows how to use
ParSpace to automatically explore every combination of the requested values
for these two parameters. If you have a function (here called launch_simu)
that performs the desired task for a given value of these two parameters, you
merely have to use ParSpace as a decorator on that function.
from parspace import ParSpace
@ParSpace(asp=[0.5, 1, 2],
density=[1, 10])
def launch_simu(asp, density):
print(f"aspect ratio {asp} and density {density}")
launch_simu()This code will print the following on screen:
aspect ratio 0.5 and density 1 aspect ratio 0.5 and density 10 aspect ratio 1 and density 1 aspect ratio 1 and density 10 aspect ratio 2 and density 1 aspect ratio 2 and density 10
In real use cases, launch_simu could e.g. perform the desired simulation or
submit a job on a supercomputer (see below for a realistic example of that).
The order of arguments of launch_simu does not matter, only their names.
In other words, if its signature is instead def launch_simu(density, asp),
you would obtain the same result. On the other hand, defining it as def
launch_simu(aspect, density) would result in a TypeError when calling the
decorated function as the asp argument fed to ParSpace does not match
any argument of launch_simu.
If you have a large number of parameters, you might prefer defining
launch_simu as taking a dictionary of keyword arguments. Parameter names
and values are then the keys and values of that dictionary.
from parspace import ParSpace
@ParSpace(asp=[0.5, 1, 2],
density=[1, 10])
def launch_simu(**pars):
asp = pars['asp']
density = pars['density']
print(f"aspect ratio {asp} and density {density}")
launch_simu()If you care about the returned values of the wrapped function rather than on its side effects, you can iterate over the decorated object as shown below.
from parspace import ParSpace
@ParSpace(asp=[0.5, 1, 2],
density=[1, 10])
def calc_mass(asp, density):
return asp * density
for pars, mass in calc_mass:
asp = pars['asp']
density = pars['density']
print(f"Mass for aspect {asp} at density {density}: {mass}")Note that iterating through a bare ParSpace instance yields the parameters
dictionary.
from parspace import ParSpace
space = ParSpace(asp=[0.5, 1, 2],
density=[1, 10])
for pars in space:
print("aspect ratio {asp} and density {density}".format(**pars))Provided that the iterables used to build the ParSpace instance can be
iterated through any number of times (mind that generators can only be iterated
through once), you can use that instance on several functions as follow.
from parspace import ParSpace
space = ParSpace(asp=[0.5, 1, 2],
density=[1, 10])
@space
def launch_simu(asp, density):
print(f"aspect ratio {asp} and density {density}")
launch_simu()
for pars, mass in space(lambda asp, density: asp * density):
asp = pars['asp']
density = pars['density']
print(f"Mass for aspect {asp} at density {density}: {mass}")This script shows a simplistic but realistic example of what ParSpace
enables you to do. This script is written for a particular system and is
therefore unlikely to work for you as-is but adapting it to your use case
should be a fairly simple task. The function submit_jobs defines what
should be done for one specific job and its decorated version automatically
explores the desired parameter space.
#!/usr/bin/env python3
"""Submit jobs on a PBS enabled cluster.
This script is for demonstration purposes only and offers no guarantee, please
adapt it to your use case.
"""
from functools import lru_cache
from pathlib import Path
import json
import subprocess
import textwrap
from parspace import ParSpace
QSUB = '/usr/local/bin/qsub'
BATCH = textwrap.dedent("""\
#!/bin/bash
#PBS -N jobname
#PBS -l nodes=1:ppn=8
#PBS -q queuename
#PBS -j oe
#PBS -V
cd {work_dir}
mpirun -np 8 /path/to/executable > out.log 2> err.log
sync
exit
""")
ROOT = Path().resolve(strict=True)
# If you need to compute an entry parameter that depends only on a subset of
# all the parameters you explore, you might want to cache its result if the
# computation is expensive. This isn't necessary in this simplistic case and
# is for illustrative purposes only.
@lru_cache(maxsize=None)
def n_horiz(aspect_ratio):
"""Compute grid size for a given aspect ratio."""
return 64 * aspect_ratio
@ParSpace(logra=range(4, 7),
aspect_ratio=[2, 4, 8])
def submit_jobs(**pars):
"""Create run directory, parameter file, and submit a job."""
case_name = 'ra_1e{logra}__asp_{aspect_ratio}'.format(**pars)
case_dir = ROOT / case_name
# Create run directory, here a subdirectory "output" is also created.
(case_dir / 'output').mkdir(parents=True, exist_ok=True)
# Generate par file, this example assumes a JSON parameter file.
asp = pars['aspect_ratio']
par_content = dict(rayleigh=10**pars['logra'],
aspect_ratio=pars['aspect_ratio'],
ny=n_horiz(asp))
par_file = case_dir / 'par.json'
with par_file.open('w') as pstream:
json.dump(par_content, pstream)
# Generate batch submission script. You can use the same approach as what
# is done for the working directory to control other job parameters (such
# as the number of cores) on a case-by-case basis.
batch_content = BATCH.format(work_dir=case_dir)
batch_file = case_dir / 'batch'
batch_file.write_text(batch_content)
# Call qsub and print the job number.
job_sub = subprocess.run((QSUB, str(batch_file)),
capture_output=True, check=True, text=True)
job_id = job_sub.stdout.splitlines()[-1].split('.')[0]
print(f"Case {case_name} treated by {job_id}")
if __name__ == "__main__":
submit_jobs()