Skip to content
A tool to easily orchestrate general computational workflows both locally and on supercomputers
Python Other
  1. Python 99.8%
  2. Other 0.2%
Branch: develop
Clone or download
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.github Introduce Contributing Guidelines (#62) Jan 2, 2018
assets Addition of a simple example and logo to the README. (#208) Dec 11, 2019
docs
maestrowf Ticked up version to 1.1.7dev1. Jan 22, 2020
samples Enhances pgen to have access to environment specified in a study. (#209) Dec 17, 2019
tests Release of v1.1.4 (#193) Jul 15, 2019
.gitignore Release of v1.1.4 (#193) Jul 15, 2019
.pylintrc Initial commit for public release. Jul 10, 2017
.travis.yml Drop support for Python 3.4 (#218) Jan 17, 2020
Dockerfile Release of v1.1.4 (#193) Jul 15, 2019
LICENSE Initial commit for public release. Jul 10, 2017
Makefile Makefile and documentation clean up (#12) Aug 1, 2017
Pipfile Release of v1.1.4 (#193) Jul 15, 2019
README.md Update Maestro logo link to full link for PyPi. Jan 22, 2020
release.sh Update to make release executable. Sep 8, 2017
requirements.txt Release of v1.1.4 (#193) Jul 15, 2019
setup.cfg First pass at a release shell script. (#39) Aug 9, 2017
setup.py Drop support for Python 3.4 (#218) Jan 17, 2020
tox.ini Drop support for Python 3.4 (#218) Jan 17, 2020

README.md

Maestro Workflow Conductor (maestrowf)

Build Status PyPI Spack Issues Forks Stars License

Downloads Downloads Downloads

Maestro can be installed via pip:

pip install maestrowf

Getting Started is Quick and Easy

Create a YAML file named study.yaml and paste the following content into the file:

description:
    name: hello_world
    description: A simple 'Hello World' study.

study:
    - name: hello_world
      description: Say hello to the world!
      run:
          cmd: |
            echo "Hello, World!" > hello_world.txt

PHILOSOPHY: Maestro believes in the principle of a clearly defined process, specified as a list of tasks, that are self-documenting and clear in their intent.

Running the hello_world study is as simple as...

maestro run study.yaml

Creating a Parameter Study is just as Easy

With the addition of the global.parameters block, and a few simple tweaks to your study block, the complete specification should look like this:

description:
    name: hello_planet
    description: A simple study to say hello to planets (and Pluto)

study:
    - name: hello_planet
      description: Say hello to a planet!
      run:
          cmd: |
            echo "Hello, $(PLANET)!" > hello_$(PLANET).txt

global.parameters:
    PLANET:
        values: [Mercury, Venus, Earth, Mars, Jupiter, Saturn, Uranus, Pluto]
        label: PLANET.%%

PHILOSOPHY: Maestro believes that a workflow should be easily parameterized with minimal modifications to the core process.

Maestro will automatically expand each parameter into its own isolated workspace, generate a script for each parameter, and automatically monitor execution of each task.

And, running the study is still as simple as:

    maestro run study.yaml

Scheduling Made Simple

But wait there's more! If you want to schedule a study, it's just as simple. With some minor modifications, you are able to run on an HPC system.

description:
    name: hello_planet
    description: A simple study to say hello to planets (and Pluto)

batch:
    type:  slurm
    queue: pbatch
    host:  quartz
    bank:  science

study:
    - name: hello_planet
      description: Say hello to a planet!
      run:
          cmd: |
            echo "Hello, $(PLANET)!" > hello_$(PLANET).txt
          nodes: 1
          procs: 1
          walltime: "00:02:00"

global.parameters:
    PLANET:
        values: [Mercury, Venus, Earth, Mars, Jupiter, Saturn, Uranus, Pluto]
        label: PLANET.%%

NOTE: This specification is configured to run on LLNL's quartz cluster. Under the batch header, you will need to make the necessary changes to schedule onto other HPC resources.

PHILOSOPHY: Maestro believes that how a workflow is defined should be decoupled from how it's run. We achieve this capability by providing a seamless interface to multiple schedulers that allows Maestro to readily port workflows to multiple platforms.

For other samples, see the samples subfolder. To continue with our Hello World example, see the Basics of Study Construction in our documentation.

An Example Study using LULESH

Maestro comes packed with a basic example using LULESH, a proxy application provided by LLNL. You can find the example here.

What is Maestro?

Maestro Workflow Conductor is a Python tool and library for specifying and automating multi-step computational workflows both locally and on supercomputers. Maestro parses a human-readable YAML specification that is self-documenting and portable from one user and environment to another.

On the backend, Maestro implements a set of standard interfaces and data structures for handling "study" construction. These objects offer you the ability to use Maestro as a library, and construct your own workflows that suit your own custom needs. We also offer other structures that make portable execution on various schedulers much easier than porting scripts by hand.

Maestro's Foundation and Core Concepts

There are many definitions of workflow, so we try to keep it simple and define the term as follows:

A set of high level tasks to be executed in some order, with or without dependencies on each other.

We have designed Maestro around the core concept of what we call a "study". A study is defined as a set of steps that are executed (a workflow) over a set of parameters. A study in Maestro's context is analogous to an actual tangible scientific experiment, which has a set of clearly defined and repeatable steps which are repeated over multiple specimen.

Maestro's core tenets are defined as follows:

Repeatability

A study should be easily repeatable. Like any well-planned and implemented science experiment, the steps themselves should be executed the exact same way each time a study is run over each set of parameters or over different runs of the study itself.

Consistent

Studies should be consistently documented and able to be run in a consistent fashion. The removal of variation in the process means less mistakes when executing studies, ease of picking up studies created by others, and uniformity in defining new studies.

Self-documenting

Documentation is important in computational studies as much as it is in physical science. The YAML specification defined by Maestro provides a few required key encouraging human-readable documentation. Even further, the specification itself is a documentation of a complete workflow.


Setting up your Python Environment

To get started, we recommend using virtual environments. If you do not have the Python virtualenv package installed, take a look at their official documentation to get started.

To create a new virtual environment:

python -m virtualenv maestro_venv
source maestro_venv/bin/activate

Getting Started for Contributors

If you plan to develop on Maestro, install the repository directly using:

pip install -r requirements.txt
pip install -e .

Once set up, test the environment. The paths should point to a virtual environment folder.

which python
which pip

Contributors

Many thanks go to MaestroWF's contributors.

If you have any questions or to submit feature requests please open a ticket.


Release

MaestroWF is released under an MIT license. For more details see the NOTICE and LICENSE files.

LLNL-CODE-734340

You can’t perform that action at this time.