Popper is a tool for defining and executing container-native workflows in Docker, as well as other container engines. With Popper, you define a workflow in a YAML file, and then execute it with a single command. A workflow file looks like this:
steps: # download CSV file with data on global CO2 emissions - id: download uses: docker://byrnedo/alpine-curl:0.1.8 args: [-LO, https://github.com/datasets/co2-fossil-global/raw/master/global.csv] # obtain the transpose of the global CO2 emissions table - id: get-transpose uses: docker://getpopper/csvtool:2.4 args: [transpose, global.csv, -o, global_transposed.csv]
Assuming the above is stored in a
.popper.yml file in your project
folder, this entire workflow gets executed by running:
cd /path/to/my/project/ popper run
Running a single step:
popper run get-transpose
Starting a shell inside the
get-transpose step container:
popper sh get-transpose
To install or upgrade Popper, run the following in your terminal:
curl -sSfL https://raw.githubusercontent.com/getpopper/popper/master/install.sh | sh
Docker is required to run Popper and the installer will
abort if the
docker command cannot be invoked from your shell. For
other installation options, including installing for use with
Singularity or for setting up a developing environment for Popper,
read the complete installation instructions.
Once installed, you can get an overview and list of available commands:
Lightweight workflow and task automation syntax. Defining a list of steps is as simple as writing file in a lightweight YAML syntax and invoking
popper run(see demo above). If you're familiar with Docker Compose, you can think of Popper as Compose but for workflows instead of services.
An abstraction over container runtimes. In addition to Docker, Popper can seamlessly execute workflows in other runtimes by interacting with distinct container engines. Popper currently supports Singularity and we are working on adding Podman.
An abstraction over resource managers. Popper can also execute workflows on a variety of resource managers and schedulers such as Kubernetes and SLURM, without requiring any modifications to a workflow YAML file. We currently support SLURM and are working on adding support for Kubernetes.
An abstraction over CI services. Define a pipeline once and then instruct Popper to generate configuration files for distinct CI services, allowing users to run the exact same workflows they run locally on Travis, Jenkins, Gitlab, Circle and others. See the
examples/folder for examples on how to automate CI tasks for multiple projects (Go, C++, Node, etc.).
What Problem Does Popper Solve?
Popper is a container-native workflow execution and task automation
engine. In practice, when we work following the container-native
paradigm, we end up interactively executing multiple
docker pull|build|run commands in order to build containers, compile code,
test applications, deploy software, etc. Keeping track of which
docker commands we have executed, in which order, and which flags
were passed to each, can quickly become unmanageable, difficult to
document (think of outdated README instructions) and error prone.
The goal of Popper is to bring order to this chaotic scenario by providing a framework for clearly and explicitly defining container-native tasks. You can think of Popper as tool for wrapping all these manual tasks in a lightweight, machine-readable, self-documented format (YAML).
While this sounds simple at first, it has significant implications: results in time-savings, improves communication and in general unifies development, testing and deployment workflows. As a developer or user of "Popperized" container-native projects, you only need to learn one tool, and leave the execution details to Popper, whether is to build and tests applications locally, on a remote CI server or a Kubernetes cluster.
Popper adheres to the code of conduct posted in this repository. By participating or contributing to Popper, you're expected to uphold this code. If you encounter unacceptable behavior, please immediately email us.
How to Cite Popper
Ivo Jimenez, Michael Sevilla, Noah Watkins, Carlos Maltzahn, Jay Lofstead, Kathryn Mohror, Andrea Arpaci-Dusseau and Remzi Arpaci-Dusseau. The Popper Convention: Making Reproducible Systems Evaluation Practical. In 2017 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW), 1561–70, 2017. (https://doi.org/10.1109/IPDPSW.2017.157)