Skip to content
master
Switch branches/tags
Code

CI Linux CI macOS CI Windows Documentation Status PyPI Conda (channel only) Conda Coverage Twitter Binder Downloads

Join our community | Newsletter | Contact us | Docs | Blog | Website | YouTube

Notebooks are hard to maintain. Teams often prototype projects in notebooks, but maintaining them is an error-prone process that slows progress down. Ploomber overcomes the challenges of working with .ipynb files allowing teams to develop collaborative, production-ready pipelines using JupyterLab or any text editor.

Get Started Open JupyerLab

Installation

Compatible with Python 3.6 and higher.

Install with pip:

pip install ploomber

Or with conda:

conda install ploomber -c conda-forge

Getting started

Use Binder to try out Ploomber without setting up an environment:

image

Or run an example locally:

# ML pipeline example
ploomber examples -n templates/ml-basic -o ml-basic
cd ml-basic

# if using pip
pip install -r requirements.txt

# if using conda
conda env create --file environment.yml
conda activate ml-basic

# run pipeline
ploomber build

Pipeline output saved in the output/ folder. Check out the pipeline definition in the pipeline.yaml file.

To get a list of examples, run ploomber examples.

Click here to go to our examples repository.

Community

Main Features

  1. Automated notebook refactoring. Automatically convert a legacy notebook into a maintainable, modular pipeline (see demo).
  2. Scripts as notebooks. Open .py files as notebooks, then execute them from the terminal and generate an output notebook to review results.
  3. Dependency resolution. Quickly build a DAG by referring to previous tasks in your code; Ploomber infers execution order and orchestrates execution.
  4. Incremental builds. Speed up iterations by skipping tasks whose source code hasn't changed since the last execution.
  5. Production-ready. Deploy to Kubernetes (via Argo Workflows), Airflow, and AWS Batch without code changes.
  6. Parallelization. Run independent tasks in parallel.
  7. Testing. Import pipelines in any testing frameworks and test them with any CI service (e.g. GitHub Actions).
  8. Flexible. Use Jupyter notebooks, Python scripts, R scripts, SQL scripts, Python functions, or a combination of them as pipeline tasks. Write pipelines using a pipeline.yaml file or with Python.

repo-lab-example

Resources