Metaflow
title: Metaflow radar_quadrant: Platforms radar_ring: Assess radar_position: inner created: 2026-05-26 last_updated: 2026-05-26 tags: [ml, data-science, pipelines, python, netflix, aws, kubernetes] source_url: https://github.com/Netflix/metaflow
An open-source Python framework for building and deploying data science and ML workflows, developed at Netflix and open-sourced in 2019. Now maintained by Outerbounds. Production-validated at Netflix scale since 2018.
Metaflow separates the data science logic from the operational concerns. You write Python functions decorated with @step; Metaflow handles versioning, compute scaling, failure recovery, and experiment tracking automatically. The workflow is defined as a directed acyclic graph of steps, each of which can run locally or be pushed to AWS Batch, Kubernetes, or other compute backends with a single decorator change.
Automatic versioning: every run's inputs, outputs, and code snapshot are stored and queryable. You can reproduce any past run or compare results across runs without additional tooling.
Compute scaling: steps annotated with @batch or @kubernetes execute on remote compute; unannotated steps run locally. Mix local and remote steps in the same workflow.
Resumability: failed runs resume from the last successful step, not from scratch. Expensive upstream steps are not re-executed on downstream failures.
Data access: the @conda decorator manages dependencies per step; self.data passes artifacts between steps with automatic serialisation.
Metaflow sits between ad-hoc Jupyter notebooks (no versioning, no scaling) and heavyweight MLOps platforms (Kubeflow, SageMaker Pipelines) that require significant infrastructure investment. It is the right choice for teams that want production-grade ML pipelines without a dedicated MLOps engineer.
Placed in Platforms / Assess / inner. Netflix production pedigree and active Outerbounds maintenance reduce adoption risk. Python-native API with minimal boilerplate, local-first development, and incremental cloud scale-out make the trial path straightforward. Inner position reflects direct applicability to any ML or data pipeline that has outgrown notebooks. Trial gate: one end-to-end ML pipeline (data loading → training → evaluation) running locally with at least one step pushed to a remote compute backend and results inspected via the Metaflow UI or client.
