add a page about Kubeflow integration

docs/index.md

@@ -51,7 +51,7 @@ Metaflow makes it easy to build and manage real-life data science, AI, and ML pr
 ## III. Deploying to Production
 
 - [Introduction to Production Deployments](production/introduction)
-- [Scheduling Metaflow Flows](production/scheduling-metaflow-flows/introduction)
+- [Scheduling Metaflow Flows](production/scheduling-metaflow-flows/introduction) ✨*New: support for Kubeflow*✨
 - [Coordinating Larger Metaflow
   Projects](production/coordinating-larger-metaflow-projects)
 - [Connecting Flows via Events](production/event-triggering/)

docs/production/introduction.md

@@ -135,6 +135,9 @@ human intervention.
       - [Scheduling flows with Apache
         Airflow](/production/scheduling-metaflow-flows/scheduling-with-airflow)
         - Choose to stay compatible with your existing Airflow deployment.
+      - [Scheduling flows with
+        Kubeflow](/production/scheduling-metaflow-flows/scheduling-with-kubeflow)
+        - Choose to stay compatible with your existing Kubeflow deployment.
 
  2. [Coordinating larger Metaflow
     projects](/production/coordinating-larger-metaflow-projects) is a more advanced pattern

docs/production/scheduling-metaflow-flows/introduction.md

@@ -10,14 +10,25 @@ it has failed? How do you see its error logs? Does the Cron server have enough c
 to handle another command? And most importantly, how should you orchestrate schedules of
 multiple commands so that their mutual dependencies are handled correctly?
 
-Metaflow currently integrates with [Argo
-Workflows](../scheduling-metaflow-flows/scheduling-with-argo-workflows.md) ([a modern,
-Kubernetes-native workflow orchestrator](https://argoproj.github.io/workflows)), [AWS
-Step Functions](../scheduling-metaflow-flows/scheduling-with-aws-step-functions.md) ([a
-managed general-purpose orchestrator](https://aws.amazon.com/step-functions/)), and
-[Apache Airflow](../scheduling-metaflow-flows/scheduling-with-airflow.md) ([a
-widely-known open-source orchestrator](https://airflow.apache.org/)) which can answer
-these questions.
+Metaflow currently integrates with a number of production orchestrators which can answer
+these questions:
+
+- [Argo Workflows](https://argoproj.github.io/workflows) - [a modern,
+Kubernetes-native workflow orchestrator,
+- [AWS Step Functions](https://aws.amazon.com/step-functions/) - a managed general-purpose orchestrator,
+- [Apache Airflow](https://airflow.apache.org/) - a
+widely-known orchestrator focused on data engineering,
+- [Kubeflow](https://www.kubeflow.org/) - an open-source
+ML/AI platform on Kubernetes.
+
+### Which production orchestrator to use?
+
+If you do not have existing orchestration infrastructure, **Argo Workflows** is a solid choice if you are ready to [deploy Metaflow on Kubernetes](/getting-started/infrastructure). **AWS Step Functions** is an easy-to-adopt alternative that runs natively on AWS, though with a more limited feature set.
+
+The **Airflow** and **Kubeflow** integrations are useful if these orchestrators are
+already deployed in your environment and you want to benefit from Metaflow’s developer experience without introducing a new orchestrator. While the feature set is more
+limited, these integrations let you get started with Metaflow without requiring
+migrations.
 
 Learn more about how to deploy your Metaflow flows to these orchestrators in the
 following subsections:
@@ -27,6 +38,7 @@ following subsections:
 - [Scheduling with AWS Step
   Functions](../scheduling-metaflow-flows/scheduling-with-aws-step-functions)
 - [Scheduling with Airflow](../scheduling-metaflow-flows/scheduling-with-airflow)
+- [Scheduling with Kubeflow](../scheduling-metaflow-flows/scheduling-with-kubeflow)
 
 :::tip
 Note that you can [manage production deployments programmatically](/metaflow/managing-flows/deployer)

docs/production/scheduling-metaflow-flows/scheduling-with-kubeflow.md

@@ -0,0 +1,181 @@
+import ReactPlayer from 'react-player'
+
+# Scheduling Metaflow Flows with Kubeflow
+
+:::info Note
+The Kubeflow integration is new and currently released as a Metaflow extension
+which you need to install separately as instructed below. The feature set is
+still evolving and hence subject to change. Reach out to us on [Metaflow Slack](http://slack.outerbounds.co) if you have questions about the integration!
+:::
+
+Metaflow’s Kubeflow integration lets you develop workflows using Metaflow’s full, developer-friendly feature set, and then deploy those flows to your existing Kubeflow infrastructure, where they run seamlessly alongside your current Kubeflow pipelines.
+
+The deployed pipelines are **both valid Metaflow flows and Kubeflow pipelines**,
+allowing you to observe them in real time in the Kubeflow UI and in the Metaflow UI,
+and to access results through Metaflow’s [Client API](/metaflow/client) as usual.
+
+You can see this in action in this short screencast (no sound):
+
+<ReactPlayer controls url="https://youtu.be/ALg0A9SzRG8" />
+<br/>
+
+### Why use Metaflow with Kubeflow
+
+The video highlights the main benefits of the integration:
+
+- Metaflow provides [**a top-notch developer
+  experience**](https://netflixtechblog.com/supercharging-the-ml-and-ai-development-experience-at-netflix-b2d5b95c63eb),
+  sparing the developer from Kubernetes-specific technical details.
+
+- You can [test flows locally at arbitrary scale](/scaling/introduction)
+  using **the same Kubernetes infrastructure** that you use with Kubeflow.
+
+- Most Metaflow features, such as
+  [resuming](/metaflow/debugging#how-to-use-the-resume-command),
+  [observability with cards](/metaflow/visualizing-results),
+  [dependency management](/scaling/dependencies),
+  [configuration management](/metaflow/configuring-flows/introduction),
+  [namespaces](/scaling/tagging),
+  [artifacts and Client API](/metaflow/client),
+  work out of the box, **greatly enhancing the functionality available
+  in Kubeflow Pipelines**.
+
+- **Deploy to Kubeflow Pipelines with a single command**, no changes
+  in the Metaflow code required.
+
+Note that Kubeflow Pipelines is built on top of Argo Workflows, so its scalability and
+high-availability characteristics closely mirror those of Metaflow’s native
+[Argo Workflows integration](/production/scheduling-metaflow-flows/scheduling-with-argo-workflows), with potential additional overhead
+introduced by the Kubeflow components.
+
+## Setting up the Kubeflow-Metaflow integration
+
+Currently, the integration is provided as a Metaflow Extension which you can
+install as follows
+
+```
+pip install metaflow-kubeflow
+```
+
+Note that you have to install the extension only on the laptop, server, or
+CI/CD worker where you deploy workflows. It doesn't need to be present on
+containers executing tasks.
+
+:::info Note
+In order to be able to deploy to Kubeflow, you need to be able to connect to
+Kubeflow. In case you don't have connectivity already set up, [see this README
+for suggestions](https://github.com/outerbounds/metaflow-kubeflow).
+:::
+
+You can specify the Kubeflow endpoint address every time you deploy, or you can
+add it in your Metaflow config (typically at `~/.metaflowconfig/config.json`):
+```
+ "METAFLOW_KUBEFLOW_PIPELINES_URL": "http://my-kubeflow",
+```
+Replace `http://my-kubeflow` with the actual address.
+
+## Pushing a flow to Kubeflow Pipelines
+
+Let's use [the flow from the section about
+parameters](../../metaflow/basics#how-to-define-parameters-for-flows) as an example:
+
+```python
+from metaflow import FlowSpec, Parameter, step
+
+class ParameterFlow(FlowSpec):
+    alpha = Parameter('alpha',
+                      help='Learning rate',
+                      default=0.01)
+
+    @step
+    def start(self):
+        print('alpha is %f' % self.alpha)
+        self.next(self.end)
+
+    @step
+    def end(self):
+        print('alpha is still %f' % self.alpha)
+
+if __name__ == '__main__':
+    ParameterFlow()
+```
+
+Save the flow in a file `parameter_flow.py`. To deploy a version to Kubeflow Pipelines,
+type
+```
+python parameter_flow.py kubeflow-pipelines create
+```
+Or, if you haven't specified `METAFLOW_KUBEFLOW_PIPELINES_URL` in your Metaflow
+config, specify a Kubeflow URL on the command line:
+```
+python parameter_flow.py kubeflow-pipelines create --url http://my-kubeflow
+```
+
+This command will [package the flow and its execution
+environment](scaling/dependencies), convert it to a Kubeflow Pipeline, and deploy
+it to the specified server.
+
+The pipeline name matches the flow name. Each deployment created with
+`kubeflow-pipelines create` produces a new pipeline version, which by default is
+named using the current timestamp, unless you explicitly override the version name
+with the `--version-name` option.
+
+:::info Note
+[Conditional and recursive steps](/metaflow/basics#conditionals)
+introduced in Metaflow 2.18, are not yet supported
+on Kubeflow deployments. Contact [the Metaflow Slack](http://slack.outerbounds.co) if
+you have a use case for this feature.
+:::
+
+## Triggering a Kubeflow run
+
+You can trigger a deployed flow to run on Kubeflow pipelines with the `trigger`
+command:
+```
+python parameter_flow.py kubeflow-pipelines trigger
+```
+Specify `--url` as above, unless the URL is specified in the config.
+
+You can pass parameters to a run as usual:
+```
+python parameter_flow.py kubeflow-pipelines trigger --alpha 0.5
+```
+
+You can also trigger a run as a specific [Kubeflow
+Experiment](https://www.kubeflow.org/docs/components/pipelines/concepts/experiment/):
+```
+python parameter_flow.py kubeflow-pipelines trigger --alpha 0.5 --experiment new_moddel
+```
+By default, the latest version of the flow is triggered. You can trigger an
+older version by specifying `--version-name`:
+```
+python parameter_flow.py kubeflow-pipelines trigger --alpha 0.5 --version-name 20251216021104161376
+```
+
+:::info Note
+Currently only manual triggering with `trigger` is supported. If you are interested
+in [scheduled](/api/flow-decorators/schedule) or
+[event-triggered runs](/production/event-triggering),
+reach out to us on [Metaflow Slack](http://slack.outerbounds.co)
+:::
+
+## Inspecting the results of a Kubeflow run
+
+Every Kubeflow run is a valid Metaflow run which you can inspect using the Metaflow UI
+and [the Client API](/metaflow/client) as usual.
+
+Note that [Metaflow's namespaces](scaling/tagging#production-namespaces) apply on
+Kubeflow too, so to access Kubeflow results, you can switch to [the global
+namespace](/scaling/tagging#global-namespace), like here:
+
+```python
+from metaflow import Flow, namespace
+namespace(None)
+run = Flow('ParameterFlow')['kfp-066dcf8a-61dd-4f61-b652-e161124bc3b3']
+print(f'Alpha is {run.data.alpha}')
+```
+
+:::tip Run IDs match
+Metaflow Run IDs corresponding to Kubeflows runs match their Kubeflow
+run IDs, prefixed with `kfp-` (e.g. `kfp-066dcf8a-61dd-4f61-b652-e161124bc3b3`) so you can easily identify and track lineage of runs between Metaflow and Kubeflow.
+:::

sidebars.js

@@ -235,6 +235,7 @@ const sidebars = {
             "production/scheduling-metaflow-flows/scheduling-with-argo-workflows",
             "production/scheduling-metaflow-flows/scheduling-with-aws-step-functions",
             "production/scheduling-metaflow-flows/scheduling-with-airflow",
+            "production/scheduling-metaflow-flows/scheduling-with-kubeflow"
           ],
         },
         "production/coordinating-larger-metaflow-projects",