Integration with Kubeflow pipelines

[BAlmeidaS]

First things first, thanks to open this project!! We are working in a very similar project, and Metaflow is going to help us a lot.

Do you plan to create a kind of integration with Kubeflow Pipelines? For us, this is very helpful to deploy these pipelines in our production environment.

[SteNicholas]

@BAlmeidaS I mentioned this integration with kubeflow in issue Support for Airflow and Kubernetescomment. I really thought that metaflow integration with kubeflow reduce work in integration on Kubernetes.

[BAlmeidaS]

Yes @SteNicholas I saw the issue, I decided opened mine anyway because I thought that one thing is integration with k8s and deploy an orchestrated pipeline with docker. Another thing is to deploy a Kubeflow pipelines targz (which is similar but not the same thing).

[savingoyal]

Thanks for opening the issue, @BAlmeidaS . Issues like these are really informative as we go about planning our roadmap.

[Jeffwan]

@BAlmeidaS @
There's different options to integrate with KFP. User could use KFP SDK in metaflow or do something further and change orchestration and computing layer to kubeflow. Do you have some ideas in mind?

[valayDave]

Kubeflow's Pipeline features are a little different from Metaflow's DAG features. Kubeflow supports conditional Steps in workflows. Metaflow doesn't have support for conditional flows.

[talebzeghmi]

Kubeflow supports conditional branching concept that Metaflow doesn't. Metaflow could compile to Kubeflow via an intermediate representation (kubeflow/pipelines#3703)

[savingoyal]

@talebzeghmi I responded in this other thread. Our current integration to AWS Step Functions' State Machine Representation is in PR #204 .

[savingoyal]

The reference implementation for AWS Step Functions is available here now. This should serve as a good starting point for any other scheduler integration.

[BAlmeidaS]

Sorry for the enormous delay to answer this, I lost the tracking about this project. I was very excited about it when it launched but we developed our own solution to integrate with kfp. Answering @Jeffwan, IMO we could use kfp to deploy via metaflow. User may not notice that she is deploying a pipeline in kubeflow, even metaflow not having some features that kubeflow pipelines has. What you guys think about it??

[savingoyal]

Closing this issue given that #992 that addresses #50 has been merged. Kubeflow Pipelines also uses Argo Workflows under the hood so a layer of indirection through Kubeflow Pipelines doesn't really serve much benefit currently.