Use Argo workflow engine for CI/CD or releases

[jlewi]

https://applatix.com/open-source/argo/

This looks like it could be much lighter weight than Airflow. Looks like it would be simpler to allow testing of changes to the workflow in presubmits.

Might be interesting to experiment with Argo for our release pipeline and see if we like it more than Airflow.

[bhack]

Seems interesting and with a "native" design.

[jimexist]

Personally, I think Argo (or any other mature k8s-native approach) would be a better fit than Airflow. It allows for better inter-operationality and a unified resource control and scheduling.

Would love to give it a try soon.

[jlewi]

We're using Argo for google/kubeflow and its much better. I think we want to migrate ASAP because the friction around Airflow is impeding developer velocity.

[bhack]

There is also https://github.com/argoproj/argo-ci in early stages.

[jlewi]

I haven't looked at argo-ci. Any idea what it provides relative to prow?

[bhack]

I suppose that is an early stage CI Argo flavour (SCM integration, Artifact, Deployment etc..)
/cc @alexmt

[jlewi]

Replacing Airflow with Argo will make it much easier for contributors to access the logs. I think we can make the Argo UI publicly available and I think its more friendly then the Airflow Ui.

I think to dump Airflow we need to do the following

Add a flag to py/release.py so we can control the label that will be used for the docker image and in the helm chart path.

• This way we know what path to use for the helm chart in subsequent steps and we won't need a mechanism like xcom to allow steps to communicate.

At this point you can write a ksonnet component to run our E2E tests. You can look at what I did for Kubeflow.

@ScorpioCPH Any help with this would be great. I think removing the friction around our E2E tests will allow us to iterate more quickly.

[ScorpioCPH]

@jlewi Thanks for the background information! Will take a look tomorrow :)

[alexmt]

@jlewi, Argo CI is envisioned as travis-ci like system, powered by argo workflows. I've briefly looked at prow and seems like argo ci is very similar. Roadmap is available here: https://github.com/argoproj/argo-ci/blob/master/ROADMAP.md . Requests and comments are very welcome.

[jlewi]

We've been using Prow for a quite a while and I think its been around longer then Argo. I think there's overlap between Prow and Argo CI. Its not obvious to me what benefit Argo CI would provide over the corresponding features in Prow.

The current plan is to use Prow to trigger Argo workflows; this is what we do for google/kubeflow and it works pretty well. We'd like to do the same for tensorflow/k8s.

[jlewi]

I've started working on this.

[jlewi]

#358 is submitted. Tests are using Argo now.

I deleted the cluster GKE cluster prow in project mlkube-testing which was used to run Airflow.