Add e2e workflows against AWS CI

[Jeffwan]

Community is asking different WG to own their infra and community won't provide a common shared testing infra anymore. Sees kubeflow/testing#752 for more details.

Pytorch migration works well and here's the PR to update test case to work on AWS Prow+ Argo cluster.
kubeflow/pytorch-operator#305

Seems mpi-operator has empty workflow defined https://github.com/kubeflow/mpi-operator/blob/master/prow_config.yaml
Whenever it plans to add real e2e tests, it can follow what pytorch operator did and test each PR before the merge.

[terrytangyuan]

cc @kubeflow/wg-training-leads

[alculquicondor]

@Jeffwan, I would like to add E2E tests.

Community is asking different WG to own their infra and community won't provide a common shared testing infra anymore.

Does this mean that the mpi-operator project has to find where to run their tests? Or is there any common infra already in place for the training WG? If so, Is there any guidebook to start adding the workflows and tests?

[Jeffwan]

@Jeffwan, I would like to add E2E tests.

Community is asking different WG to own their infra and community won't provide a common shared testing infra anymore.

Does this mean that the mpi-operator project has to find where to run their tests? Or is there any common infra already in place for the training WG? If so, Is there any guidebook to start adding the workflows and tests?

https://github.com/kubeflow/testing/tree/master/aws We use this infra for e2e testing. It can provision the cluster and you can run test against it.

[Jeffwan]

/cc @alculquicondor

[alculquicondor]

We added local tests using kind. This should be enough for now. But thanks for the pointer, maybe it will be useful in the future for more complex workloads.