Run v1alpha1 and v1alpha2 TfJob tests in GKE workflow

[ankushagarwal]

• Update deploy_kubeflow_gcp to add tfjob_version flag
• In GKE Testing workflow, bring up two k8s clusters - one to test v1alpha1 and other to test v1alpha2
• Deploy kubeflow on both clusters using bootstrapper - set tfjob_version different for both clusters
• Using different kubeConfig for both clusters
• Teardown both clusters at the end

Fixes #852

This change is 

[ankushagarwal]

/retest

[ankushagarwal]

/retest

[ankushagarwal]

/test all

[ankushagarwal]

/retest

[ankushagarwal]

/retest

[ankushagarwal]

/retest

[ankushagarwal]

/retest

[ankushagarwal]

/cc @kunmingg
/cc @jlewi

This is ready for review

Here is a sample run of this workflow : http://testing-argo.kubeflow.org/workflows/kubeflow-test-infra/kubeflow-presubmit-kubeflow-e2e-gke-918-d810986-2005-1454?tab=workflow

[jlewi]

Did you consider parameterizing the workflow by TFVersion and then submitting two different workflows? One for v1alpha1 and v1alpha2?

[ankushagarwal]

Yes. That wasn't a clean approach either. A couple of problems in that approach:

1. Tests would be repeated - pytorchjob tests, jsonnet tests - needed to add logic to skip tests
2. The workflow copies all the junit-xml files to a common artifacts directory at the end of the test. Running multiple copies of the test would cause the artifacts to be overridden - we would have to add logic to create files with different names for each of the workflow steps

[jlewi]

/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jlewi

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [jlewi]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment