Re-enable click_deploy_test

[lluunn]

https://github.com/kubeflow/kubeflow/blob/master/prow_config.yaml#L6

I think it's not being run:
https://k8s-testgrid.appspot.com/sig-big-data#kubeflow-periodic-master

@kunmingg mentioned the cluster might need more quota.

[kunmingg]

An easy approach is to make e2e test clean up before each run, we can reuse what prober test is doing today:
https://github.com/kubeflow/kubeflow/blob/master/testing/test_deploy_app.py#L363

[lluunn]

It looks like it should be running in kubeflow-ci/kubeflow-testing cluster:
https://github.com/kubeflow/kubeflow/blob/master/testing/workflows/components/click_deploy_test.jsonnet#L192

@kunmingg If it's quota problem, shouldn't the test still be run, and then failed?

[kunmingg]

Let's setup a dedicated cluster for it.
Will double check periodic test config, maybe some PRs break the periodic test config

[lluunn]

From this run: https://gubernator.k8s.io/build/kubernetes-jenkins/logs/kubeflow-periodic-master/719

The log shows: the container main in pod XXX is not available: Bad Request

[lluunn]

manually launched the workflow. Saw:

CreateContainerConfigError: secrets "kubeflow-oauth" not found

[lluunn]

The plan is to deploy to a new project (kubflow_ci_bootstrap or something under kubeflow.org) so that quotas/resources are easier to cleanup.

[lluunn]

Created project: kubeflow-ci-deployment

[lluunn]

Still failing:

go: finding github.com/prometheus/client_model v0.0.0-20180712105110-5c3871d89910
go: finding google.golang.org/genproto v0.0.0-20180831171423-11092d34479b
go: finding go.opencensus.io v0.18.1-0.20181204023538-aab39bd6a98b
go: finding github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973
go: error loading module requirements
+ result=1
+ set -e
+ [[ 1 -eq 0 ]]
+ echo Command failed: /mnt/test-data-volume/kubeflow-periodic-master-deployapp-736-7f63/src/github.com/kubeflow/kubeflow/bootstrap
/build_image.sh /mnt/test-data-volume/kubeflow-periodic-master-deployapp-736-7f63/src/github.com/kubeflow/kubeflow/bootstrap/Docker
file gcr.io/kubeflow-ci/bootstrapper kubeflow-periodic-master-deployapp-736-7f63
Command failed: /mnt/test-data-volume/kubeflow-periodic-master-deployapp-736-7f63/src/github.com/kubeflow/kubeflow/bootstrap/build_
image.sh /mnt/test-data-volume/kubeflow-periodic-master-deployapp-736-7f63/src/github.com/kubeflow/kubeflow/bootstrap/Dockerfile gc
r.io/kubeflow-ci/bootstrapper kubeflow-periodic-master-deployapp-736-7f63
+ echo 'command didn'\''t succeed'
+ exit 1

The cleanup also failed:

+ kubectl delete namespace kubeflow-periodic-master-deployapp-736-7f63
Error from server (NotFound): namespaces "kubeflow-periodic-master-deployapp-736-7f63" not found

[lluunn]

Verify the error is the same when building locally. The problem is one dependency needs hg to clone

[lluunn]

Now the error is this line:
https://github.com/kubeflow/kubeflow/blob/master/bootstrap/cmd/bootstrap/app/ksServer.go#L1176

While it's waiting for the deployment manager, got an error read: connection reset by peer
@kunmingg have you seen this before?

The deployment succeeded eventually

[lluunn]

https://gubernator.k8s.io/build/kubernetes-jenkins/logs/kubeflow-periodic-master/756
it's passing now, but now shown on testgrid,
will fix in kubeflow/testing#301