e2e tests should allow multiple concurrent runs

[roberthbailey]

Refactor the e2e tests to allow multiple concurrent runs (assuming it is supported by the cloud provider). For GCE, this will require that the kubectl client can authenticate to multiple clusters (#1755). For GKE, this is already supported in gcloud.

[zmerlynn]

Actually, you can mostly do this on gce in a ghetto way by bonking HOME and
overriding the parameters I added. But I need to add the CIDR to that set,
and do more testing.

On Tue, Dec 23, 2014, 13:19 roberthbailey notifications@github.com wrote:

Refactor the e2e tests to allow multiple concurrent runs (assuming it is
supported by the cloud provider). For GCE, this will require that the
kubectl client can authenticate to multiple clusters (#1755
#1755). For
GKE, this is already supported in gcloud.

—
Reply to this email directly or view it on GitHub
#3132.

[lavalamp]

Can you clarify? You want to stand up multiple e2e clusters and run a complete set of tests on each cluster, right? That's already possible in the way @zmerlynn says, and will be less hacky in the future when we solve the general problem of keeping auth to multiple clusters around.

Another possibility is to just run all the tests in parallel. The downside to this is that they might interfere with each other, and that it might make failures harder to understand. But I think that it might be worth it for the extra speed.

[zmerlynn]

I believe the intent of this bug came out of my angst over things like
Jenkins being able to do multiple things in parallel, since it's a limiter
right now.

On Tue, Dec 23, 2014 at 3:00 PM, Daniel Smith notifications@github.com
wrote:

Can you clarify? You want to stand up multiple e2e clusters and run a
complete set of tests on each cluster, right? That's already possible in
the way @zmerlynn https://github.com/zmerlynn says, and will be less
hacky in the future when we solve the general problem of keeping auth to
multiple clusters around.

Another possibility is to just run all the tests in parallel. The downside
to this is that they might interfere with each other, and that it might
make failures harder to understand. But I think that it might be worth it
for the extra speed.

—
Reply to this email directly or view it on GitHub
#3132 (comment)
.

[roberthbailey]

It would also increase our confidence in a release if we had multiple e2e test runs passing for the release (since that will help detect flakes in the tests as well as the code). Being able to parallelize a bunch of e2e tests (e.g. ./run100e2etests.sh ) would be nice.

[zmerlynn]

I actually intend to setup a --times=10 long job on Jenkins, but the
stability of the main e2e jobs is ... lacking. With the half hour drumbeat,
it's pretty easy to see how bad we're doing today. :( But yes, we will need
to parallelize after we do better.

On Tue, Dec 23, 2014 at 3:20 PM, roberthbailey notifications@github.com
wrote:

It would also increase our confidence in a release if we had multiple e2e
test runs passing for the release (since that will help detect flakes in
the tests as well as the code). Being able to parallelize a bunch of e2e
tests (e.g. ./run100e2etests.sh ) would be nice.

—
Reply to this email directly or view it on GitHub
#3132 (comment)
.

[ikehz]

We now have PARALLEL test suites in hack/jenkins/e2e.sh.

Proposal to close this issue in favor of a work-issue to track moving as many e2es into the parallel suite as possible.

[zmerlynn]

Yup.