Backport smoke/conformance test harness changes to release-1.1 from master #18686

karlkfi · 2015-12-14T23:19:34Z

The following changes (hopefully) make it feasible to run release-1.1 branch conformance tests on master code in mesos environments. This should also fix mesos smoke tests on cherry-pick PRs to release-1.1.

Backport contrib/mesos/ci to release-1.1 MESOS: Add mesos/docker CI scripts #17511 MESOS/CI: call docker-run interactively and with terminal #17842 MESOS: add CONFORMANCE_BRANCH override #17835
Skip [Skipped] tests when testing for conformance e2e tag cleanup #17180
Add cluster/test-conformance.sh script Add cluster/test-conformance.sh script #17997
Support passing args to cluster/test-smoke.sh Support passing args to cluster/test-smoke.sh #15404
Support annotated tags in conformance test version detection conformance: support annotated tags #17839

(This only affects smoke/conformance tests. So e2e tests are not required.)

- Skip [Skipped] tests when testing for conformance kubernetes#17180 - Add cluster/test-conformance.sh script kubernetes#17997 - Support passing args to cluster/test-smoke.sh kubernetes#15404 - Support annotated tags in conformance test version detection kubernetes#17839

k8s-github-robot · 2015-12-14T23:29:37Z

Labelling this PR as size/L

karlkfi · 2015-12-14T23:41:54Z

Mesosphere CI

pull/18686/head:

master with CONFORMANCE_BRANCH=pull/18686/head:

Conformance Tests

k8s-bot · 2015-12-14T23:47:22Z

GCE e2e build/test failed for commit ffa5655.

karlkfi · 2015-12-15T01:36:29Z

Conformance Results:

pull/18686/head:

[16:34:07][Step 7/7] Summarizing 3 Failures:
[16:34:07][Step 7/7] 
[16:34:07][Step 7/7] [Fail] SchedulerPredicates [It] validates resource limits of pods that are allowed to run [Conformance] 
[16:34:07][Step 7/7] /go/src/github.com/GoogleCloudPlatform/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:87
[16:34:07][Step 7/7] 
[16:34:07][Step 7/7] [Fail] Kubectl client Guestbook application [It] should create and stop a working application [Conformance] 
[16:34:07][Step 7/7] /go/src/github.com/GoogleCloudPlatform/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:1010
[16:34:07][Step 7/7] 
[16:34:07][Step 7/7] [Fail] Networking [It] should function for intra-pod communication [Conformance] 
[16:34:07][Step 7/7] /go/src/github.com/GoogleCloudPlatform/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/networking.go:180
[16:34:07][Step 7/7] 
[16:34:07][Step 7/7] Ran 72 of 175 Specs in 3704.877 seconds
[16:34:07][Step 7/7] FAIL! -- 69 Passed | 3 Failed | 2 Pending | 101 Skipped >>> testContext.KubeConfig: /root/.kube/config
[16:34:07][Step 7/7] Skipping SSH core dump, which is not implemented for --- FAIL: TestE2E (3704.91s)
[16:34:07][Step 7/7] FAIL
[16:34:07][Step 7/7] 
[16:34:07][Step 7/7] Ginkgo ran 1 suite in 1h1m45.322904307s
[16:34:07][Step 7/7] Test Suite Failed

master with CONFORMANCE_BRANCH=pull/18686/head:

[16:32:17][Step 7/7] Summarizing 3 Failures:
[16:32:17][Step 7/7] 
[16:32:17][Step 7/7] [Fail] SchedulerPredicates [It] validates resource limits of pods that are allowed to run [Conformance] 
[16:32:17][Step 7/7] /go/src/github.com/GoogleCloudPlatform/kubernetes/conformance/_output/local/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:141
[16:32:17][Step 7/7] 
[16:32:17][Step 7/7] [Fail] EmptyDir volumes [It] should support (non-root,0777,default) [Conformance] 
[16:32:17][Step 7/7] /go/src/github.com/GoogleCloudPlatform/kubernetes/conformance/_output/local/go/src/k8s.io/kubernetes/test/e2e/util.go:1173
[16:32:17][Step 7/7] 
[16:32:17][Step 7/7] [Fail] Networking [It] should function for intra-pod communication [Conformance] 
[16:32:17][Step 7/7] /go/src/github.com/GoogleCloudPlatform/kubernetes/conformance/_output/local/go/src/k8s.io/kubernetes/test/e2e/networking.go:159
[16:32:17][Step 7/7] 
[16:32:17][Step 7/7] Ran 72 of 175 Specs in 2624.442 seconds
[16:32:17][Step 7/7] FAIL! -- 69 Passed | 3 Failed | 2 Pending | 101 Skipped >>> testContext.KubeConfig: /root/.kube/config
[16:32:17][Step 7/7] Skipping SSH core dump, which is not implemented for --- FAIL: TestE2E (2624.47s)
[16:32:17][Step 7/7] FAIL
[16:32:17][Step 7/7] 
[16:32:17][Step 7/7] Ginkgo ran 1 suite in 43m44.875094434s
[16:32:17][Step 7/7] Test Suite Failed

SchedulerPredicates test is fixed by #18348 (pending merge).

The other failures need inspection, but are probably related to code or test changes, not the test harness.

karlkfi · 2015-12-15T01:38:05Z

@k8s-bot e2e test this

karlkfi · 2015-12-15T01:42:57Z

@kubernetes/goog-testing Can one of you guys look this over and give it a thumb?

It's a manual backport because #17511 was already half backported and cherry-picking 6.5 PRs sounded like a PITA. No tests were changed, just harness scripts.

Full e2e shouldn't be affected/required, but I kicked it off for completeness.

k8s-bot · 2015-12-15T02:15:08Z

GCE e2e build/test failed for commit ffa5655.

karlkfi · 2015-12-15T18:16:44Z

Ran master with CONFORMANCE_BRANCH=pull/18686/head again. Looks like 2/3 of the errors were flakes. This time, only the expected SchedulerPredicates test failed (since then, #18348 has been merged).

[18:45:49][Step 7/7] Summarizing 1 Failure:
[18:45:49][Step 7/7] 
[18:45:49][Step 7/7] [Fail] SchedulerPredicates [It] validates resource limits of pods that are allowed to run [Conformance] 
[18:45:49][Step 7/7] /go/src/github.com/GoogleCloudPlatform/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:139
[18:45:49][Step 7/7] 
[18:45:49][Step 7/7] Ran 79 of 206 Specs in 2792.012 seconds
[18:45:49][Step 7/7] FAIL! -- 78 Passed | 1 Failed | 1 Pending | 126 Skipped Dec 15 02:45:49.040: INFO: >>> testContext.KubeConfig: /root/.kube/config
[18:45:49][Step 7/7] 
[18:45:49][Step 7/7] Skipping SSH core dump, which is not implemented for --- FAIL: TestE2E (2802.05s)
[18:45:49][Step 7/7] FAIL
[18:45:49][Step 7/7] 
[18:45:49][Step 7/7] Ginkgo ran 1 suite in 46m42.552753932s
[18:45:49][Step 7/7] Test Suite Failed

karlkfi · 2015-12-15T18:33:25Z

A rerun of the conformance tests on this PR shows 2/3 failures were flakes, and 2 more flakes popped up. SchedulerPredicates is the only consistently failing test. We'll likely cherry-pick #18348

[19:22:02][Step 7/7] Summarizing 3 Failures:
[19:22:02][Step 7/7] 
[19:22:02][Step 7/7] [Fail] SchedulerPredicates [It] validates resource limits of pods that are allowed to run [Conformance] 
[19:22:02][Step 7/7] /go/src/github.com/GoogleCloudPlatform/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:87
[19:22:02][Step 7/7] 
[19:22:02][Step 7/7] [Fail] Examples e2e [Example]ClusterDns [It] should create pod that uses dns [Conformance] 
[19:22:02][Step 7/7] /go/src/github.com/GoogleCloudPlatform/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/examples.go:505
[19:22:02][Step 7/7] 
[19:22:02][Step 7/7] [Fail] Docker Containers [It] should be able to override the image's default commmand (docker entrypoint) [Conformance] 
[19:22:02][Step 7/7] /go/src/github.com/GoogleCloudPlatform/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/util.go:1173
[19:22:02][Step 7/7] 
[19:22:02][Step 7/7] Ran 72 of 175 Specs in 4427.685 seconds
[19:22:02][Step 7/7] FAIL! -- 69 Passed | 3 Failed | 2 Pending | 101 Skipped >>> testContext.KubeConfig: /root/.kube/config
[19:22:02][Step 7/7] Skipping SSH core dump, which is not implemented for --- FAIL: TestE2E (4427.72s)
[19:22:02][Step 7/7] FAIL
[19:22:02][Step 7/7] 
[19:22:02][Step 7/7] Ginkgo ran 1 suite in 1h13m48.114844558s
[19:22:02][Step 7/7] Test Suite Failed

karlkfi · 2015-12-15T19:00:15Z

From Google's Jenkins, here's the e2e failure. Either it's a flake or release-1.1 doesn't pass conformance. Will re-run to check.

Summarizing 1 Failure:

[Fail] Services [It] should serve multiport endpoints from pods [Conformance] 
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:1048

Ran 110 of 175 Specs in 1047.911 seconds
FAIL! -- 109 Passed | 1 Failed | 2 Pending | 63 Skipped 

Ginkgo ran 1 suite in 17m36.672718671s
Test Suite Failed

karlkfi · 2015-12-15T19:00:22Z

@k8s-bot e2e test this

k8s-bot · 2015-12-15T19:15:20Z

GCE e2e build/test failed for commit ffa5655.

karlkfi · 2015-12-15T19:22:00Z

Jenkins GCE seems to be hosed.

ERROR: (gcloud.compute.firewall-rules.create) Some requests did not succeed:
 - Quota 'FIREWALLS' exceeded.  Limit: 100.0

karlkfi · 2015-12-15T19:28:55Z

Ran master with CONFORMANCE_BRANCH=pull/18686/head again, now that #18348 is merged.

100% conformance!

[11:11:21][Step 7/7] Ran 79 of 206 Specs in 2906.263 seconds
[11:11:21][Step 7/7] SUCCESS! -- 79 Passed | 0 Failed | 1 Pending | 126 Skipped Dec 15 19:11:21.932: INFO: >>> testContext.KubeConfig: /root/.kube/config
[11:11:21][Step 7/7] 
[11:11:21][Step 7/7] Skipping SSH core dump, which is not implemented for PASS
[11:11:21][Step 7/7] 
[11:11:21][Step 7/7] Ginkgo ran 1 suite in 48m36.832110792s
[11:11:21][Step 7/7] Test Suite Passed

karlkfi · 2015-12-15T22:45:36Z

@k8s-bot e2e test this

k8s-bot · 2015-12-15T22:54:49Z

GCE e2e build/test failed for commit ffa5655.

karlkfi · 2015-12-15T23:07:03Z

Same problem. It appears google's quota is filled, and I can't get ahold of an on-call to fix it.

@kubernetes/goog-testing You guys have any control over this?

ERROR: (gcloud.compute.firewall-rules.create) Some requests did not succeed:
 - Quota 'FIREWALLS' exceeded.  Limit: 100.0

karlkfi · 2015-12-16T22:19:49Z

@k8s-bot e2e test this

k8s-bot · 2015-12-16T23:22:52Z

GCE e2e test build/test passed for commit ffa5655.

lavalamp · 2016-01-04T18:05:11Z

Sorry for delay, I've been out of the office. @ixdy, can you take a look or find an owner?

ikehz · 2016-01-04T18:46:50Z

I'll have a look.

ikehz · 2016-01-04T18:50:20Z

LGTM.

k8s-github-robot · 2016-01-04T18:59:21Z

@k8s-bot test this

Tests are more than 48 hours old. Re-running tests.

k8s-bot · 2016-01-04T19:36:43Z

GCE e2e test build/test passed for commit ffa5655.

k8s-github-robot · 2016-01-04T23:30:36Z

@k8s-bot test this [submit-queue is verifying that this PR is safe to merge]

k8s-bot · 2016-01-05T00:03:27Z

GCE e2e test build/test passed for commit ffa5655.

k8s-github-robot · 2016-01-05T00:03:47Z

Automatic merge from submit-queue

Auto commit by PR queue bot

…-ci-backport Auto commit by PR queue bot

Karl Isenberg added 2 commits December 14, 2015 14:37

Backport contrib/mesos/ci to release-1.1

1e35260

karlkfi added area/test area/test-infra team/mesosphere labels Dec 14, 2015

karlkfi self-assigned this Dec 14, 2015

googlebot added the cla: yes label Dec 14, 2015

karlkfi mentioned this pull request Dec 14, 2015

Backport e2e tag/harness changes to release-1.1 mesosphere/kubernetes-mesos#659

Closed

k8s-github-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Dec 14, 2015

karlkfi changed the title ~~WIP Backport smoke/conformance test harness changes to release-1.1~~ Backport smoke/conformance test harness changes to release-1.1 Dec 15, 2015

karlkfi removed the e2e-not-required label Dec 15, 2015

karlkfi removed their assignment Dec 15, 2015

karlkfi changed the title ~~Backport smoke/conformance test harness changes to release-1.1~~ Backport smoke/conformance test harness changes to release-1.1 from master Dec 15, 2015

k8s-github-robot assigned lavalamp Dec 15, 2015

karlkfi mentioned this pull request Dec 15, 2015

Modify conformance test policy to use tests from release branches #18162

Closed

15 tasks

karlkfi mentioned this pull request Dec 15, 2015

Fix SchedulerPredicates conformance test on release-1.1 branch mesosphere/kubernetes-mesos#704

Open

lavalamp assigned ixdy and unassigned lavalamp Jan 4, 2016

ikehz assigned ikehz and unassigned ixdy Jan 4, 2016

ikehz added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jan 4, 2016

k8s-github-robot pushed a commit that referenced this pull request Jan 5, 2016

Merge pull request #18686 from mesosphere/release-1.1-mesos-ci-backport

b9dfea3

Auto commit by PR queue bot

k8s-github-robot merged commit b9dfea3 into kubernetes:release-1.1 Jan 5, 2016

karlkfi deleted the release-1.1-mesos-ci-backport branch January 5, 2016 00:25

shyamjvs pushed a commit to shyamjvs/kubernetes that referenced this pull request Dec 1, 2016

Merge pull request kubernetes#18686 from mesosphere/release-1.1-mesos…

7f62056

…-ci-backport Auto commit by PR queue bot

shouhong pushed a commit to shouhong/kubernetes that referenced this pull request Feb 14, 2017

Merge pull request kubernetes#18686 from mesosphere/release-1.1-mesos…

66be470

…-ci-backport Auto commit by PR queue bot

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Backport smoke/conformance test harness changes to release-1.1 from master #18686

Backport smoke/conformance test harness changes to release-1.1 from master #18686

karlkfi commented Dec 14, 2015

k8s-github-robot commented Dec 14, 2015

karlkfi commented Dec 14, 2015

k8s-bot commented Dec 14, 2015

karlkfi commented Dec 15, 2015

karlkfi commented Dec 15, 2015

karlkfi commented Dec 15, 2015

k8s-bot commented Dec 15, 2015

karlkfi commented Dec 15, 2015

karlkfi commented Dec 15, 2015

karlkfi commented Dec 15, 2015

karlkfi commented Dec 15, 2015

k8s-bot commented Dec 15, 2015

karlkfi commented Dec 15, 2015

karlkfi commented Dec 15, 2015

karlkfi commented Dec 15, 2015

k8s-bot commented Dec 15, 2015

karlkfi commented Dec 15, 2015

karlkfi commented Dec 16, 2015

k8s-bot commented Dec 16, 2015

lavalamp commented Jan 4, 2016

ikehz commented Jan 4, 2016

ikehz commented Jan 4, 2016

k8s-github-robot commented Jan 4, 2016

k8s-bot commented Jan 4, 2016

k8s-github-robot commented Jan 4, 2016

k8s-bot commented Jan 5, 2016

k8s-github-robot commented Jan 5, 2016

Backport smoke/conformance test harness changes to release-1.1 from master #18686

Backport smoke/conformance test harness changes to release-1.1 from master #18686

Conversation

karlkfi commented Dec 14, 2015

k8s-github-robot commented Dec 14, 2015

karlkfi commented Dec 14, 2015

k8s-bot commented Dec 14, 2015

karlkfi commented Dec 15, 2015

karlkfi commented Dec 15, 2015

karlkfi commented Dec 15, 2015

k8s-bot commented Dec 15, 2015

karlkfi commented Dec 15, 2015

karlkfi commented Dec 15, 2015

karlkfi commented Dec 15, 2015

karlkfi commented Dec 15, 2015

k8s-bot commented Dec 15, 2015

karlkfi commented Dec 15, 2015

karlkfi commented Dec 15, 2015

karlkfi commented Dec 15, 2015

k8s-bot commented Dec 15, 2015

karlkfi commented Dec 15, 2015

karlkfi commented Dec 16, 2015

k8s-bot commented Dec 16, 2015

lavalamp commented Jan 4, 2016

ikehz commented Jan 4, 2016

ikehz commented Jan 4, 2016

k8s-github-robot commented Jan 4, 2016

k8s-bot commented Jan 4, 2016

k8s-github-robot commented Jan 4, 2016

k8s-bot commented Jan 5, 2016

k8s-github-robot commented Jan 5, 2016