e2e tag cleanup #17180

karlkfi · 2015-11-12T22:40:06Z

Skip [Skipped] tests during conformance testing
Remove "Suite" from e2e tag names
Move group type e2e tags to the front of names, and to as high a context as applicable
Move Conformance e2e tags to the end of test names (test specific)
Move Skipped e2e tags to the end of names, but to as high a context as applicable
Move ClusterDns e2e test to its own file (because it's not skipped like the others)
Group FSGroup e2e tests

k8s-github-robot · 2015-11-12T22:40:09Z

Labelling this PR as size/XL

karlkfi · 2015-11-12T22:44:51Z

No new tests here, just refactored existing tests, mostly their names and contexts/describes.

k8s-bot · 2015-11-12T23:22:12Z

GCE e2e test build/test passed for commit da94bdbb22a5613a3e26a47d18049b7711d9d54b.

karlkfi · 2015-11-12T23:35:22Z

Blocked by #17054

ghost · 2015-11-12T23:47:27Z

Hurrah! Thanks for taking on this much-needed tidy-up @karlkfi

jdef · 2015-11-13T00:18:05Z

👍 ditto -- glad to see the cleanup

sttts · 2015-11-13T09:25:18Z

test/e2e/cluster_size_autoscaling.go

@@ -34,7 +34,7 @@ const (
 	scaleDownTimeout = 30 * time.Minute
 )

-var _ = Describe("Autoscaling", func() {
+var _ = Describe("[Autoscaling] [Skipped]", func() {


Why not "Autoscaling" without the quotes?

The [Autoscaling] tag used in other tests, not just this describe block. So I wanted to be consistent in case people filter with the brackets. It's a subtle difference between a group tag used for filtering and a describe/context block just used for abstraction.

karlkfi · 2015-11-13T18:17:21Z

Rebased to get fix from #17054

@quinton-hoole you got a second to officially review this? Need a googler sign off.

k8s-bot · 2015-11-13T18:48:35Z

GCE e2e test build/test passed for commit 9af8a01a819ad05aefe09a5d3946206782369392.

ixdy · 2015-11-13T23:23:46Z

test/e2e/density.go

@@ -162,21 +162,18 @@ var _ = Describe("Density", func() {
 		// (metrics from other tests affects this one).
 		// TODO: Reenable once we can measure latency only from a single test.


it seems like we should either remove this TODO or leave the option to make these tests unskipped by default? @fgrzadkowski

They were all being skipped. So I removed the functionality to skip them individually and skipped the whole block.
You can still unskip them by specifying them as a group or individually.

karlkfi · 2015-11-14T19:12:44Z

Once this PR is merged, the Mesos/Docker cluster will be 100% conformant on master.
The 4 FSGroup tests are the only ones still failing: https://teamcity.mesosphere.io/viewLog.html?buildId=75719&tab=buildResultsDiv&buildTypeId=Oss_KubernetesMesos_ConformanceTests&guest=1

karlkfi · 2015-11-14T21:24:48Z

Conformance has passed on the pull/17180/merge branch: https://teamcity.mesosphere.io/viewLog.html?buildId=75866&tab=buildResultsDiv&buildTypeId=Oss_KubernetesMesos_ConformanceTests&guest=1

karlkfi · 2015-11-17T19:24:23Z

Rebased to resolve conflicts:

Horizontal pod autoscaling for deployments is now marked [Autoscaling]
Example tests now use runKubectlOrDie instead of runKubectl

k8s-bot · 2015-11-17T19:32:23Z

GCE e2e build/test failed for commit cb928f1e114ec00dbb560b9fd8153c31f2940ae1.

k8s-bot · 2015-11-17T20:13:11Z

GCE e2e test build/test passed for commit 7d7fb77367e5853ae8221270714c9275dcf44af1.

k8s-bot · 2015-11-17T22:13:36Z

GCE e2e build/test failed for commit f597fb9540e0073b5b6d5fe5ee3b86609cbe0bb5.

karlkfi · 2015-11-17T22:30:33Z

That e2e is very not happy... a million timed out waiting for the condition...

ixdy · 2015-11-18T00:06:50Z

lgtm, please squash

piosz · 2015-11-24T08:50:38Z

@karlkfi @ixdy
This PR disabled some important tests in Autoscaling suite. This change may also break some other suites (please check it). Please be really, really careful when merging such change.

wojtek-t · 2015-11-24T08:53:31Z

This PR also change what is running in scalability suite. I'm fixing it in #17714

+1000 to what @piosz said - we don't want to spend hours on fixing problems, especially, if those are cleanup changes that doesn't introduce any new features...

karlkfi · 2015-11-24T17:10:49Z

@wojtek-t As you can see from the change log of this PR, the Latency test was already marked as a part of the [Performance Suite]. It's not new. The only difference was that it renamed the tags to remove the "Suite", and then renamed the focus/skip ginkgo flags to use the full tag with \[Performance\]. How did that change what was running?

karlkfi · 2015-11-24T17:17:23Z

@piosz That looks like an oversight on my part when I rebased to include the Horizontal pod autoscaling for deployments as well as for replication controllers (mentioned in #17180 (comment)). It must have gotten reverted in one of the subsequent rebases, and then the record lost in a squash. However, your fix is incomplete. The Deployment describe block should now also be marked [Autoscaling], which means that actually the parent describe block should have the tag.

wojtek-t · 2015-11-24T17:53:07Z

@karlkfi - there must have been some configuration detail, because the latency test was NOT running in our scalability suite

ikehz · 2015-11-24T18:18:18Z

We're also running into problems with porting this to the release-1.1 branch. Our tests on the release branch are now confused because not everything is properly labeled. @karlkfi, can you cherry pick it, (use hack/cherry_pick_pull.sh,) and resolving conflicts?

cc @kubernetes/goog-testing

karlkfi · 2015-11-24T18:22:24Z

@ihmccreery are you running the master:head test scripts against the release branch? Why not use the scripts from the branch you're testing?

ikehz · 2015-11-24T18:48:42Z

Yes, we run the master hack/jenkins/e2e.sh config script on master, but do run tests against old branches. hack/jenkins/e2e.sh is supposed to be version agnostic, but isn't really, especially with changes like this that redefine how we run tests.

ixdy · 2015-11-24T19:53:35Z

OK, so is someone owning fixing the release-1.1 branch? I'm not sure if that's something to discuss in #17734 or elsewhere.

Auditing tests right now on kubernetes-e2e-gce-release-1.1. We're now running the following tests (which we weren't before):

Skipped Cluster upgrade kube-push of master should maintain responsive services
Skipped Cluster upgrade upgrade-cluster should maintain a functioning cluster
Skipped Cluster upgrade upgrade-master should maintain responsive services

Unsurprisingly, this is breaking a lot of the other tests.

ixdy · 2015-11-24T20:01:24Z

Good news is that the builds running against master all appear to be working correctly.

karlkfi · 2015-11-24T20:02:28Z

Well, I guess it's a good thing this came up, even if it did so in a somewhat painful way. Running unversioned test scripts against versioned tests is a good way to not be able to make any changes... I was unaware that was happening and didn't really have any way of knowing without access to your jenkins.

The obvious solution is to run the tests on the branch you're testing. We're doing this in the Mesosphere CI, with some exception code that can override the CI scripts with ones from a specific repo/branch in case we need to override the scripts from another branch.

Is there some reason the google jenkins can't do this?

ixdy · 2015-11-24T20:04:27Z

Let's take this discussion to #17734 perhaps?

ixdy · 2015-11-24T20:17:45Z

I just created #17735 and #17736 to fix the "Skipped" tag on release-1.0 and release-1.1, which should fix our issues with kubernetes-e2e-gce-release-1.1. It's not a full cherrypick of this PR because I don't want to pull in all of the other changes.

- Skip [Skipped] tests when testing for conformance kubernetes#17180 - Add cluster/test-conformance.sh script kubernetes#17997 - Support passing args to cluster/test-smoke.sh kubernetes#15404 - Support annotated tags in conformance test version detection kubernetes#17839

googlebot added the cla: yes label Nov 12, 2015

k8s-github-robot assigned ixdy Nov 12, 2015

k8s-github-robot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Nov 12, 2015

karlkfi added area/test area/test-infra team/mesosphere labels Nov 12, 2015

karlkfi mentioned this pull request Nov 12, 2015

Conformance Tests should not run [Skipped] tests mesosphere/kubernetes-mesos#623

Closed

sttts reviewed Nov 13, 2015
View reviewed changes

karlkfi force-pushed the skipped-cleanup branch from da94bdb to 9af8a01 Compare November 13, 2015 18:16

ixdy reviewed Nov 13, 2015
View reviewed changes

k8s-github-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 17, 2015

karlkfi force-pushed the skipped-cleanup branch from 9af8a01 to cb928f1 Compare November 17, 2015 19:23

k8s-github-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 17, 2015

karlkfi force-pushed the skipped-cleanup branch from cb928f1 to 7d7fb77 Compare November 17, 2015 19:44

sttts deleted the skipped-cleanup branch November 24, 2015 08:44

wojtek-t mentioned this pull request Nov 24, 2015

Do not run Latency test as part of scalability suite #17714

Merged

piosz mentioned this pull request Nov 24, 2015

Re-enabled HPA e2e in autoscaling suite #17716

Merged

ikehz mentioned this pull request Nov 24, 2015

Make hack/jenkins/e2e.sh versioned with releases #17734

Closed

This was referenced Nov 24, 2015

Fix skipped tag for cluster upgrade tests in release-1.1 #17735

Closed

Fix skipped tag for cluster upgrade tests in release-1.0 #17736

Closed

This was referenced Nov 25, 2015

Backport hack/jenkins/e2e.sh to release-1.0 #17749

Merged

Backport hack/jenkins/e2e.sh to release-1.1 #17750

Merged

Remove all non-release job definitions #17751

Merged

This was referenced Nov 30, 2015

Skipped e2e patch for v0.7-v1.1 branch mesosphere/kubernetes-mesos#650

Closed

Backport e2e tag/harness changes to release-1.1 mesosphere/kubernetes-mesos#659

Closed

downward API support is broken mesosphere/kubernetes-mesos#654

Closed

jdef mentioned this pull request Dec 9, 2015

AS a Quality Engineer I WANT to have clarity regarding which test script harness is running the unit/integration/smoke/conformance/end-to-end tests for a particular test run. mesosphere/kubernetes-mesos#684

Open

karlkfi mentioned this pull request Dec 14, 2015

Backport smoke/conformance test harness changes to release-1.1 from master #18686

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

e2e tag cleanup #17180

e2e tag cleanup #17180

karlkfi commented Nov 12, 2015

k8s-github-robot commented Nov 12, 2015

karlkfi commented Nov 12, 2015

k8s-bot commented Nov 12, 2015

karlkfi commented Nov 12, 2015

ghost commented Nov 12, 2015

jdef commented Nov 13, 2015

sttts Nov 13, 2015

karlkfi Nov 13, 2015

karlkfi commented Nov 13, 2015

k8s-bot commented Nov 13, 2015

ixdy Nov 13, 2015

karlkfi Nov 17, 2015

karlkfi commented Nov 14, 2015

karlkfi commented Nov 14, 2015

karlkfi commented Nov 17, 2015

k8s-bot commented Nov 17, 2015

k8s-bot commented Nov 17, 2015

k8s-bot commented Nov 17, 2015

karlkfi commented Nov 17, 2015

ixdy commented Nov 18, 2015

piosz commented Nov 24, 2015

wojtek-t commented Nov 24, 2015

karlkfi commented Nov 24, 2015

karlkfi commented Nov 24, 2015

wojtek-t commented Nov 24, 2015

ikehz commented Nov 24, 2015

karlkfi commented Nov 24, 2015

ikehz commented Nov 24, 2015

ixdy commented Nov 24, 2015

ixdy commented Nov 24, 2015

karlkfi commented Nov 24, 2015

ixdy commented Nov 24, 2015

ixdy commented Nov 24, 2015

		@@ -162,21 +162,18 @@ var _ = Describe("Density", func() {
		// (metrics from other tests affects this one).
		// TODO: Reenable once we can measure latency only from a single test.

e2e tag cleanup #17180

e2e tag cleanup #17180

Conversation

karlkfi commented Nov 12, 2015

k8s-github-robot commented Nov 12, 2015

karlkfi commented Nov 12, 2015

k8s-bot commented Nov 12, 2015

karlkfi commented Nov 12, 2015

ghost commented Nov 12, 2015

jdef commented Nov 13, 2015

sttts Nov 13, 2015

Choose a reason for hiding this comment

karlkfi Nov 13, 2015

Choose a reason for hiding this comment

karlkfi commented Nov 13, 2015

k8s-bot commented Nov 13, 2015

ixdy Nov 13, 2015

Choose a reason for hiding this comment

karlkfi Nov 17, 2015

Choose a reason for hiding this comment

karlkfi commented Nov 14, 2015

karlkfi commented Nov 14, 2015

karlkfi commented Nov 17, 2015

k8s-bot commented Nov 17, 2015

k8s-bot commented Nov 17, 2015

k8s-bot commented Nov 17, 2015

karlkfi commented Nov 17, 2015

ixdy commented Nov 18, 2015

piosz commented Nov 24, 2015

wojtek-t commented Nov 24, 2015

karlkfi commented Nov 24, 2015

karlkfi commented Nov 24, 2015

wojtek-t commented Nov 24, 2015

ikehz commented Nov 24, 2015

karlkfi commented Nov 24, 2015

ikehz commented Nov 24, 2015

ixdy commented Nov 24, 2015

ixdy commented Nov 24, 2015

karlkfi commented Nov 24, 2015

ixdy commented Nov 24, 2015

ixdy commented Nov 24, 2015