Update conformance test policy

contrib/mesos/ci/test-conformance.sh

@@ -31,7 +31,7 @@ TEST_ARGS="$@"
 
 KUBE_ROOT=$(cd "$(dirname "${BASH_SOURCE}")/../../.." && pwd)
 
-TEST_CMD="KUBECONFIG=~/.kube/config hack/conformance-test.sh"
+TEST_CMD="KUBERNETES_CONFORMANCE_TEST=y KUBECONFIG=~/.kube/config go run hack/e2e.go --test --test_args=\"--ginkgo.focus=\\[Conformance\\]\""
 if [ -n "${CONFORMANCE_BRANCH}" ]; then
 	# create a CONFORMANCE_BRANCH clone in a subdirectory
 	TEST_CMD="

docs/devel/e2e-tests.md

@@ -77,12 +77,15 @@ go run hack/e2e.go -v --pushup
 # Run all tests
 go run hack/e2e.go -v --test
 
-# Run tests matching the regex "\[Conformance\]" (the conformance tests)
-go run hack/e2e.go -v -test --test_args="--ginkgo.focus=\[Conformance\]"
+# Run tests matching the regex "\[Feature:Performance\]"
+go run hack/e2e.go -v -test --test_args="--ginkgo.focus=\[Feature:Performance\]"
 
 # Conversely, exclude tests that match the regex "Pods.*env"
 go run hack/e2e.go -v -test --test_args="--ginkgo.focus=Pods.*env"
 
+# Run tests in parallel, skip any that must be run serially
+GINKGO_PARALLEL=y go run hack/e2e.go --v --test --test_args="--ginkgo.skip=\[Serial\]"
+
 # Flags can be combined, and their actions will take place in this order:
 # --build, --push|--up|--pushup, --test|--tests=..., --down
 #
@@ -96,9 +99,6 @@ KUBERNETES_PROVIDER=aws go run hack/e2e.go -v --build --pushup --test --down
 # kubectl output.
 go run hack/e2e.go -v -ctl='get events'
 go run hack/e2e.go -v -ctl='delete pod foobar'
-
-# Alternately, if you have the e2e cluster up and no desire to see the event stream, you can run ginkgo-e2e.sh directly:
-hack/ginkgo-e2e.sh --ginkgo.focus=\[Conformance\]
 ```
 
 The tests are built into a single binary which can be run used to deploy a Kubernetes system or run tests against an already-deployed Kubernetes system.  See `go run hack/e2e.go --help` (or the flag definitions in `hack/e2e.go`) for more options, such as reusing an existing cluster.
@@ -208,13 +208,31 @@ We are working on implementing clearer partitioning of our e2e tests to make run
 
 ### Conformance tests
 
-Finally, `[Conformance]` tests are tests we expect to pass on **any** Kubernetes cluster.  The `[Conformance]` label does not supersede any other labels.  `[Conformance]` test policies are a work-in-progress (see #18162).
+Finally, `[Conformance]` tests represent a subset of the e2e-tests we expect to pass on **any** Kubernetes cluster.  The `[Conformance]` label does not supersede any other labels.
+
+As each new release of Kubernetes providers new functionality, the subset of tests necessary to demonstrate conformance grows with each release.  Conformance is thus considered versioned, with the same backwards compatibility guarantees as laid out in [our versioning policy](../design/versioning.md#supported-releases).  Conformance tests for a given version should be run off of the release branch that corresponds to that version.  Thus `v1.2` conformance tests would be run from the head of the `release-1.2` branch. eg:
 
-End-to-end testing, as described above, is for [development distributions](writing-a-getting-started-guide.md).  A conformance test is used on a [versioned distro](writing-a-getting-started-guide.md).  (Links WIP)
+ - A v1.3 development cluster should pass v1.1, v1.2 conformance tests
+ - A v1.2 cluster should pass v1.1, v1.2 conformance tests
+ - A v1.1 cluster should pass v1.0, v1.1 conformance tests, and fail v1.2 conformance tests
 
-The conformance test runs a subset of the e2e-tests against a manually-created cluster.  It does not require support for up/push/down and other operations.  To run a conformance test, you need to know the IP of the master for your cluster and the authorization arguments to use.  The conformance test is intended to run against a cluster at a specific binary release of Kubernetes.  See [conformance-test.sh](http://releases.k8s.io/HEAD/hack/conformance-test.sh).
+Conformance tests are designed to be run with no cloud provider configured.  Conformance tests can be run against clusters that have not been created with `hack/e2e.go`, just provide a kubeconfig with the appropriate endpoint and credentials.
+
+```sh
+# setup for conformance tests
+export KUBECONFIG=/path/to/kubeconfig
+export KUBERNETES_CONFORMANCE_TEST=y
+
+# run all conformance tests
+go run hack/e2e.go -v --test_args="--ginkgo.focus=\[Conformance\]"
+
+# run all parallel-safe conformance tests in parallel
+GINKGO_PARALLEL=y go run hack/e2e.go --v --test --test_args="--ginkgo.focus=\[Conformance\] --ginkgo.skip=\[Serial\]"
+# ... and finish up with remaining tests in serial
+go run hack/e2e.go --v --test --test_args="--ginkgo.focus=\[Serial\].*\[Conformance\]"
+```
 
-### Defining what Conformance means
+### Defining Conformance Subset
 
 It is impossible to define the entire space of Conformance tests without knowing the future, so instead, we define the compliment of conformance tests, below.
 

docs/devel/testing.md

@@ -174,55 +174,7 @@ hack/test-integration.sh  # Run all integration tests.
 
 ## End-to-End tests
 
-* e2e tests build kubernetes and deploy a cluster of nodes.
-  - Generally on a specific cloud provider.
-* Access gcr.io images
-* Access a specific, non-latest image tag (unless testing pulling).
-* Tests may not flake due to intermittent issues.
-* Use ginko to desribe steps.
-  - See [should run a job to completion when tasks succeed](../../test/e2e/job.go)
-* Use [NewDefaultFramework](../../test/e2e/framework.go)
-  - Contains clients, namespace and auto resource cleanup
-* See [coding conventions](coding-conventions.md).
-
-### e2e test philosophy
-
-In general passing unit and integration tests should provide sufficient
-confidence to allow code to merge.  If that is not the case,
-please *invest more time adding unit and integration test coverage*.
-These tests run faster and have a smaller failure domain.
-
-However, end-to-end (e2e) tests provide maximum confidence that
-the system is working in exchange for reduced performance and a
-higher debugging cost.
-
-e2e tests deploy a real kubernetes cluster of real nodes on a concrete provider
-such as GCE. The tests then manipulate the cluster in certain ways and
-assert the expected results.
-
-For a more in depth discussion please read [End-to-End Testing in Kubernetes](e2e-tests.md).
-
-### Running e2e tests
-
-```sh
-cd kubernetes
-go run hack/e2e.go -v --build --up --test --down
-
-# Change code, run unit and integration tests
-# Push to an existing cluster, or bring up a cluster if it's down.
-go run hack/e2e.go -v --pushup
-
-# Run all tests on an already up cluster
-go run hack/e2e.go -v --test
-
-# Run only conformance tests
-go run hack/e2e.go -v -test --test_args="--ginkgo.focus=\[Conformance\]"
-
-# Run tests on a specific provider
-KUBERNETES_PROVIDER=aws go run hack/e2e.go --build --pushup --test --down
-```
-
-For a more in depth discussion please read [End-to-End Testing in Kubernetes](e2e-tests.md).
+Please refer to [End-to-End Testing in Kubernetes](e2e-tests.md).
 
 <!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
 [![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/devel/testing.md?pixel)]()

docs/devel/writing-a-getting-started-guide.md

@@ -68,7 +68,7 @@ These guidelines say *what* to do.  See the Rationale section for *why*.
    own repo.
  - Add or update a row in [The Matrix](../../docs/getting-started-guides/README.md).
  - State the binary version of Kubernetes that you tested clearly in your Guide doc.
- - Setup a cluster and run the [conformance test](development.md#conformance-testing) against it, and report the
+ - Setup a cluster and run the [conformance tests](e2e-tests.md#conformance-tests) against it, and report the
    results in your PR.
  - Versioned distros should typically not modify or add code in `cluster/`.  That is just scripts for developer
    distros.