Skip to content

Commit

Permalink
Reconcile testing docs, fixes #18606
Browse files Browse the repository at this point in the history
  • Loading branch information
Isaac Hollander McCreery committed Feb 11, 2016
1 parent 11c42f7 commit c25c7c7
Show file tree
Hide file tree
Showing 2 changed files with 106 additions and 128 deletions.
104 changes: 1 addition & 103 deletions docs/devel/development.md
View file
Expand Up @@ -294,109 +294,7 @@ hack/test-integration.sh

## End-to-End tests

You can run an end-to-end test which will bring up a master and two nodes, perform some tests, and then tear everything down. Make sure you have followed the getting started steps for your chosen cloud platform (which might involve changing the `KUBERNETES_PROVIDER` environment variable to something other than "gce".

```sh
cd kubernetes
hack/e2e-test.sh
```

Pressing control-C should result in an orderly shutdown but if something goes wrong and you still have some VMs running you can force a cleanup with this command:

```sh
go run hack/e2e.go --down
```

### Flag options

See the flag definitions in `hack/e2e.go` for more options, such as reusing an existing cluster, here is an overview:

```sh
# Build binaries for testing
go run hack/e2e.go --build

# Create a fresh cluster. Deletes a cluster first, if it exists
go run hack/e2e.go --up

# Create a fresh cluster at a specific release version.
go run hack/e2e.go --up --version=0.7.0

# Test if a cluster is up.
go run hack/e2e.go --isup

# Push code to an existing cluster
go run hack/e2e.go --push

# Push to an existing cluster, or bring up a cluster if it's down.
go run hack/e2e.go --pushup

# Run all tests
go run hack/e2e.go --test

# Run tests matching the regex "Pods.*env"
go run hack/e2e.go -v -test --test_args="--ginkgo.focus=Pods.*env"

# Alternately, if you have the e2e cluster up and no desire to see the event stream, you can run ginkgo-e2e.sh directly:
hack/ginkgo-e2e.sh --ginkgo.focus=Pods.*env
```

### Combining flags

```sh
# Flags can be combined, and their actions will take place in this order:
# -build, -push|-up|-pushup, -test|-tests=..., -down
# e.g.:
go run hack/e2e.go -build -pushup -test -down

# -v (verbose) can be added if you want streaming output instead of only
# seeing the output of failed commands.

# -ctl can be used to quickly call kubectl against your e2e cluster. Useful for
# cleaning up after a failed test or viewing logs. Use -v to avoid suppressing
# kubectl output.
go run hack/e2e.go -v -ctl='get events'
go run hack/e2e.go -v -ctl='delete pod foobar'
```

## Local clusters

It can be much faster to iterate on a local cluster instead of a cloud-based one. To start a local cluster, you can run:

```sh
# The PATH construction is needed because PATH is one of the special-cased
# environment variables not passed by sudo -E
sudo PATH=$PATH hack/local-up-cluster.sh
```

This will start a single-node Kubernetes cluster than runs pods using the local docker daemon. Press Control-C to stop the cluster.

### E2E tests against local clusters

In order to run an E2E test against a locally running cluster, use the `e2e.test` binary built by `hack/build-go.sh`
directly:

```sh
export KUBECONFIG=/path/to/kubeconfig
e2e.test --host=http://127.0.0.1:8080
```

To control the tests that are run:

```sh
e2e.test --host=http://127.0.0.1:8080 --ginkgo.focus="Secrets"
```

## Conformance testing

End-to-end testing, as described above, is for [development
distributions](writing-a-getting-started-guide.md). A conformance test is used on
a [versioned distro](writing-a-getting-started-guide.md).

The conformance test runs a subset of the e2e-tests against a manually-created cluster. It does not
require support for up/push/down and other operations. To run a conformance test, you need to know the
IP of the master for your cluster and the authorization arguments to use. The conformance test is
intended to run against a cluster at a specific binary release of Kubernetes.
See [conformance-test.sh](http://releases.k8s.io/HEAD/hack/conformance-test.sh).
See [End-to-End Testing in Kubernetes](e2e-tests.md).

## Testing out flaky tests

Expand Down
130 changes: 105 additions & 25 deletions docs/devel/e2e-tests.md
View file
Expand Up @@ -32,33 +32,91 @@ Documentation for other releases can be found at

<!-- END MUNGE: UNVERSIONED_WARNING -->

# End-2-End Testing in Kubernetes
# End-to-End Testing in Kubernetes

## Overview

The end-2-end tests for kubernetes provide a mechanism to test behavior of the system, and to ensure end user operations match developer specifications. In distributed systems it is not uncommon that a minor change may pass all unit tests, but cause unforseen changes at the system level. Thus, the primary objectives of the end-2-end tests are to ensure a consistent and reliable behavior of the kubernetes code base, and to catch bugs early.
End-to-end (e2e) tests for Kubernetes provide a mechanism to test end-to-end behavior of the system, and is the last signal to ensure end user operations match developer specifications. Although unit and integration tests should ideally provide a good signal, the reality is in a distributed system like Kubernetes it is not uncommon that a minor change may pass all unit and integration tests, but cause unforseen changes at the system level. e2e testing is very costly, both in time to run tests and difficulty debugging, though: it takes a long time to build, deploy, and exercise a cluster. Thus, the primary objectives of the e2e tests are to ensure a consistent and reliable behavior of the kubernetes code base, and to catch hard-to-test bugs before users do, when unit and integration tests are insufficient.

The end-2-end tests in kubernetes are built atop of [ginkgo] (http://onsi.github.io/ginkgo/) and [gomega] (http://onsi.github.io/gomega/). There are a host of features that this BDD testing framework provides, and it is recommended that the developer read the documentation prior to diving into the tests.
The e2e tests in kubernetes are built atop of [Ginkgo](http://onsi.github.io/ginkgo/) and [Gomega](http://onsi.github.io/gomega/). There are a host of features that this BDD testing framework provides, and it is recommended that the developer read the documentation prior to diving into the tests.

The purpose of *this* document is to serve as a primer for developers who are looking to execute, or add tests, using a local development environment.
The purpose of *this* document is to serve as a primer for developers who are looking to execute or add tests using a local development environment.

## Building and Running the Tests

**NOTE:** The tests have an array of options. For simplicity, the examples will focus on leveraging the tests on a local cluster using `sudo ./hack/local-up-cluster.sh`
There are a variety of ways to run e2e tests, but we aim to decrease the number of ways to run e2e tests to a canonical way: `hack/e2e.go`.

### Building the Tests
You can run an end-to-end test which will bring up a master and nodes, perform some tests, and then tear everything down. Make sure you have followed the getting started steps for your chosen cloud platform (which might involve changing the `KUBERNETES_PROVIDER` environment variable to something other than "gce").

The tests are built into a single binary which can be run against any deployed kubernetes system. To build the tests, navigate to your source directory and execute:
To build Kubernetes, up a cluster, run tests, and tear everything down, use:

`$ make all`
```sh
go run hack/e2e.go -v --build --up --test --down
```

If you'd like to just perform one of these steps, here are some examples:

```sh
# Build binaries for testing
go run hack/e2e.go -v --build

# Create a fresh cluster. Deletes a cluster first, if it exists
go run hack/e2e.go -v --up

# Create a fresh cluster at a specific release version.
go run hack/e2e.go -v --up --version=0.7.0

# Test if a cluster is up.
go run hack/e2e.go -v --isup

# Push code to an existing cluster
go run hack/e2e.go -v --push

# Push to an existing cluster, or bring up a cluster if it's down.
go run hack/e2e.go -v --pushup

# Run all tests
go run hack/e2e.go -v --test

The output for the end-2-end tests will be a single binary called `e2e.test` under the default output directory, which is typically `_output/local/bin/linux/amd64/`. Within the repository there are scripts that are provided under the `./hack` directory that are helpful for automation, but may not apply for a local development purposes. Instead, we recommend familiarizing yourself with the executable options. To obtain the full list of options, run the following:
# Run tests matching the regex "\[Conformance\]" (the conformance tests)
go run hack/e2e.go -v -test --test_args="--ginkgo.focus=\[Conformance\]"

`$ ./e2e.test --help`
# Conversely, exclude tests that match the regex "Pods.*env"
go run hack/e2e.go -v -test --test_args="--ginkgo.focus=Pods.*env"

### Running the Tests
# Flags can be combined, and their actions will take place in this order:
# --build, --push|--up|--pushup, --test|--tests=..., --down
#
# You can also specify an alternative provider, such as 'aws'
#
# e.g.:
KUBERNETES_PROVIDER=aws go run hack/e2e.go -v --build --pushup --test --down

For the purposes of brevity, we will look at a subset of the options, which are listed below:
# -ctl can be used to quickly call kubectl against your e2e cluster. Useful for
# cleaning up after a failed test or viewing logs. Use -v to avoid suppressing
# kubectl output.
go run hack/e2e.go -v -ctl='get events'
go run hack/e2e.go -v -ctl='delete pod foobar'

# Alternately, if you have the e2e cluster up and no desire to see the event stream, you can run ginkgo-e2e.sh directly:
hack/ginkgo-e2e.sh --ginkgo.focus=\[Conformance\]
```

The tests are built into a single binary which can be run used to deploy a Kubernetes system or run tests against an already-deployed Kubernetes system. See `go run hack/e2e.go --help` (or the flag definitions in `hack/e2e.go`) for more options, such as reusing an existing cluster.

### Cleaning up

During a run, pressing `control-C` should result in an orderly shutdown, but if something goes wrong and you still have some VMs running you can force a cleanup with this command:

```sh
go run hack/e2e.go -v --down
```

## Advanced testing

### Bringing up a cluster for testing

If you want, you may bring up a cluster in some other manner and run tests against it. To do so, or to do other non-standard test things, you can pass arguments into Ginkgo using `--test_args` (e.g. see above). For the purposes of brevity, we will look at a subset of the options, which are listed below:

```
-ginkgo.dryRun=false: If set, ginkgo will walk the test hierarchy without actually running anything. Best paired with -v.
Expand All @@ -75,7 +133,7 @@ For the purposes of brevity, we will look at a subset of the options, which are
-repo-root="../../": Root directory of kubernetes repository, for finding test files.
```

Prior to running the tests, it is recommended that you first create a simple auth file in your home directory, e.g. `$HOME/.kube/config` , with the following:
Prior to running the tests, you may want to first create a simple auth file in your home directory, e.g. `$HOME/.kube/config` , with the following:

```
{
Expand All @@ -84,23 +142,39 @@ Prior to running the tests, it is recommended that you first create a simple aut
}
```

Next, you will need a cluster that you can test against. As mentioned earlier, you will want to execute `sudo ./hack/local-up-cluster.sh`. To get a sense of what tests exist, you may want to run:
As mentioned earlier there are a host of other options that are available, but they are left to the developer.

`e2e.test --host="127.0.0.1:8080" --provider="local" --ginkgo.v=true -ginkgo.dryRun=true --kubeconfig="$HOME/.kube/config" --repo-root="$KUBERNETES_SRC_PATH"`
**NOTE:** If you are running tests on a local cluster repeatedly, you may need to periodically perform some manual cleanup.

If you wish to execute a specific set of tests you can use the `-ginkgo.focus=` regex, e.g.:
- `rm -rf /var/run/kubernetes`, clear kube generated credentials, sometimes stale permissions can cause problems.
- `sudo iptables -F`, clear ip tables rules left by the kube-proxy.

`e2e.test ... --ginkgo.focus="DNS|(?i)nodeport(?-i)|kubectl guestbook"`
### Local clusters

Conversely, if you wish to exclude a set of tests, you can run:
It can be much faster to iterate on a local cluster instead of a cloud-based one. To start a local cluster, you can run:

`e2e.test ... --ginkgo.skip="Density|Scale"`
```sh
# The PATH construction is needed because PATH is one of the special-cased
# environment variables not passed by sudo -E
sudo PATH=$PATH hack/local-up-cluster.sh
```

As mentioned earlier there are a host of other options that are available, but are left to the developer
This will start a single-node Kubernetes cluster than runs pods using the local docker daemon. Press Control-C to stop the cluster.

**NOTE:** If you are running tests on a local cluster repeatedly, you may need to periodically perform some manual cleanup.
- `rm -rf /var/run/kubernetes`, clear kube generated credentials, sometimes stale permissions can cause problems.
- `sudo iptables -F`, clear ip tables rules left by the kube-proxy.
#### Testing against local clusters

In order to run an E2E test against a locally running cluster, point the tests at a custom host directly:

```sh
export KUBECONFIG=/path/to/kubeconfig
go run hack/e2e.go -v --test_args="--host=http://127.0.0.1:8080"
```

To control the tests that are run:

```sh
go run hack/e2e.go -v --test_args="--host=http://127.0.0.1:8080" --ginkgo.focus="Secrets"
```

## Kinds of tests

Expand All @@ -114,7 +188,13 @@ We are working on implementing clearer partitioning of our e2e tests to make run
- `[Skipped]`: `[Skipped]` is a legacy label that we're phasing out. If a test is marked `[Skipped]`, there should be an issue open to label it properly. `[Skipped]` tests are by default not run, unless a `focus` or `skip` argument is explicitly given.
- `[Feature:.+]`: If a test has non-default requirements to run or targets some non-core functionality, and thus should not be run as part of the standard suite, it receives a `[Feature:.+]` label, e.g. `[Feature:Performance]` or `[Feature:Ingress]`. `[Feature:.+]` tests are not run in our core suites, instead running in custom suites. If a feature is experimental or alpha and is not enabled by default due to being incomplete or potentially subject to breaking changes, it does *not* block the merge-queue, and thus should run in some separate test suites owned by the feature owner(s) (see #continuous_integration below).

Finally, `[Conformance]` tests are tests we expect to pass on **any** Kubernetes cluster. The `[Conformance]` label does not supersede any other labels. `[Conformance]` test policies are a work-in-progress; see #18162.
### Conformance tests

Finally, `[Conformance]` tests are tests we expect to pass on **any** Kubernetes cluster. The `[Conformance]` label does not supersede any other labels. `[Conformance]` test policies are a work-in-progress (see #18162).

End-to-end testing, as described above, is for [development distributions](writing-a-getting-started-guide.md). A conformance test is used on a [versioned distro](writing-a-getting-started-guide.md). (Links WIP)

The conformance test runs a subset of the e2e-tests against a manually-created cluster. It does not require support for up/push/down and other operations. To run a conformance test, you need to know the IP of the master for your cluster and the authorization arguments to use. The conformance test is intended to run against a cluster at a specific binary release of Kubernetes. See [conformance-test.sh](http://releases.k8s.io/HEAD/hack/conformance-test.sh).

## Continuous Integration

Expand Down Expand Up @@ -166,7 +246,7 @@ If we have determined that a test is known-flaky and cannot be fixed in the shor

## Performance Evaluation

Another benefit of the end-2-end tests is the ability to create reproducible loads on the system, which can then be used to determine the responsiveness, or analyze other characteristics of the system. For example, the density tests load the system to 30,50,100 pods per/node and measures the different characteristics of the system, such as throughput, api-latency, etc.
Another benefit of the e2e tests is the ability to create reproducible loads on the system, which can then be used to determine the responsiveness, or analyze other characteristics of the system. For example, the density tests load the system to 30,50,100 pods per/node and measures the different characteristics of the system, such as throughput, api-latency, etc.

For a good overview of how we analyze performance data, please read the following [post](http://blog.kubernetes.io/2015/09/kubernetes-performance-measurements-and.html)

Expand Down

1 comment on commit c25c7c7

@k8s-teamcity-mesosphere
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TeamCity OSS :: Kubernetes Mesos :: 4 - Smoke Tests Build 15345 outcome was SUCCESS
Summary: Tests passed: 1, ignored: 225 Build time: 00:04:13

Please sign in to comment.