Add integration test design document. #1490
Conversation
|
|
||
| ### Add kube cluster | ||
|
|
||
| We will find a place to keep a long-lived testing cluster, to run the automated test suite. |
There was a problem hiding this comment.
I think ephemeral clusters might be a better option. There's no maintenance of it, also, no worry that an old run will taint the current run. Additionally, this allows for parallel tests to happen.
I've set this up with KinD in the past.
There was a problem hiding this comment.
You also need a mechanism to build a custom version of contour (e.g. a branch), push that image, then pull back into the cluster.
Kind has an ability to sideload images, so build your images, build the cluster, load images, run tests.
| ### Add kube cluster | ||
|
|
||
| We will find a place to keep a long-lived testing cluster, to run the automated test suite. | ||
| The testing cluster should be as close to a vanilla Kubernetes cluster as possible. |
There was a problem hiding this comment.
Any specific versions to target? Just latest stable?
|
|
||
| We will find a place to keep a long-lived testing cluster, to run the automated test suite. | ||
| The testing cluster should be as close to a vanilla Kubernetes cluster as possible. | ||
| Deploying the example requires a `LoadBalancer` Service, so this should run in a cloud provider that can supply LoadBalancers. |
There was a problem hiding this comment.
Why do we need an external LB? We just need to send requests to Envoy. This could be run inside the cluster and target the Envoy service, or use the external mapping of a single node (e.g. port 80/443)
|
|
||
| ### Design test objects | ||
|
|
||
| The test objects should cover all the supported configuration types (at time of writing, Ingress, IngressRoute, and HTTPProxy). |
There was a problem hiding this comment.
Is the effort worth investing for IngressRoute? How long will we support it post 1.0?
|
|
||
| An idea to note is that these test manifests may have an annotation (perhaps `testing.projectcontour.io/valid` or similar) that indicates to the automation if the file is part of a test case that should succeed or fail. | ||
|
|
||
| These object manifests should be stored in a central location, most likely inside the Contour repository. |
There was a problem hiding this comment.
I've been thinking about the test manifests. It would be neat to have the /internal/e2e tests use the same ones for the proposed integration tests. I have some generator logic which could transform the go structs to yaml manifests. This way, if the api changes, the tooling updates as well.
A side-benefit could also be examples we give out to users. We could mark specific tests as "examples" or something, then have them get auto-generated into the examples users could use to show off features (e.g. blue-green, canary, etc)
|
|
||
| ### Templating for test objects | ||
|
|
||
| The test manifests will need some form of templating, to allow them to be able to run on other clusters than the default testing cluster. |
There was a problem hiding this comment.
I'm not against tooling, but you could also just pass a host in the headers of the test to get around this.
|
|
||
| ## Goals | ||
|
|
||
| - Enable full integration testing of Contour and Envoy, when combined with Kubernetes. |
There was a problem hiding this comment.
Can you add a definitions section to the document. Define what you mean by integration test.
| ## Goals | ||
|
|
||
| - Enable full integration testing of Contour and Envoy, when combined with Kubernetes. | ||
| - Ensure that the testing rig is extensible enough to allow further additions later (for example performance testing and benchmarking) |
There was a problem hiding this comment.
I'd like to see this more concrete. Perhaps make it an explicit goal that the testing rig can be deployed to multiple clusters (ie, its not hard coded to one cluster we run)
|
|
||
| - Enable full integration testing of Contour and Envoy, when combined with Kubernetes. | ||
| - Ensure that the testing rig is extensible enough to allow further additions later (for example performance testing and benchmarking) | ||
|
|
There was a problem hiding this comment.
I would like to see an explicit goal that a test cluster is part of the deliverable, say test-cluster.projectcontour.io. This will be a service which we run and can be used for anyone who wants to run the integration tests as well us a target for say a nightly run of the integration tests of :master
|
|
||
| Some examples of things that are hard to test currently: | ||
|
|
||
| - Validating that session affinity routes requests as configured. |
There was a problem hiding this comment.
timeouts are untested
the way that envoy will ignore health checks and route to down clusters is particularly surprising and has caught us several times already.
| - Feature tests (currently called `e2e` although that is a misnomer, see [#1436](https://github.com/heptio/contour/issues/1436) (in-repo, run as a gate on PRs) | ||
| - Integration tests (run in a testing Kubernetes cluster). | ||
|
|
||
| Initially, the new integration tests will be run either manually or on a cron schedule, or possibly after a merge to master. In the future, it would be good to have a facility to run per-branch, but that is not required in the first cut. |
There was a problem hiding this comment.
please break lines after a period where possible
|
|
||
| - uses technology developed for testing Kubernetes things | ||
| - produces Go test files, which means less for prospective test-adders to learn | ||
| - has a large degree of extensibility. |
There was a problem hiding this comment.
I'm pretty negative on the use of the go testing package for anything outside unit tests. It not intended for this and won't respond well to being pressed into this service.
youngnick
force-pushed
the
1155-testing-design
branch
from
d167a9b
to
3934660
Compare
Fixes projectcontour#1155. Signed-off-by: Nick Young <ynick@vmware.com>
youngnick
force-pushed
the
1155-testing-design
branch
from
3934660
to
3393168
Compare
|
Marking this PR stale since there has been no activity for 14 days. It will be closed if there is no activity for another 90 days. |
github-actions
bot
added
the
lifecycle/stale
|
Closing out this PR as it's very stale, @jpeach is doing much better work, and we won't be going ahead with this approach. |
Fixes #1155.
Signed-off-by: Nick Young ynick@vmware.com