Support integration test framework that brings up a new control plane

[pwittrock]

Time commitment: 30-60 hours (assuming knowledge of Go and the Ginkgo test framework)

We need a way to write integration tests for kubectl and point it at a control plane (apiserver, controller-manager, etcd, scheduler) running locally.

As a kubectl developer, when I write an integration test for kubectl, there should be a ginkgo test framework that launches a new control plane, and provides a kubeconfig to the test both as a tmp file and as a datastructure.

Framework should take the control plane binaries as flags instead of compiling them from source.

See example of a test framework that brings up the control plane here.

Link to existing e2e tests

[apelisse]

@pwittrock wrote somewhere else:

User story:
As a developer of CLIs that talk to Kubernetes APIs, I want to be able to write integration tests for my commands and libraries that run hermetically and locally in my development environment. I want these tests to experience what happens when run against a real apiserver and control plane, but run fast and locally.

Solution:
Provide a test harness that will bring up the Kubernetes control plane locally (apiserver, etcd, controller-manager) locally for each tests.

Individual steps:

• Write a library to bring up etcd from a binary
• Write a library to bring up apiserver from a binary, talk to the local etcd, and provide a kubeconfig file to talk to the apiserver from kubectl or client-go
• Write a library to bring up the controller-manager from a binary and have it talk to the apiserver
• Wrap the libraries in a test harness that will bring them up / tear them down for each test suite
• Setup the pull / merge tests run against kubernetes/kubectl PRs so that the correct binaries are present and flags are provided to support the harness
• Support brining up other pieces of the system such as the scheduler and faking out nodes
• Build helper test libraries for forcing error states

As a high level goal, I would like to be able to publish the framework for any Kubernetes clients to use when testing so that tools such as helm could also benefit.

The goal is to deprecate test-cmd.sh for the many reasons that makes it a bad thing (it's very hard to debug, it's monolithic, and much more)

[apelisse]

As a developer of CLIs

I think it might be useful to think broader. If I work on nodes, wouldn't I need to test against these components too? Let's just say we should have composable testing components, that lets a kubernetes engineer, or even an engineer working on a kubernetes solution, test his software easily, reliably and quickly.

[pwittrock]

Agreed, the next big step for this is helping folks test their CRDs, which would mean some way of loading the CRD and controller from the harness as well.

[totherme]

This looks awesome. @hoegaarden and I would like to have a crack at it. We chatted with @pwittrock about it on a call, and have a good idea where to start.

cc @seans3 @mengqiy I would like to take this

[shiywang]

@totherme cool, since we can't assign to you now, assign myself as a placeholder : )

[pwittrock]

+1

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[nikhita]

/remove-lifecycle stale

[mengqiy]

This has been done. It's at https://github.com/kubernetes-sigs/testing_frameworks