Add e2e test proposal

Signed-off-by: GuessWhoSamFoo <foos@vmware.com>

proposals/20200330-e2e-testing.md

@@ -0,0 +1,97 @@
+# Octant e2e Testing
+
+Create a stable end-to-end testing developer workflow. The previous iteration of e2e tests required a given cluster state with expected Octant UI performance before executing tests resulting in flakey tests.
+
+## Goals
+
+ - Mock backend data to avoid maintaining cluster state for tests
+ - Remove dependencies such as kubectl and kind needed to start e2e tests
+ - Provide an alternative for difficult testing cases
+ - Gain feedback for automated user interaction
+
+## Non-Goals
+
+ - Test Octant behavior under restricted RBAC
+ - Benchmarking performance
+ - Expanding test coverage
+ - Test electron
+
+### Current implmentation
+
+The previous implementation removed after [v0.10](https://github.com/vmware-tanzu/octant/tree/release-0.10) had the following steps:
+
+1. Install and run `kind`
+1. Install kubectl and start Octant
+1. Apply yaml to create a test namespace and resources
+1. Run Cypress tests
+1. Clean up test namespace
+
+A wrapper that executes before tests would use `cy.execute` to run kubectl commands that could create a namespace.
+
+There were several problems as a result of this setup:
+ - Test/cypress failure would leave the cluster in a dirty state
+ - Sharing cluster state across multiple tests lead to multiple failures
+ - Unreliable reporting as a result of cascading failures
+
+Running this setup locally became problematic because it would often leave behind unused resources if failing. The greater amount of compute resources available compared to a CI environment also hid undesired loading UI behaviors. For example, a resource may not be ready on the cluster, but Octant loads then proceed to fail the test.
+
+### Suggested implementation
+
+Tests can generally be divided into two types: mocked and cluster-dependent. This split is used to reflect the future development of action which allows running without a kubeconfig. Mocked tests will use stubbed websocket messages and are not truly e2e tests. These should check components that are not dependent on cluster states such as the context dropdown, navigation, and input filter.
+
+Rather than using kubectl for cluster dependent tests, use the javascript/typescript [client-go](https://github.com/kubernetes-client/javascript) library to programmatically create resources as needed by a given test.
+
+The wrapper that runs before all tests will only create a namespace and delete that namespace upon completion.
+
+Each test will be responsible for validating the needed cluster state before attempting to run tests.
+
+#### Mocked
+
+Mocked tests will have a list of stubbed messages read from file. Stubbed messages can be sent through various libraries for sending websocket messages with an API similar to below:
+
+```
+  sockets.mockServer = new Server("ws://localhost:7777/api/v1/stream")
+
+  sockets.mockServer.on("connection", socket => {
+    sockets.server = socket
+
+    // Will be sent any time a client connects
+    socket.send("Hello, world!")
+
+    socket.on("message", data => {
+      // Do something
+    }
+  }
+```
+
+A disadvantage of this method is that changes to components will also require updates to the stubbed messages.
+
+This method should be available because it avoids the need to provide cluster resources to test user interactivity. It also provides an alternative for testing potentially complex interactions. For example, we may want to test the color of an object in the resource viewer. Rather than creating objects which are known to have a certain object status in-cluster, this can provide a simplified alternative.
+
+#### Cluster-Dependent
+
+An example of a new implementation of tests using the javascript client-go library can be demonstrated with a case of testing port forwards.
+
+The current implementation looks for the port forward button HTML element in the summary view of a pod, clicks to start, then watches for the stop port forward button.
+
+Instead, the new implementation will [start a port forward](https://github.com/kubernetes-client/javascript/blob/master/examples/typescript/port-forward/port-forward.ts) for a given pod only check for the existence of a stop port forward button.
+
+The difference allows for creating a more specific by checking port forward functionality without having potential failures in generating the start button, click the button, etc. Another test for checking the contents of `Summary` tab will effectively test the same functionality without requiring a cluster state of an active port forward and a closed connection within the same test.
+
+### Implementation strategy
+
+A new Github Action workflow is added using [kinD](https://github.com/marketplace/actions/kind-kubernetes-in-docker-action) and [Cypress](https://github.com/cypress-io/github-action).
+
+Make a list of features to be tested based on the two groups described above.
+
+Create a `server.js` that will create a mock websocket to send messages.
+
+Create a wrapper that loads a given kubeconfig and create a namespace using the imported javascript client-go library.
+
+Depending on the test, `cy.before` will call the mocked websocket or do cluster setup.
+
+Each test that needs to create a resource will be responsible for doing so within the test. Reusable logic such as checking if a pod is running can be moved to a common library to be shared.
+
+Document the method for running e2e tests locally.
+
+Recordings of automated user interactions are recorded with the provided `CYPRESS_KEY` and send to a service which allows manual review if needed.