Testing

2. Introduction

2.1 Objectives.

This test plan focuses on testing the horizontal scalability of the Express project when deployed on a Kubernetes cluster. It supports the following objectives.

1. To

2.2 Background

This test plan was created to evaluate the ability

2.3 Scope

3. Test Items

The integration of the services within Express environment is to be tested. The tests focus on the ability for the services to automatically scale horizontally as the number of requests increases. This means the following services will be tested together for Express v1.0.0-alpha.1.

1. Authorization and Identity Service.
   • Documentation
   • Design
2. Nginx Reverse Proxy.
   • Documentation
   • Design
3. Test API Application Container.
4. Distributed Cassandra Database.

4. Features to be Tested

The primary feature that will be tested is the ability for the Express environment to scale when an application container is deployed inside of it. This will test the following features.

1. Automatic Horizontal Scaling. The ability for the services in the Express environment to automatically scale across multiple physical nodes as the number of requests increases.
   1. The authorization and identity service.
   2. The deployed application container.
   3. The distributed Cassandra database.
2. Request Latency. The impact horizontal scalability has on request latency.
3. Service Integration. The integration of all services within the Express environment.
   1. Reverse proxy delegation of requests for protected resources to the authorization and identity service.
   2. Sharing of the distributed Cassandra database between the authorization and identity service and the deployed application container.
   3. Identity management for the deployed application container by the authorization and identity service.

5. Features Not to be Tested

The Express environment consists of multiple separate services. The correctness of these individual services will not be evaluated directly by these tests. As a result, the following features will not be tested.

1. Express Controller. The Express Controller will be used as part of the tests to deploy, update and teardown application contains but will not be tested itself. It will not be directly evaluated because unit and functional tests are included separately.
2. Express. Correctness or functionality of the individual services will not be tested. Correctness will be assumed based on separate unit tests and the presence of errors, or lack thereof, while performing the testing tasks.
   1. Authorization and Identity Service
   2. Deployed Application Container
   3. Nginx Reverse Proxy

6. Approach

The goal of the testing is to evaluate the Express project in a production environment with realistic traffic. Because of this, an approach based on separating the Express project from the test driver is used. The Express project is to be deployed to a Kubernetes cluster with 1 to 4 machines each with their own dedicated resources. The test driver is to be run on a separate machine that communicates with the Express services over HTTP. Using this approach ensures that the test driver doesn't have any impact on the performance of the Express services running on the cluster.

Deploying the Kubernetes cluster on machines with their own dedicated resources allows the Express services to be tested in a production environment.

8. Suspension Criteria and Resumption Requirements

8.1 Suspension Criteria.

9. Test Deliverables

The following documents shall be included:

1. Test Plan. This document is the test plan for the Express project.
2. Test Summary Reports. A brief summary will be included each time the testing tasks are performed to discuss any notable trends. The summary will not include analysis of these trends.
3. Test Input Data. This will be in the form of the number of users to simulate and the number of users spawned/second.
4. Test Output Data. Each time the testing tasks are performed the following will be produced.
   • distribution.csv containing the absolute number of each HTTP request sent during the test.
   • exceptions.csv containing any errors that occurred during the test.
   • requests.csv containing the requests per second, average sliding-window response time, and number of users.

10. Testing Tasks

The testing tasks are to performed four times. First on a Kubernetes cluster consisting of a single physical or virtual machine. The tasks must then be repeated on the same cluster after the addition of another machine. This process will be performed until the testing tasks have been completed on clusters consisting of one, two, three and four machines.

10.1 Preparation

All preparation will be performed from the single test machine. The preparation assumes that the environmental needs in section 11 have been met. It is also required that the express-controller executable and Express v1.0.0-alpha.1 archive have been downloaded to the single test machine from the Releases section of the Express Github repository.

Cluster Reset

The Kubernetes cluster must be in a consistent state when starting the tasks. This can be accomplished by performing the following tasks. These tasks are only necessary if the testing tasks have already been performed on the cluster.

1. Remove the previously deployed application using express-controller teardown testapi.
2. Remove the Express environment from the cluster using helm delete terrifying-bronco.
3. Remove all persistent volume claims using kubectl delete pvc terrifying-bronco-database-0.
4. Remove all persistent volumes using kubectl delete pv terrifying-bronco-database-0.
5. Wait until No resources found. is shown when using kubectl get pods.

Cluster Setup

The Kubernetes cluster must be prepared for running the testing tasks. This can be done by performing the following tasks.

1. Navigate to the root directory of the Express v1.0.0-alpha.1 download in bash.
2. Add the Express environment to the cluster using helm install --replace --name terrifying-bronco ./express/.
3. Wait until all pods are ready when using kubectl get pods.
4. Deploy the test CRUD API using express-controller deploy testapi allankerr/api --port 8080 --max X --endpoint-config ./containers/api/endpoints.yaml. The parameter max must be changed to the number of nodes in the cluster.
5. Wait until all pods are ready when using kubectl get pods.
6. Create a new admin OAuth2 client for the authorization service using ./bin/authorization-cli.sh client create admin demo-password.
7. Add the user scope to the admin OAuth2 client using ./bin/authorization-cli.sh scope add admin user.
8. Wait until the cluster is stable by ensuring the horizontal pod autoscalers do not have <unknown> targets when using kubectl get hpa.

10.2 Test Procedure

11. Environmental Needs

11.1 Hardware Needs

Four physical or virtual machines with identical hardware specifications are required to run the Kubernetes cluster. Each machine must have a minimum of 2 GB of memory and the following minimum CPU requirements.

1. 2 AWS vCPU
2. 2 GCP Core
3. 2 Azure vCore
4. 2 Hyperthread on a bare-metal Intel processor with Hyperthreading

A single physical or virtual test machine is required to run the tests and interface with the Kubernetes cluster. It is recommended that the machine's CPU is at minimum an Intel Core i5-3330 or equivalent.

The test machine must be able to communicate with the four physical machines over a public or private network using HTTP.

11.2 Software Needs

The four machines must meet the following software requirements.

1. Kubernetes cluster. A Kubernetes cluster running version 1.8.5 must be deployed on the machines. Documentation on setting up a Kubernetes cluster can be found here.
2. Helm. The Helm server must be initialized on the Kubernetes cluster using the helm init command after installing the Helm client and kubectl on the single test machine.

The single test machine must meet the following software requirements.

1. kubectl. kubectl must be installed to interface with the Kubernetes cluster. Documentation on setting up kubectl can be found here. The kubectl config must be configured to access the deployed Kubernetes cluster.
2. Helm. The Helm client must be installed. Documentation on setting up the Helm client can be found here.
3. Python 3.0.
4. Bash.

11.3 Description of Actual Testing Environment

For executing the tests, Kubernetes Engine on the Google Cloud Platform will be utilized. The node pool will be setup in the us-central1-a zone with autoscaling disabled. Four Compute Engine VM instances with 2 vCPU cores and 4 GB of memory will be utilized. The cluster will be setup to use Kubernetes v1.8.5.

To execute the tests on the Kubernetes cluster, a Mid 2014 13-inch MacBook Pro running macOS Sierra Version 10.12.6 with 16 GB of memory and a 3 GHz Intel Core i7 processor will be utilized. The University of Saskatchewan's wireless network will be used to perform the tests.