Testing

Table of Contents

1. Test Plan Identifier
2. Introduction
3. Test Items
4. Features to be Tested
5. Features not to be Tested
6. Approach
7. Item Pass/Fail Criteria
8. Suspension Criteria and Resumption Requirements
9. Test Deliverables
10. Testing Tasks
11. Environmental Needs
12. Responsibilities
13. Staffing and Training
14. Schedule
15. Approvals

1. Test-Plan Identifier

2. Introduction

2.1 Objectives.

This test plan focuses on testing the horizontal scalability of the Express project when deployed on a Kubernetes cluster. It supports the following objectives.

1. To

2.2 Background

This test plan was created to evaluate the ability

2.3 Scope

3. Test Items

The integration of the services within Express environment is to be tested. The tests focus on the ability for the services to automatically scale horizontally as the number of requests increases. This means the following services will be tested together for Express v1.0.0-alpha.1.

1. Authorization and Identity Service.
   • Documentation
   • Design
2. Nginx Reverse Proxy.
   • Documentation
   • Design
3. Test API Application Container.
4. Distributed Cassandra Database.

4. Features to be Tested

The primary feature that will be tested is the ability for the Express environment to scale when an application container is deployed inside of it. This will test the following features.

1. Automatic Horizontal Scaling. The ability for the services in the Express environment to automatically scale across multiple physical nodes as the number of requests increases.
   1. The authorization and identity service.
   2. The deployed application container.
   3. The distributed Cassandra database.
2. Request Latency. The impact horizontal scalability has on request latency.
3. Service Integration. The integration of all services within the Express environment.
   1. Reverse proxy delegation of requests for protected resources to the authorization and identity service.
   2. Sharing of the distributed Cassandra database between the authorization and identity service and the deployed application container.
   3. Identity management for the deployed application container by the authorization and identity service.

5. Features Not to be Tested

The Express environment consists of multiple separate services. The correctness of these individual services will not be evaluated directly by these tests. As a result, the following features will not be tested.

1. Express Controller. The Express Controller will be used as part of the tests to deploy, update and teardown application contains but will not be tested itself. It will not be directly evaluated because unit and functional tests are included separately.
2. Express. Correctness or functionality of the individual services will not be tested. Correctness will be assumed based on separate unit tests and the presence of errors, or lack thereof, while performing the testing tasks.
   1. Authorization and Identity Service
   2. Deployed Application Container
   3. Nginx Reverse Proxy

6. Approach

The goal of the testing is to evaluate the Express project in a production environment with realistic traffic. Because of this, an approach based on separating the Express project from the test driver is used.

6.1 Express Services

The Express services are to be deployed to a Kubernetes cluster with 1 to 4 machines each with their own dedicated resources. Having their own dedicated resources allows the Express services to be tested in a production environment.

6.2 Test Driver

The test driver is to be run on a separate machine that communicates with the Express services over HTTP. Using this approach ensures that the test driver does not have any impact on the performance of the Express services running on the cluster.

Simulating Traffic

To simulate realistic traffic to the Express services, Locust will be used. This allows traffic to be modelled using a user simulation with each test case defined by a maximum number of users, the starting number of users and the rate at which new users are to be created.

This approach allows different parameters to be used to test different factors.

1. Maximum Requests Per Second. Start with a high maximum number of users and a low user spawn rate to test where latency starts to increase and/or requests start to timeout.
2. Request Spikes. Start with a maximum number of users equal to or below the maximum requests per second and use a high spawn rate to test how latency, timeout and error frequency change with rapid increases in the number of requests per second.

User Model

To use this approach to test the Express services, a user model must be created to simulate the HTTP requests that a user might send.

7. Item Pass/Fail Criteria

8. Suspension Criteria and Resumption Requirements

8.1 Suspension Criteria.

9. Test Deliverables

The following documents shall be included:

1. Test Plan. This document is the test plan for the Express project.
2. Test Summary Reports. A brief summary will be included each time the testing tasks are performed to discuss any notable trends. The summary will not include analysis of these trends.
3. Test Input Data. This will be in the form of the number of users to simulate and the number of users spawned/second.
4. Test Output Data. Each time the testing tasks are performed the following will be produced.
   • distribution.csv containing the absolute number of each HTTP request sent during the test.
   • exceptions.csv containing any errors that occurred during the test.
   • requests.csv containing the requests per second, average sliding-window response time, and number of users.

10. Testing Tasks

The testing tasks are to performed four times. First on a Kubernetes cluster consisting of a single physical or virtual machine. The tasks must then be repeated on the same cluster after the addition of another machine. This process will be performed until the testing tasks have been completed on clusters consisting of one, two, three and four machines.

10.1 Preparation

All preparation will be performed from the single test machine. The preparation assumes that the environmental needs in section 11 have been met. It is also required that the express-controller executable and Express v1.0.0-alpha.1 archive have been downloaded to the single test machine from the Releases section of the Express Github repository.

Cluster Reset

The Kubernetes cluster must be in a consistent state when starting the tasks. This can be accomplished by performing the following tasks. These tasks are only necessary if the testing tasks have already been performed on the cluster.

1. Remove the previously deployed application using express-controller teardown testapi.
2. Remove the Express environment from the cluster using helm delete terrifying-bronco.
3. Remove all persistent volume claims using kubectl delete pvc terrifying-bronco-database-0.
4. Remove all persistent volumes using kubectl delete pv terrifying-bronco-database-0.
5. Wait until No resources found. is shown when using kubectl get pods.

Cluster Setup

The Kubernetes cluster must be prepared for running the testing tasks. This can be done by performing the following tasks.

1. Navigate to the root directory of the Express v1.0.0-alpha.1 download in bash.
2. Add the Express environment to the cluster using helm install --replace --name terrifying-bronco ./express/.
3. Wait until all pods are ready when using kubectl get pods.
4. Deploy the test CRUD API using express-controller deploy testapi allankerr/api --port 8080 --max X --endpoint-config ./containers/api/endpoints.yaml. The parameter max must be changed to the number of nodes in the cluster.
5. Wait until all pods are ready when using kubectl get pods.
6. Create a new admin OAuth2 client for the authorization service using ./bin/authorization-cli.sh client create admin demo-password.
7. Add the user scope to the admin OAuth2 client using ./bin/authorization-cli.sh scope add admin user.
8. Wait until the cluster is stable by ensuring the horizontal pod autoscalers do not have <unknown> targets when using kubectl get hpa.

10.2 Test Procedure

11. Environmental Needs

11.1 Hardware Needs

Four physical or virtual machines with identical hardware specifications are required to run the Kubernetes cluster. Each machine must have a minimum of 2 GB of memory and the following minimum CPU requirements.

1. 2 AWS vCPU
2. 2 GCP Core
3. 2 Azure vCore
4. 2 Hyperthread on a bare-metal Intel processor with Hyperthreading

A single physical or virtual test machine is required to run the tests and interface with the Kubernetes cluster. It is recommended that the machine's CPU is at minimum an Intel Core i5-3330 or equivalent.

The test machine must be able to communicate with the four physical machines over a public or private network using HTTP.

11.2 Software Needs

The four machines must meet the following software requirements.

1. Kubernetes cluster. A Kubernetes cluster running version 1.8.5 must be deployed on the machines. Documentation on setting up a Kubernetes cluster can be found here.
2. Helm. The Helm server must be initialized on the Kubernetes cluster using the helm init command after installing the Helm client and kubectl on the single test machine.

The single test machine must meet the following software requirements.

1. kubectl. kubectl must be installed to interface with the Kubernetes cluster. Documentation on setting up kubectl can be found here. The kubectl config must be configured to access the deployed Kubernetes cluster.
2. Helm. The Helm client must be installed. Documentation on setting up the Helm client can be found here.
3. Python 3.0.
4. Bash.

11.3 Description of Actual Testing Environment

For executing the tests, Kubernetes Engine on the Google Cloud Platform will be utilized. The node pool will be setup in the us-central1-a zone with autoscaling disabled. Four Compute Engine VM instances with 2 vCPU cores and 4 GB of memory will be utilized. The cluster will be setup to use Kubernetes v1.8.5.

To execute the tests on the Kubernetes cluster, a Mid 2014 13-inch MacBook Pro running macOS Sierra Version 10.12.6 with 16 GB of memory and a 3 GHz Intel Core i7 processor will be utilized. The University of Saskatchewan's wireless network will be used to perform the tests.

12. Responsibilities

13. Staffing and Training

Testing will be performed by Allan Kerr; however, the tests can be performed by anyone. Knowledge in the following areas is required to perform the tests.

• Express. The test plan assumes knowledge of how to use the Express project. Understanding the following documentation is sufficient.
  • Authorization / Identity Service Documentation
  • Express Controller Documentation
• Environment. A basic understanding of Kubernetes is assumed for purposes of setting up the test environment.
  • Kubernetes Documentation
• Test Driver. A basic understanding of Python and Locust is assumed for running the tests.
  • Locust Documentation

14. Schedule

Testing will be completed by February 18, 2018. Testing may continue beyond this date on a new version Express v1.0.0-alpha.2 containing changes made as a result of the tests.

15. Approvals

No approvals are required.