Perfomancing testing
Performance testing was implemented after researching the subject in spring 2021. This research document can be found in the drive of Dex The reason why Dex uses performance tests is that it can be used to detect performance bottlenecks earlier on in the development cycle, this then allows developers to quickly find and eliminate the source of bottlenecks. Performance tests are software tests that are focused on testing: speed, scalability, stability, and reliability of a system. Usually, the product owners or stakeholders decide what type of performance tests system a system should use and what is important enough to be tested. Setting up these kinds of tests takes effort and time, but it could give an insight about how an app is performing with each version, this could decrease the risk of bad user experience and/or problematic performance. There are many tools available to test API’s and websites, some of them are less suitable to be used in DEX, though most of these tools will probably suffice just fine.
For now Dex uses K6, K6 is a cli that is based on the programming language GO. Tests are written in JavaScript that K6 uses for each virtual user (VU).For example you may write a script that calls an endpoint each second and then you check if the response time was below 500ms and the statuscode was 200. This single script can now be run by multiple VU's at the same time.
To visualize the performance, Dex makes use of Grafana this application is used for creating dashboards with live data. The Data is written from a timestamp based database called Influx-db. K6 writes data during tests to Influx DB while Grafana automatically refreshes it's data every 5 seconds by default (this can be increased or decreased).
The tests (and visualization tools) are run through Docker using multiple Docker-Compose files. Each Docker-Compose file executes a different kind of test: load, stress, volume, etc. so for example to execute load tests a user would execute a command like docker-compose -f docker-compose-load-tests.yml (or --file docker-compose-load-tests.yml) up. This will run all applications through docker and the performance of the staging environment (by default) will be tested.
As mentioned earlier the tests and infrastructure for testing is all run through Docker. Therefore you will need to have Docker installed for your OS including the Docker-Compose functionality. Further more you will also need the source code from git on your system.
To execute each type of test the user needs to navigate to the root directory of DEX and then navigate to the PerformanceTests directory on Windows this can be done with cd "YOUR_ROOT_DIRECTORY"/PerformanceTests.
Then the user can enter the following command the execute the available tests
docker-compose -f docker-compose-[testtype]-tests.yml up
The [testtype] is one of the following: load, stress or soak.
Docker will now start 3 containers for the K6-cli, Grafana and Influx-DB users can then navigate to https://localhost:3000 to view the performance in Grafana.
Congratulations you are now succesfully running performance tests! 🥳
Note: you can change the http(s) address to test in the .env file. You need to create it yourself. Add the line below
export BASE_ADDRESS=https://api.staging.dex.software
Writing performancetests is fairly easy, but it is very important to know what you are actually testing before writing your tests. Like for example, if you would like to test if the GetProjects endpoint can handle a load of 1000 VU's for 1 hour, than you have a well defined scenario that you can write tests for. These scenario's should be defined together with the stakeholders as they should decide how performant they want to have their application, but keep in mind that you as developer are a technical expert and you should advise the stakeholders accordingly.
After you have defined the test scenario, you should determine what type of test your scenario is. This can be for example a loadtest, a stresstest a soaktest, etc. To learn more about different types of performance test please visit the research in the DEX drive or other sources like Qualitest.
When you have a clearly defined testscenario and you know what type of test you are writing, then you are ready to actually write performance tests. In the example below we are writing a stress test for the GetProjects endpoint. Stress testing is used to test the robustness of an application and to push the application beyond it's normal workload. (All tests are written in JavaScript)
We start by writing the code the Virtual Users (VU's) will execute during the test. Take a look at the example below:
// Import the used K6 libraries
import http from 'k6/http';
import { Rate } from 'k6/metrics';
// This rate will be used to determine the amount of error's there were during the tests.
export let errorRate = new Rate('errors');
// This is the function the Virtual User will execute
export default function() {
// The http client of K6 will make a GET request to the DEX API
// Read about making requests here https://k6.io/docs/using-k6/http-requests/
const res = http.get(`${__ENV.BASE_ADDRESS}/Project?amountOnPage=12`);
// After the request has been made K6 can perform multiple types of checks
// Read about K6 checks here https://k6.io/docs/using-k6/checks/
const result = check(res, {
'status is 200': (r) => r.status == 200,
});
// If our request check failed we will add an error to our Error Rate
if (!result) {
errorRate.add(1)
}
}
After you have written your VU test you will need to define you scenario in the right file. Since we are writing stresstests our scenarios should be defined in the following directory (dex-root)/PerformanceTests/StressTests/stress-tests.js. Take a look at the example below on how you should define your scenario for K6.
// Import the test code the default function is what every VU will execute.
import getProjectsTest from './Tests/getprojects-load-tests';
// This is the main function for running load tests using K6.
// For more information please read the K6 documentation found here: https://k6.io/docs/
// or the research about performancing testing found in the DEX Drive.
// Scenario runtime in seconds
const defaultScenarioRuntime = 300;
// Amount of virtual users for each test
const virtualUsersPerScenario = 5000;
export let options = {
scenarios: {
getProjectsScenario: {
// The executor the scenario will use see THIS IS VERY IMPORTANT FOR EVERY TEST TYPE : https://k6.io/docs/using-k6/scenarios/executors/
executor: 'constant-vus',
// Amount of virtual users
preAllocatedVUs: virtualUsersPerScenario,
maxVUs: virtualUsersPerScenario,
// amount of requests per second
rate: 5000, // 5000 RPS, since timeUnit is the default 1s
// No gracefull stop the time the tests right.
gracefulStop: '0s',
// Tracking tags.
tags: { test_type: 'getProjectsScenario' },
// Test duration
duration: defaultScenarioRuntime + 's',
// The amount of time before the test will start since the cli is executed.
startTime: defaultScenarioRuntime * 1 + 's',
// The function the scenario will execute.
exec: "testGetProjectEndpoints"
},
},
thresholds: {
// Request duration for getting datasources should be smaller than 250ms for 95% of the requests and under 350ms for 99% of the requests.
'http_req_duration{test_type:getProjectsScenario}': ['p(95)<250', 'p(99)<350'],
// Requests for the getDataSource scenario that are allowed to fail must no be more than 1%.
'checks{scenario:getProjectsScenario}': ['rate<0.01'],
}
}
// This function will be called using the exec property of the scenario.
export function testGetProjectEndpoints() {
getProjectsTest();
}
After having defined your scenario you are done and ready to execute your tests by looking at the step above this article if you ran your tests succesfull K6 will write it's Data to K6 and the console it ran in. Keep in mind that stress test data is logged after the tests has finished. This has to do with the Constant VU's executor.
The data logged to the console should look like this:
k6_stress_tests | running (20.1s), 0000/5000 VUs, 495 complete and 2098 interrupted iterations
k6_stress_tests | getProjectsScenario ✓ [ 100% ] 5000 VUs 10s
k6_stress_tests |
k6_stress_tests | data_received.........................: 2.5 MB 124 kB/s
k6_stress_tests | data_sent.............................: 1.7 MB 84 kB/s
k6_stress_tests | http_req_blocked......................: avg=1.13s min=300ns med=500ns max=5.72s p(90)=4.53s p(95)=4.64s
k6_stress_tests | http_req_connecting...................: avg=431.74ms min=0s med=0s max=1.68s p(90)=1.66s p(95)=1.67s
k6_stress_tests | http_req_duration.....................: avg=626.96ms min=28.97ms med=139.86ms max=5.79s p(90)=2.2s p(95)=4.05s
k6_stress_tests | ✗ { test_type:getProjectsScenario }...: avg=626.96ms min=28.97ms med=139.86ms max=5.79s p(90)=2.2s p(95)=4.05s
k6_stress_tests | http_req_failed.......................: 100.00% ✓ 495 ✗ 0
k6_stress_tests | http_req_receiving....................: avg=112.49ms min=0s med=7.16ms max=3.3s p(90)=62.99ms p(95)=830.62ms
k6_stress_tests | http_req_sending......................: avg=122.47µs min=24.9µs med=76.6µs max=2.19ms p(90)=207.62µs p(95)=257.39µs
k6_stress_tests | http_req_tls_handshaking..............: avg=675.94ms min=0s med=0s max=4.14s p(90)=2.78s p(95)=2.89s
k6_stress_tests | http_req_waiting......................: avg=514.34ms min=28.8ms med=122.52ms max=5.79s p(90)=2.12s p(95)=3.99s
k6_stress_tests | http_reqs.............................: 495 24.617949/s
k6_stress_tests | iteration_duration....................: avg=1.75s min=29.11ms med=221.6ms max=9.94s p(90)=4.83s p(95)=5.05s
k6_stress_tests | iterations............................: 495 24.617949/s
k6_stress_tests | vus...................................: 5000 min=0 max=5000
k6_stress_tests | vus_max...............................: 5000 min=5000 max=5000
k6_stress_tests |
k6_stress_tests | time="2021-06-15T08:32:32Z" level=error msg="some thresholds have failed"
k6_stress_tests exited with code 99
