Welcome to the TDS Load Test Project
The tds-loadtest project contains a jMeter test plan used to profile the performance of the Test Delivery System (TDS). The project also contains Java code for generating response data for test questions, which is used by the load test, as well as python scripts for generating test seed data.
The load test plan is built to simulate a large number of students taking a test simultaneously, as well as their respective proctors logging in and managing test sessions. The test plan works with two groups of threads running in parallel.
The first phase begins with proctor threads logging in and generating new test sessions. After generating a new session id and writing it into memory, each proctor will wait while students begin to join the test sessions. Proctors wait to approve students, while the student threads begin to login and select their tests, chosen at random.
During the second phase, students login and begin the test, as the proctor threads approve each student. Students then take the test to completion, and proctors begin to close sessions after all respective students have completed their tests.
- Apache JMeter 2.13 - For analyzing test output (.jtl) file, editing test plan file (.jmx), and executing load tests
- Java 7 - Dependency of jmeter
- Python 2.7+ (for running the scripts to generate the proctor/user .csv and for automated load testing)
- Docker + docker-machine
- Both can be downloaded for Mac OS X and Windows, bundled in the Docker Toolbox Download HERE
Load Test Setup and Configuration
Each jMeter server node will need a copy of the tds-loadtest.jmx test plan. This test plan contains the properties and logic used by jMeter to execute a performance test. Each jMeter node must also contain a unique set of student and proctor logins (student_logins.csv and proctor_logins.csv).
JMeter Single Node Instructions
Create a directory for the load test
mkdir ~/load-test cd ~/load-test
Generate student and proctor login account credentials by running the python scripts in the project's /bootstrapping/ directory. Place the .csv files that are generated by these scripts into the load test directory on the jMeter server. The .csv files must be named student_logins.csv and proctor_logins.csv respectively. The student_logins.csv file can also be imported directly into ART in order to create the student accounts in TDS. Provide the -h flag to either seed script for more options.
python /project/path/bootstrapping/create_students.py -n \<num of students\> -g \<grade of students\> python /project/path/bootstrapping/create-proctors.py -n \<num of proctors\>
Configure jMeter test variables by editing the tds-loadtest.jtl file and changing the values for the following properties:
- TOTAL_STUDENT_COUNT - The total number of students threads for each jmeter node that will take a test.
- STUDENTS_PER_PROCTOR - The ratio of the amount of students for each proctor.
- PROCTOR_HOST - The proctor host URL.
- SSO_HOST - The SSO host URL.
- STUDENT_START_DELAY_IN_MILLIS - The amount of milliseconds that the student threads should be delayed to allow the proctor threads to create sessions.
- PROCTOR_RAMP_UP_TIME_IN_SECS - The period of time that proctor threads should begin executing over.
- STUDENT_RAMP_UP_TIME_IN_SECS - The period of time that student threads should begin executing over.
Configure jMeter, Java, and system settings on the jMeter to allow for a large number of open file pointers and sufficient Java heapspace.
After manually smoke testing the TDS and ensuring that all other other components are functional, you are ready to execute the jmeter test plan.
jmeter -n -t ./tds-loadtest.jmx -l <testoutput-filename>.jtl &
- A progress summary line will be printed to standard output every 30 seconds. All test progress will the written to the .jtl file specified, and the jmeter.log will be placed at the test root level. Upon the conclusion of the test, test results can be analyzed. Analysis can be done on a machine running jMeter in GUI mode. Open the .jtl file using the "View Results Tree" or "Summary Report" tool to view the result data.
Automated TDS Loadtest Deployment for Distributed JMeter Mode (Multi-Node)
For running very large-scale loadtests that require a load larger than what can be generated in a single EC2 instance with JMeter, the tds distributed jmeter loadtest script can be utilized. In short, this script can, with very little manual effort, deploy multiple jmeter-server ec2 instances and a jmeter-client ec2 instance and run a loadtest using jmeter's distributed clustered mode.
In this mode, a jmeter instance acts a client, distributing the load test plan to each individual server (worker) node. The results of the loadtest are then communicated back to and aggregated by the jmeter client node. In this mode, each worker node is required to have a copy of the subset of seed data. For example, when running a 100,000 student test (with 10,000 proctors), the server-1 node would be responsible for the first 50,000 students and 5,000 proctors, the server-2 would be responsible for the remaining students and proctors. Each server needs to have a unique copy of the subset of total seed data it is responsible for.
The python script, "tds-distributed-loadtest.py" has been created to automate the creation of each jmeter node, using docker-swarm and docker-remote to create the ec2 instances and deploy the jmeter images, as well as distributing the generated student and proctor seed data equally between the number of worker nodes provided.
For more information regarding Apache jMeter's Distributed mode, please click here.
In summary, the tds-distributed-loadtest.py takes the following actions:
- Creates an ec2 instance for the jmeter-client node (labelled "tds-jmeter-client") and n jmeter server nodes ("tds-jmeter-server-n")
- Deploys the proper jmeter docker image on each node, which includes installing jmeter and all its dependencies.
- Creates an AWS security group (tds-jmeter-sg) and opens the 1099 port (used for jmeter inter-node communication)
- Creates the load_tests directory on the tds-jmeter-client ec2 node and copies the test plan (.jtl) file supplied as the script's argument to the tds-jmeter-client node.
- Generates and distributes student and proctor seed data evenly amongst the nodes.
- Prints a listing of the remote docker containers, including container ids for reference.
- Runs the jmeter test on the jmeter client node, which signals the workers to begin generating load and collecting results.
Running High-Scale TDS Distributed Load Test
- Edit the app_aws.env file and replace the values of the variables with the proper AWS credentials. These credentials and IDs will be used to create the ec2 instances with Docker.
- Ensure that ART and TDS contains the loadtest students and proctors and that the accounts are configured correctly.
- Execute the loadtest using the tds-distributed-loadtest.py script. Note the following useful flags:
- -h / --help : Script help and instructions, including optional flags
- -w / --workers : The number of worker (server) nodes to create and use for the loadtest (Default: 2)
- -s / --students : The number of students to execute the test with (Default: 20)
- -p / --proctors : The number of students per proctor (Default: 10)
- Start a loadtest with 200,000 students and 20,000 proctors, distributed between 4 worker nodes:
python tds-distributed-loadtest.py -w 4 -s 200000 -p 10 /path/to/tds-loadtest.jmx
- Monitor the tds-jmeter-client docker log using the following command.
docker logs -f <tds-jmeter-client container id>
- Copy the test .jtl file from the tds-jmeter-client node to the host machine using the following command:
docker-machine scp tds-jmeter-client:/load_tests/tds-loadtest.jtl .
Post-Distributed Load Test Cleanup
To clean up a distributed load testing environment after running a test and collecting the resulting test data, simply run the "tds-distributed-loadtest.py" python script with the -c (--cleanup) flag. This will kill each tds-jmeter docker-machine remote instance, which will effectively terminate each EC2 instance and remove the associated keypairs from AWS. Allow a few minutes for AWS to terminate these instances.
Useful commands for viewing test run errors
- cat jmeter.log | grep "ERROR" - all general errors
- cat jmeter.log | grep "NullPointer" -c - count of null pointer exceptions
- cat jmeter.log | grep "OutOfBounds" -c - count of all "ArrayIndexOutOfBounds" errors
- cat jmeter.log | grep "Connection has timed out" -c - count of errors related to leaky connections
- cat testoutput.jtl | grep "1.1 5" -c - count of all "5xx" HTTP errors
- cat testoutput.jtl | grep "1.1 4" -c - count of all "4xx" HTTP errors
NOTE: Be sure to clear the jmeter.log and .jtl file after each test execution. For accurate results, be sure to use fresh proctor and student accounts between subsequent load test executions.