Uploader Performance Tests

Locust-based infrastructure load tests for the ONS Council Tax Uploader. Tests the full backend upload flow - CloudFront edge delivery, API Gateway -> Lambda pre-signed URL generation, and S3 multipart and single-file uploads - under load patterns representative of peak council usage.

Three test types are supported: Load, Stress and Spike.

Setup

Install dependencies:

poetry install

Configure environment variables by creating a new file called .env and copying in the contents of .env.example. Fill in the following values:

CLOUDFRONT_BASE_URL

The base URL of the CloudFront distribution serving the uploader.

.env.example defaults to the Dev environment - change this as appropriate for the environment you are testing against

In the AWS Console: CloudFront -> Distributions -> Uploader distribution -> Distribution domain name

API_GATEWAY_BASE_URL

The invoke URL of the API Gateway that fronts the pre-signed URL Lambda.

In the AWS Console: API Gateway -> Uploader API -> Stages -> Uploader stage -> Invoke URL

Example: API_GATEWAY_BASE_URL=https://abcdefghij.execute.api-eu-west-2.amazonaws.com

❗This value is sensitive. Do not commit it to Git❗ .env is listed in .gitignore but always double-check your staged changes before pushing. Never hardcode these values directly in source files.

Separation of concerns

The Uploader flow logic - requesting pre-signed URLs, uploading the EXTRACT and MANI files, handling responses - is defined once in behaviours/uploader_behaviour.py and shared across all test types via inheritance. This means changes to the Uploader flow only need to be made in one place.

Each test file (uploader_load_test.py, uploader_stress_test.py, uploader_spike_test.py) is a thin wrapper that defines only what makes that test type distinct:

• User class - the wait_time between iterations, which controls request pressure
• Shape class (stress and spike only) - the VU ramp profile over time, imported from the co-located shapes.py
• Threshold monitoring (stress only) - a background greenlet that stops the test automatically when a metric breach is detected

The .conf files control Locust runtime settings - headless mode, output paths, run time - and are intentionally kept free of environment-specific values. All environment-specific configuration (URLs, file sizes, thresholds) lives in .env.

Running Tests

Load Test (baseline)

Validates performance under normal council usage by simulating multiple councils uploading simultaneously. Gradually ramps to 20 concurrent users over 4 seconds, holds for 20 minutes, then exits automatically.

To run the tests, execute the below, replacing the --host value with the appropriate CLOUDFRONT_BASE_URL for different environments:

poetry run locust -f performance_tests/load/uploader_load_test.py --config=performance_tests/load/load.conf --host=https://uploader.ingest-dev.aws.onsdigital.uk

load.conf

To adjust the load profile, edit load.conf directly - for example, increase users to simulate a higher number of simultaneous council uploads, or extend run-time to observe longer-term stability.

Setting	Value	Description
locustfile	uploader_load_test.py	Defines the Locust test script that contains the user behaviour and tasks executed during the load test.
headless	false	Opens the Locust web UI at http://localhost:8089 for live monitoring — set to true to suppress
autostart	true	Starts the test automatically without waiting for input
autoquit	5	Exits automatically 5 seconds after the test completes
loglevel	INFO	Controls the verbosity of Locust logging
users	20	Total number of virtual users (councils) to simulate
spawn-rate	5	How many users to add per second during ramp-up
run-time	20m	How long to hold at peak load before stopping

The test runs for approximately 20 minutes at peak load after a short ramp-up. Compare performance metrics during ramp-up and sustained load to assess system stability under normal expected usage and confirm consistent behaviour over time without degradation.

Spike Test

Validates system resilience to sudden surges in council uploads by introducing an abrupt increase in concurrent users, maintaining peak load briefly, and then returning to a low baseline. This helps identify burst-handling capability, cold-start behaviour, error rates during rapid scaling events, and recovery time after the spike subsides.

Monitor the test in real time using the Locust UI and observe:

• Response time degradation immediately following the spike
• Increased 4xx or 5xx error rates under burst traffic
• Autoscaling or cold-start behaviour
• Time taken for performance metrics to return to baseline after load decreases

To run the tests, execute the below, replacing the --host value with the appropriate CLOUDFRONT_BASE_URL for different environments:

poetry run locust -f performance_tests/spike/uploader_spike_test.py --config=performance_tests/spike/spike.conf --host=https://uploader.ingest-dev.aws.onsdigital.uk

spike.conf

Unlike the load test, the spike test uses a custom Locust LoadTestShape (shapes.py) to define the load pattern. The configuration file primarily controls execution behaviour and reporting.

Setting	Value	Description
locustfile	uploader_spike_test.py	Defines the Locust test script that contains the user behaviour and tasks executed during the spike test.
headless	false	Opens the Locust web UI at http://localhost:8089 for live monitoring — set to true to suppress
autostart	true	Starts the test automatically without waiting for input
autoquit	5	Exits automatically 5 seconds after the test completes
loglevel	INFO	Controls the verbosity of Locust logging
spawn-rate	5	Initial spawn-rate before the custom shape takes over

The test completes automatically after approximately 4 minutes. Compare performance metrics before, during, and after the spike to assess the platform's ability to absorb sudden traffic bursts and recover once demand returns to normal levels.

Stress Test

The stress test gradually increases concurrent load while continuously monitoring system performance. It is designed to identify the system’s breaking point and automatically stop the run when safe operating limits are exceeded.

During execution, Locust checks:

• p95 response time
• error rate (failures vs total requests)

If either threshold is breached, the test stops automatically using environment.runner.quit(). All results up to that point are still saved (CSV and HTML reports), so nothing is lost.

To avoid false positives during warm-up, threshold checks only start after a minimum request count per endpoint (e.g. 50 requests).

What you should do if a breach occurs

A breach means the system has exceeded acceptable performance limits under load. When this happens:

• Review the Locust reports to identify when degradation began
• Correlate the breach timestamp with the active user count in the UI or stats CSV
• Investigate scaling limits, bottlenecks, or dependency failures

Because the stop is driven by metrics rather than a fixed load level, you’ll need to use timestamps to determine the approximate concurrency at failure.

To run the tests, execute the below, replacing the --host value with the appropriate CLOUDFRONT_BASE_URL for different environments:

poetry run locust -f performance_tests/stress/uploader_stress_test.py --config=performance_tests/stress/stress.conf --host=https://uploader.ingest-dev.aws.onsdigital.uk

stress.conf

Like the spike test, the stress test uses a custom Locust LoadTestShape (shapes.py) to control how user numbers increase over time. The configuration file primarily controls execution behaviour and reporting.

Setting	Value	Description
locustfile	uploader_stress_test.py	Defines the Locust test script that contains the user behaviour and tasks executed during the stress test.
headless	false	Opens the Locust web UI at http://localhost:8089 for live monitoring — set to true to suppress
autostart	true	Starts the test automatically without waiting for input
autoquit	5	Exits automatically 5 seconds after the test completes
loglevel	INFO	Controls the verbosity of Locust logging
csv	results/stress	Writes test metrics to CSV files using the specified filename prefix

Viewing tests via Browser

To view the load tests in a browser, run the tests and open the Locust web UI at http://localhost:8089, allowing you to monitor results in real time.