Skip to content

Load test environment

Jump to bottom
Wesley Hindle edited this page May 13, 2026 · 15 revisions

Load Test Environment

All work related to the load test env was conducted by Ben Corlett, who is no longer here, so we can't gain any articulation into the 'whys' of design decisions. No ADRs exist.

This page describes the load testing process we are moving to, based on:

  • Concourse-driven performance test jobs
  • a sanitised production-like database snapshot
  • a dedicated dev-loadtest environment configured for load testing

As of today parts are complete, and parts are still being wired to final state.

This page does not explain how Gatling works and assumes you understand how to deploy a dev env.

Current State vs Target State

Current state

  • Sanitise workflow is implemented and currently being tested in staging and prod.
  • dev-loadtest exists and has load-test-like settings (for example stubs enabled, Sentry disabled, production-like tuning in tfvars).
  • dev-loadtest snapshot source cutover is not yet finalised in tfvars. The PR is ready, but is dependent on a FT change in API (See next point).
  • notifications-api PR requires amendments FTs to allow the sanitised snap to pass FTs requires work. This involves:
  1. Checking if the current changes work when merged in as-is, the cache is cleared and the FTs pass consistently.
  2. If they do, there is nothing else to do.
  3. If they don't, then revert the change, clear the cache, ensure FTs pass. Amend the fixtures so the cache is auto cleared to prevent this issue.
  • Performance test jobs exist in deploy pipeline templates and should be enabled for dev-loadtest.

Target state

  • Sanitise workflow runs against production notifydb and snapshot is available.
  • Sanitised snapshot is shared and restored into dev-loadtest by default.

Current known issues

Weird functional test behaviour

Fix 'weird' behaviour with functional tests breaking when used with the sanitised database. The functional tests pass fine on a 'fresh' deploy of the load test environment, but they break if a load test has been run. The functional tests also break if we do a 'database cutover' from an unsanitised database to a sanitised database.

Given the load tests and database cutover don't change the database, we would not expect the functional tests to break. Passing FTs, broken FTs, 37-45 after load test has been run.

Load tests cause out-of-memory issues.

We would need to investigate which component is running out of memory, but we would not expect this to happen as the autoscaling would increase the amount of available memory.

Example

Load tests are not producing accurate results

One test suggests that Notify cannot handle 100 TPS, but our day-to-day running shows this is not the case.

Download this test report

We cannot debug the load tests which run in Concourse

They produce such large outputs that they cause your browser to crash, meaning it is impossible to debug. Users would need at least a machine with 16GB memory (it could be higher than this.)

Example

Load Testing Caveats

There is no perfect load test setup for Notify. We cannot fully simulate production traffic. Set expectations accordingly. Do not expect these tests to provide a full picture of production behavior.

We can improve these tests over time, but they will never be fully production-like.

Performance tests are not a tick-box Concourse activity where passing means everything is fine. Results must be examined and scrutinised.

This being said, here are some examples where you might find the load testing useful:

  • debugging complex issues that only happen under significant load
  • testing how the application autoscaling reacts to traffic
  • testing how different Gunicorn settings affect message delivery performance
  • testing how a particular change might behave under load, for example a database migration

High-Level Operator Flow

WARNING: Currently the changes are not merged into main. To deploy you must set the notifications-aws packbag into to this branch, which will deploy the prod snapshot, a prod-like DB instance, prod-like ECS task min/maxs.

notifications-api must be set to this branch.

If you are freshly deploying this environment ensure you have rebased these branches.

NOTE: You must only run one load test at a time. You must not run functional tests at the same time as load tests. Ensure you have the environment expectations set before running load tests.

  1. Bring up dev-loadtest like any other dev environment, except you must set the snapshot_identifier to the arn of the sanitised DB snapshot, here.
# dev-loadtest.tfvars
notifydb_rds_instances = {
  "notifydb" = {
    "snapshot_identifier" = "arn:aws:rds:eu-west-1:888450439860:snapshot:notifydb-sanitise-production" }}
  1. Manually kick off the deployment bag for the environment. Wait for completion.

If you are deploying a staging snapshot only, this will take ~1 hour. If you are deploying a fresh DB and a snapshot DB this will take ~2 hours. If you are deploying a prod snapshop only this will take ~2.5 hours.

  1. Run desired load test from desired group of tests. See this example group
  2. Download Gatling performance report zip.

The following jobs' report can be downloaded directly from the performance test job

  • performance-tests-send-sms-and-email-1-rps-60s
  • performance-tests-send-sms-and-email-50-rps-5m-ramp-2m-2-steps
  • performance-tests-send-letters-sms-and-email-1-rps-60s
  • performance-tests-send-letters-sms-and-email-50-rps-5m-ramp-2m-2-steps

These jobs' reports should be download from the performance-test-results job in the performance-testing-* tab. All test results going into the same job, so you should match the timestamp the loadtest finished, with the timestamp on the results' filename.

  • performance-tests-send-sms-and-email-200-rps-30m-ramp-10m-10-steps
  • performance-tests-send-sms-and-email-1000-rps-60m-ramp-20m-20-steps
  1. Destroy the environment by running the start destroy job

See Deploying a Notify Development Environment for more information on how to manage development environments.

Analyzing results

Depending on your use case, Gatling results may not be all the information you need. Gatling primarily shows client-side behavior. It does not give full visibility of backend activity, such as queue backlogs. To get a complete view, monitor Grafana, Kibana, and CloudWatch as needed.

Load test stubs

Our load tests typically send large volumes of simultaneous messages. At this scale we should not send real traffic to email and SMS providers. Requests to SES, Firetext, or MMG cost money, so dev-loadtest should normally remain in stub mode.

Be very careful when disabling stubs, especially after running a load test. There may still be many messages queued in SQS, and disabling stub mode too early can send those messages to real providers.

Common Load Test settings

  • Enabling and disabling stubs can be set here
  • Enabling performance concourse jobs can be set here
  • Setting the sanitised snapshot source can be set here. Add a restore instance under notifydb_rds_instances with snapshot_identifier set to the full ARN for the shared snapshot, validate it, then switch live_notifydb_rds_instance to that restore instance.

Database Sanitisation Workflow

The sanitise workflow is intentionally automated as a manually triggered operator job. This is expected to be a very long-running job because of database size. For staging this took 38 hours and 25 hours for prod.

Go to Concourse > staging | production team > operator tab > notifydb-sanitise job

At a high level, the sanitise job does this:

  1. Restore a fresh notifydb-sanitise-<env> from notifydb latest restorable point.
  2. Run sanitise-db ECS task (4 SQL parts) against that restore. See the SQL scripts here.
  3. Delete restored instance with a temporary final snapshot.
  4. Copy snapshot to a final named sanitised snapshot with a different KMS key (so we do not need to share the production snapshot KMS key with another environment).
  5. Share final snapshot with configured account IDs.
  6. Delete temporary snapshot.

Safety behavior in scripts includes:

  • hard checks that identifiers include sanitise before delete operations
  • hard checks that the database identifier has the word sanitise in it before running queries
  • waits with status logging for long RDS operations
  • auto-resume of a running ECS task if the Concourse job is interrupted (for example if a worker dies during a run)
  • cleanup handling when jobs fail
  • cleanup job to stop running sanitise tasks and remove transient resources
  • restricted permissions in IAM roles
  • restricted permissions in AWS Organizations SCP rules

Enabling Sanitisation Jobs in an environment

The sanitisation process should be enabled in production. In order to enable in other environments change the following settings:

dev-loadtest Environment Expectations

dev-loadtest should be configured for repeatable load testing, including:

  • provider stubs enabled by default
  • production-like app/scaling configuration where practical
  • performance test jobs available in Concourse pipeline
  • restore from sanitised snapshot source (staging during testing, then production)

If expected jobs are missing in Concourse, check the rendered deploy pipeline inputs and flags for the environment profile.

By default, dev-loadtest has the same ECS task min/max values as all other dev-* envs. These values must be changed to match those in prod to run a meaningful loadtest. The exception here being sms/email_provider_stub which should be set to ?

Logging into dev-loadtest

To log into dev-loadtest you may need to sign up. This process currently sends an email confirmation link. When the SES stub is enabled, you will not receive that email.

This means you may need to disable stub mode temporarily to complete initial login setup and use an OTP method other than SMS or email.

Before disabling stub mode, confirm SQS queues are drained. If you disable stubs while backlog still exists, queued notifications can be sent to real providers.

Possible future options to avoid this manual step include:

  • Update the ses-stub to log the message content. The log could then be used to gain the OTP code.
  • Add application options to disable all the verification
  • Add 'provider' option to notifications-api message send requests.

All of these options have downsides and should be considered carefully before implementation.

Fixtures And Performance Test Configuration

Performance tests in Concourse do not hardcode API keys/template IDs in pipeline config. They come from fixture-generated SSM environment data.

How it works:

  1. Terraform runs a Celery task via ECS from the notifications-api Docker image. The task creates or repairs canonical Functional Tests and Performance Tests data. See the Celery task code here
  2. It generates two env payloads:
    • functional test env
    • performance test env
  3. It uploads those payloads to SSM paths under /notify/<env>/.../generated/environment.
  4. Performance test task reads /notify/<env>/performance-tests/generated/environment and writes simulation.conf dynamically before running Gatling.

Performance env includes at least:

  • PERFORMANCE_TESTS_API_HOST
  • PERFORMANCE_TESTS_SERVICE_API_KEY
  • PERFORMANCE_TEST_SMS_TEMPLATE_ID
  • PERFORMANCE_TEST_EMAIL_TEMPLATE_ID
  • PERFORMANCE_TEST_EMAIL_WITH_FILE_TEMPLATE_ID
  • PERFORMANCE_TEST_LETTER_TEMPLATE_ID

This keeps credentials and test IDs environment-specific and avoids committing secrets in pipeline templates.

Converting Another Environment For Load Tests

If you need to temporarily use another environment as a load-test environment, apply the same principles used for dev-loadtest.

  1. Enable stubs for SMS and email before high-volume testing.
  2. Apply production-like scaling/settings where appropriate.
  3. Ensure performance test jobs are enabled in the environment's deploy pipeline.
  4. Ensure fixture generation runs so performance SSM config exists.
  5. Point the environment at a sanitised snapshot source appropriate for that phase (staging during test rollout, production once live).
  6. Run a low-rate smoke test first, then scale up.

Keep changes explicit and reversible, and record what was toggled so environment rollback is safe.

Stubs And Queue Safety

Provider stubs prevent real delivery and are a major cost/safety control during load testing.

Critical risk when toggling stubs:

  • You can finish a test and still have large backlogs in SQS queues.
  • If stubs are turned off while those queues are still draining, queued notifications can flow to real providers.
  • This can send a large volume of real SMS/email and incur significant cost.

Safe practice:

  1. Keep stubs enabled until queues are confirmed drained.
  2. Check SQS depths are zero (or expected baseline) before switching stubs off.
  3. Validate workers have finished processing backlog.
  4. Only then disable stubs.

Treat stub toggling as a change requiring explicit checks, not a routine switch.

Legacy EC2 Manual Gatling Process

The EC2-based manual Gatling process is deprecated.

  • Prefer Concourse-based tests for repeatability and shared visibility.
  • Manual EC2 Gatling runs are still possible for ad hoc investigation.
  • If running manually, follow existing guidance in Load-tests.md.

Concourse Jobs To Know

  • notifydb-sanitise: runs the sanitised snapshot workflow
  • cleanup-notifydb-sanitise: stops active sanitise task and cleans transient resources
  • performance-tests-*: runs Gatling simulation profiles and uploads results

Gotchas

Functional test errors

If you're using the notifydb-sanitise-staging created on April 13th 2026 then you will see functional test failures if you deploy the dev-loadtest env using this snapshot. This is because the personalisation used in the service called by the functional tests was not nullified, causing errors. This should only happen with this snapshot, as the PR to fix this has been merged.

If you need to fix it, then you must connect to the DB via the scripts/db-connect/db-connect.sh script. Then check the service_id personalisation used by the FTs has not been nullified i.e.

SELECT _personalisation from notifications
WHERE service_id='d6aa2c68-a2d9-4437-ab19-3ae8eb202553'
LIMIT 10;
                      _personalisation                       
-------------------------------------------------------------
 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
(10 rows)

And nullify this manually with

update notifications set _personalisation=NULL where service_id='d6aa2c68-a2d9-4437-ab19-3ae8eb202553';

Clicking on a performance test causes my browser to crash / system to slow up

This is because the performance test is long-running, so produces a huge Concourse job output, causing the issue. Instead, you should download your test report from the concourse-dev-loadtest-public-vvuvok in the notify-deploy account.

Clone this wiki locally