Load test environment

Load Test Environment

All work related to the load test env was conducted by Ben Corlett, who is no longer here, so we can't gain any articulation into the 'whys' of design decisions. No ADRs exist.

This page describes the load testing process we are moving to, based on:

• Concourse-driven performance test jobs
• a sanitised production-like database snapshot
• a dedicated dev-loadtest environment configured for load testing

As of today parts are complete, and parts are still being wired to final state.

This page does not explain how Gatling works and assumes you understand how to deploy a dev env.

Current State vs Target State

Current state

• Sanitise workflow is implemented and currently being tested in staging
• dev-loadtest exists and has load-test-like settings (for example stubs enabled, Sentry disabled, production-like tuning in tfvars).
• dev-loadtest snapshot source cutover is not yet finalised in tfvars. Add a restore entry under notifydb_rds_instances (for example notifydb-2) with snapshot_identifier set to the full snapshot ARN. Keep live_notifydb_rds_instance = "notifydb" until the restore is validated, then switch live_notifydb_rds_instance to the restore instance.
• Performance test jobs exist in deploy pipeline templates and should be enabled for dev-loadtest.

Target state

• Sanitise workflow runs against production notifydb and snapshot is available.
• Sanitised snapshot is shared and restored into dev-loadtest by default.

Load Testing Caveats

There is no perfect load test setup for Notify. We cannot fully simulate production traffic. Set expectations accordingly. Do not expect these tests to provide a full picture of production behavior.

We can improve these tests over time, but they will never be fully production-like.

Performance tests are not a tick-box Concourse activity where passing means everything is fine. Results must be examined and scrutinised.

This being said, here are some examples where you might find the load testing useful:

• debugging complex issues that only happen under significant load
• testing how the application autoscaling reacts to traffic
• testing how different Gunicorn settings affect message delivery performance
• testing how a particular change might behave under load, for example a database migration

High-Level Operator Flow

1. Bring up dev-loadtest like any other dev environment, except you must set the snapshot_identifier to the arn of the sanitised DB snapshot, here.

link needed

2. Manually kick off the deployment bag for the environment. Wait for completion, this will take over an hour.
3. Run desired load test from desired group of tests. See this example group
4. Download Gatling performance report zip from the Concourse job.
5. Destroy the environment by running the start destroy job

See Deploying a Notify Development Environment for more information on how to manage development environments.

Analyzing results

Depending on your use case, Gatling results may not be all the information you need. Gatling primarily shows client-side behavior. It does not give full visibility of backend activity, such as queue backlogs. To get a complete view, monitor Grafana, Kibana, and CloudWatch as needed.

Load test stubs

Our load tests typically send large volumes of simultaneous messages. At this scale we should not send real traffic to email and SMS providers. Requests to SES, Firetext, or MMG cost money, so dev-loadtest should normally remain in stub mode.

Be very careful when disabling stubs, especially after running a load test. There may still be many messages queued in SQS, and disabling stub mode too early can send those messages to real providers.

Common Load Test settings

• Enabling and disabling stubs can be set here
• Enabling performance concourse jobs can be set here
• Setting the sanitised snapshot source can be set here. Add a restore instance under notifydb_rds_instances with snapshot_identifier set to the full ARN for the shared snapshot, validate it, then switch live_notifydb_rds_instance to that restore instance.

Database Sanitisation Workflow

The sanitise workflow is intentionally automated as a manually triggered operator job.
Go to Concourse > staging | production team > operator tab > notifydb-sanitise
This is expected to be a very long-running job because of database size. For staging this took 38 hours.

At a high level, the sanitise job does this:

1. Restore a fresh notifydb-sanitise-<env> from notifydb latest restorable point.
2. Run sanitise-db ECS task (4 SQL parts) against that restore. See the SQL scripts here.
3. Delete restored instance with a temporary final snapshot.
4. Copy snapshot to a final named sanitised snapshot with a different KMS key (so we do not need to share the production snapshot KMS key with another environment).
5. Share final snapshot with configured account IDs.
6. Delete temporary snapshot.

Safety behavior in scripts includes:

• hard checks that identifiers include sanitise before delete operations
• hard checks that the database identifier has the word sanitise in it before running queries
• waits with status logging for long RDS operations
• auto-resume of a running ECS task if the Concourse job is interrupted (for example if a worker dies during a run)
• cleanup handling when jobs fail
• cleanup job to stop running sanitise tasks and remove transient resources
• restricted permissions in IAM roles
• restricted permissions in AWS Organizations SCP rules

Enabling Sanitisation Jobs in an environment

The sanitisation process should be enabled in production. In order to enable in other environments change the following settings:

• pipeline setting
• terraform setting

dev-loadtest Environment Expectations

dev-loadtest should be configured for repeatable load testing, including:

• provider stubs enabled by default
• production-like app/scaling configuration where practical
• performance test jobs available in Concourse pipeline
• restore from sanitised snapshot source (staging during testing, then production)

If expected jobs are missing in Concourse, check the rendered deploy pipeline inputs and flags for the environment profile.

Logging into dev-loadtest

To log into dev-loadtest you may need to sign up. This process currently sends an email confirmation link. When the SES stub is enabled, you will not receive that email.

This means you may need to disable stub mode temporarily to complete initial login setup and use an OTP method other than SMS or email.

Before disabling stub mode, confirm SQS queues are drained. If you disable stubs while backlog still exists, queued notifications can be sent to real providers.

Possible future options to avoid this manual step include:

• Update the ses-stub to log the message content. The log could then be used to gain the OTP code.
• Add application options to disable all the verification
• Add 'provider' option to notifications-api message send requests.

All of these options have downsides and should be considered carefully before implementation.

Fixtures And Performance Test Configuration

Performance tests in Concourse do not hardcode API keys/template IDs in pipeline config. They come from fixture-generated SSM environment data.

How it works:

1. Terraform runs a Celery task via ECS from the notifications-api Docker image. The task creates or repairs canonical Functional Tests and Performance Tests data. See the Celery task code here
2. It generates two env payloads:
   • functional test env
   • performance test env
3. It uploads those payloads to SSM paths under /notify/<env>/.../generated/environment.
4. Performance test task reads /notify/<env>/performance-tests/generated/environment and writes simulation.conf dynamically before running Gatling.

Performance env includes at least:

• PERFORMANCE_TESTS_API_HOST
• PERFORMANCE_TESTS_SERVICE_API_KEY
• PERFORMANCE_TEST_SMS_TEMPLATE_ID
• PERFORMANCE_TEST_EMAIL_TEMPLATE_ID
• PERFORMANCE_TEST_EMAIL_WITH_FILE_TEMPLATE_ID
• PERFORMANCE_TEST_LETTER_TEMPLATE_ID

This keeps credentials and test IDs environment-specific and avoids committing secrets in pipeline templates.

Converting Another Environment For Load Tests

If you need to temporarily use another environment as a load-test environment, apply the same principles used for dev-loadtest.

1. Enable stubs for SMS and email before high-volume testing.
2. Apply production-like scaling/settings where appropriate.
3. Ensure performance test jobs are enabled in the environment's deploy pipeline.
4. Ensure fixture generation runs so performance SSM config exists.
5. Point the environment at a sanitised snapshot source appropriate for that phase (staging during test rollout, production once live).
6. Run a low-rate smoke test first, then scale up.

Keep changes explicit and reversible, and record what was toggled so environment rollback is safe.

Stubs And Queue Safety

Provider stubs prevent real delivery and are a major cost/safety control during load testing.

Critical risk when toggling stubs:

• You can finish a test and still have large backlogs in SQS queues.
• If stubs are turned off while those queues are still draining, queued notifications can flow to real providers.
• This can send a large volume of real SMS/email and incur significant cost.

Safe practice:

1. Keep stubs enabled until queues are confirmed drained.
2. Check SQS depths are zero (or expected baseline) before switching stubs off.
3. Validate workers have finished processing backlog.
4. Only then disable stubs.

Treat stub toggling as a change requiring explicit checks, not a routine switch.

Legacy EC2 Manual Gatling Process

The EC2-based manual Gatling process is deprecated.

• Prefer Concourse-based tests for repeatability and shared visibility.
• Manual EC2 Gatling runs are still possible for ad hoc investigation.
• If running manually, follow existing guidance in Load-tests.md.

Concourse Jobs To Know

• notifydb-sanitise: runs the sanitised snapshot workflow
• cleanup-notifydb-sanitise: stops active sanitise task and cleans transient resources
• performance-tests-*: runs Gatling simulation profiles and uploads results

Gotchas

Functional test errors

If you're using the notifydb-sanitise-staging created on April 13th 2026 then you will see functional test failures if you deploy the dev-loadtest env using this snapshot. This is because the personalisation used in the service called by the functional tests was not nullified, causing errors. This should only happen with this snapshot, as the PR to fix this has been merged.