Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 8 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,7 @@ This tool can be configured through two configuration file.
It supports the usual locust configuration variables documented on the [locust documentation](https://docs.locust.io/en/stable/configuration.html#configuration-file). On top of these, this tool exposes the following variables:

| Variable | Description | Default |
|---|---|---|
| --- | --- | --- |
| `env` | Name of the environment file to load at runtime (e.g. `local`, `prod`) | `local` |
| `competitions` | Space-separated list of competition names to test, matching the `<COMPETITION_NAME>` folders under `data/`. Omit to run all competitions found. | *(all)* |

Expand All @@ -94,22 +94,22 @@ It supports the usual locust configuration variables documented on the [locust d
This file will be loaded at runtime based on the `env` variable. It exposes the following variables:

| Variable | Description | Default | Required |
|---|---|---|---|
| --- | --- | --- | --- |
| `CODABENCH_HOST` | Target Codabench instance URL. It will override the host provided by locust| `http://localhost:8000` | Yes |
| `CODABENCH_CADDY_HOSTNAME` | Overrides the HTTP `Host` header when connecting via IP behind Caddy (e.g. `localhost` if Caddy's `DOMAIN_NAME` is `localhost`) | | No |
| `CODABENCH_API_TOKEN` | API token for authentication (takes priority over username/password if set) | | No* |
| `CODABENCH_CADDY_HOSTNAME` | Overrides the HTTP `Host` header when connecting via IP behind Caddy (e.g. `localhost` if Caddy's `DOMAIN_NAME` is `localhost`) | - | No |
| `CODABENCH_API_TOKEN` | API token for authentication (takes priority over username/password if set) | - | No* |
| `CODABENCH_USERNAME` | Username for authentication (used if no API token is set) | - | No* |
| `CODABENCH_PASSWORD` | Password for authentication (used if no API token is set) | | No* |
| `CODABENCH_PASSWORD` | Password for authentication (used if no API token is set) | - | No* |
| `CODABENCH_POLL_INTERVAL` | Delay (seconds) between submission status polls | `5.0` | No |
| `CODABENCH_POLL_TIMEOUT` | Max time (seconds) to wait for a submission to reach a terminal status | `3600.0` | No |
| `CODABENCH_MAX_RESPONSE_TIME_P95` | Performance threshold: max acceptable p95 response time (seconds) | `2.0` | No |
| `CODABENCH_MAX_ERROR_RATE` | Performance threshold: max acceptable error rate (0–1) | `0.01` | No |
| `CODABENCH_MINIO_ENDPOINT` | MinIO endpoint, for storage monitoring. Required even if MinIO isn't used, provide an empty string ("") in that case | `http://localhost:9000` | Yes |
| `CODABENCH_MINIO_ACCESS_KEY` | MinIO access key | | No |
| `CODABENCH_MINIO_SECRET_KEY` | MinIO secret key | | No |
| `CODABENCH_MINIO_ACCESS_KEY` | MinIO access key | - | No |
| `CODABENCH_MINIO_SECRET_KEY` | MinIO secret key | - | No |
| `CODABENCH_RABBITMQ_URL` | RabbitMQ management URL, for queue monitoring | `http://localhost:15672` | No |
| `CODABENCH_RABBITMQ_USER` | RabbitMQ username | `guest` | No |
| `CODABENCH_RABBITMQ_PASSWORD` | RabbitMQ password | | No |
| `CODABENCH_RABBITMQ_PASSWORD` | RabbitMQ password | - | No |

### How to manage the bundles ?

Expand Down
2 changes: 1 addition & 1 deletion cloud/aws/deploy/modules/paris/security_group.tf
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ resource "aws_vpc_security_group_egress_rule" "to_internet" {

# --- Ingress rules ---

# Locust master web UI (port 8089) only from within the Locust subnet
# Locust master web UI (port 8089) - only from within the Locust subnet
resource "aws_vpc_security_group_ingress_rule" "master_ui" {
security_group_id = aws_security_group.locust.id
cidr_ipv4 = var.locust_subnet_cidr
Expand Down
2 changes: 1 addition & 1 deletion cloud/aws/scripts/collect_results.sh
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ set -euo pipefail

RUN_NAME="${1:?Usage: $0 <run-name>}"
BUCKET="${RESULTS_BUCKET:-codabench-loadtest-results}"
# AWS CLI profile set AWS_PROFILE env var to use a non-default profile.
# AWS CLI profile - set AWS_PROFILE env var to use a non-default profile.
PROFILE="${AWS_PROFILE:-default}"
LOCAL_DIR="runs/${RUN_NAME}"

Expand Down
2 changes: 1 addition & 1 deletion cloud/aws/scripts/run_test.sh
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ set -euo pipefail
# - Terraform deployed (cloud/aws/deploy)
# - Run from the repo root

# AWS CLI profile set AWS_PROFILE env var to use a non-default profile.
# AWS CLI profile - set AWS_PROFILE env var to use a non-default profile.
PROFILE="${AWS_PROFILE:-default}"
TF_DIR="deploy"

Expand Down
2 changes: 1 addition & 1 deletion cloud/aws/scripts/stop_test.sh
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ set -euo pipefail
# - Terraform deployed (cloud/aws/deploy)
# - Run from the repo root

# AWS CLI profile set AWS_PROFILE env var to use a non-default profile.
# AWS CLI profile - set AWS_PROFILE env var to use a non-default profile.
PROFILE="${AWS_PROFILE:-default}"
TF_DIR="/deploy"
PARIS_ONLY=false
Expand Down
2 changes: 1 addition & 1 deletion codabench_loadtest/setup/config.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ class Settings(BaseSettings):
# hostname, e.g. "localhost", but we connect via IP).
caddy_hostname: str = "localhost:80"

# Authentication defaults empty so anonymous web/ scenarios work.
# Authentication - defaults empty so anonymous web/ scenarios work.
# Scenarios requiring auth must call require_auth() before use.
username: str = ""
password: SecretStr = SecretStr("")
Expand Down
12 changes: 6 additions & 6 deletions docs/cloud/aws/codabench.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,10 +60,10 @@ uses AWS SSM Session Manager (no SSH bastion needed).
| Resource | Name | Details |
| ---------- | ------ | --------- |
| VPC | `codabench-prodlike-vpc` | `10.0.0.0/16` |
| Public subnet 1 | `codabench-prodlike-public-1` | `10.0.1.0/24` `eu-west-1a` |
| Public subnet 2 | `codabench-prodlike-public-2` | `10.0.2.0/24` `eu-west-1b` |
| Private subnet 1 | `codabench-prodlike-private-1` | `10.0.11.0/24` `eu-west-1a` |
| Private subnet 2 | `codabench-prodlike-private-2` | `10.0.12.0/24` `eu-west-1b` |
| Public subnet 1 | `codabench-prodlike-public-1` | `10.0.1.0/24` - `eu-west-1a` |
| Public subnet 2 | `codabench-prodlike-public-2` | `10.0.2.0/24` - `eu-west-1b` |
| Private subnet 1 | `codabench-prodlike-private-1` | `10.0.11.0/24` - `eu-west-1a` |
| Private subnet 2 | `codabench-prodlike-private-2` | `10.0.12.0/24` - `eu-west-1b` |
| NAT Gateway | `codabench-prodlike-nat-gw` | In public subnet 1; gives private instances internet access |
| Internet Gateway | `codabench-prodlike-igw` | Attached to VPC for public subnets |

Expand Down Expand Up @@ -120,7 +120,7 @@ All security groups allow **all egress** (`0.0.0.0/0`, all protocols).
## Access

- **Web UI:** via ALB DNS on port 80 (output: `alb_dns`)
- **Admin/SSH:** via AWS SSM Session Manager all instances have the
- **Admin/SSH:** via AWS SSM Session Manager - all instances have the
`codabench-prodlike-ssm-role` IAM role with `AmazonSSMManagedInstanceCore`
- **MinIO Console:** via ALB at path `/minio-console`
- **MinIO API:** via the dedicated MinIO API ALB on port 80
Expand All @@ -136,7 +136,7 @@ aws ssm start-session --target <instance-id> --profile codabench
| ------ | --------- |
| `provider.tf` | AWS provider config, S3 backend |
| `variables.tf` | All input variables (CIDRs, instance types, credentials, images) |
| `main.tf` | Root module wires VPC, SGs, EC2, workers, ALBs, NAT gateway |
| `main.tf` | Root module - wires VPC, SGs, EC2, workers, ALBs, NAT gateway |
| `outputs.tf` | ALB DNS, Codabench IPs, MinIO/RabbitMQ endpoints |
| `modules/vpc/` | VPC, subnets, route tables, internet gateway |
| `modules/security_groups/` | All security groups and their rules |
Expand Down
14 changes: 7 additions & 7 deletions docs/cloud/aws/loadtest_infrastructure.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,13 +5,13 @@ nav_order: 1
---

This Terraform project deploys the load-testing infrastructure for Codabench
across three AWS regions. It creates the machines that **generate** the load
across three AWS regions. It creates the machines that **generate** the load -
the Codabench target infrastructure (the thing we're testing) is deployed
separately and already exists.

## What gets deployed

### Paris (eu-west-1) same VPC as Codabench
### Paris (eu-west-1) - same VPC as Codabench

Lives inside the existing Codabench VPC so it can reach the ALB internally
and monitor RabbitMQ directly.
Expand All @@ -22,20 +22,20 @@ and monitor RabbitMQ directly.
| Locust workers | ASG 2–6 × t3.medium | Generate HTTP load against the ALB |
| Playwright | 1 × t3.large | Measures perceived frontend latency with headless Chromium |

New private subnet: `10.0.21.0/24` internet access goes through the
New private subnet: `10.0.21.0/24` - internet access goes through the
existing NAT Gateway.

### US East (us-east-1) separate VPC
### US East (us-east-1) - separate VPC

| Resource | Type | Role |
| ---------- | ------ | ------ |
| Locust workers | ASG 2–4 × t3.medium | Headless load from the US |

New VPC `10.1.0.0/16` with a public subnet. Traffic goes over the **public
internet** to the ALB in Paris no VPC peering, because we want to measure
internet** to the ALB in Paris - no VPC peering, because we want to measure
real user latency from North America.

### Asia Pacific (ap-southeast-1) separate VPC
### Asia Pacific (ap-southeast-1) - separate VPC

| Resource | Type | Role |
| ---------- | ------ | ------ |
Expand All @@ -54,7 +54,7 @@ Same design as US East, VPC `10.2.0.0/16`. Measures real latency from Asia.

## Architecture diagram

![Architecture diagram](/docs/images/loadtest_architecture.jpg)
![Architecture diagram](/images/loadtest_architecture.jpg)

## Prerequisites

Expand Down
Loading