LavinMQ Benchmark Tool

Infrastructure-as-code benchmarking tool for LavinMQ, using Terraform to provision AWS resources and lavinmqperf to run performance tests.

Prerequisites

• Terraform >= 1.3.0 — installation guide
• AWS credentials with permission to create EC2 and VPC resources
• An SSH key pair for instance access

Scenarios

lavinmq_throughput

Provisions a broker and one or more load generator instances, then runs a single user-supplied lavinmqperf command. Useful for ad-hoc throughput or latency measurements where you control the exact test parameters.

multiple_run_throughput

Runs sequential AMQP throughput tests across a configurable set of message sizes, repeated N times. Produces a CSV file of raw per-run results and a JSON config file on the load generator. Aggregated (median) summaries are written to results/v{version}/throughput.md by scripts/aggregate_results.py.

multiple_run_latency

Runs sequential AMQP latency tests across a configurable set of message sizes and rate limits, repeated N times. Produces a CSV file of raw per-run results and a JSON config file on the load generator. Aggregated (median) summaries are written to results/v{version}/latency_P95.md and latency_P99.md by scripts/aggregate_results.py.

mqtt_throughput

Runs sequential MQTT throughput tests across a configurable set of message sizes, repeated N times. Installs emqtt-bench and mqttloader on the load generator. Produces a CSV file of raw per-run results and a JSON config file on the load generator. Aggregated (median) summaries are written to results/v{version}/mqtt_throughput.md by scripts/aggregate_results.py.

Configuration

Variables can be supplied in three ways:

terraform.auto.tfvars — loaded automatically by Terraform:

terraform apply

terraform.tfvars — loaded explicitly:

terraform apply -var-file="terraform.tfvars"

.env file via dotenv — environment variables prefixed with TF_VAR_:

dotenv terraform apply

Use the templates in modules/providers/aws/variables_template/ as a starting point:

• terraform_tfvars.txt — for .tfvars files
• env.txt — for .env files

AWS credentials must be set as environment variables regardless of the method used:

export AWS_ACCESS_KEY=***
export AWS_SECRET_KEY=***

CI / Parallel Benchmarks

The Benchmark GitHub Actions workflow provisions infrastructure, runs benchmarks across all supported instance types in parallel, aggregates the results into markdown summary files, and opens a pull request against main with the results committed to a results/v{version} branch.

See docs/pipeline.md for diagrams of the AWS infrastructure and the CI workflow, from release trigger to published results.

Triggering a run

Via the GitHub UI: go to Actions → Benchmark → Run workflow, fill in the version and pick a scenario.

Via the GitHub CLI:

# Run all benchmarks (AMQP throughput, AMQP latency, MQTT throughput) for all instance types
gh workflow run benchmark.yml \
  -f broker_version=2.7.0 \
  -f scenarios=all

# Latency only
gh workflow run benchmark.yml \
  -f broker_version=2.7.0 \
  -f scenarios=latency

# AMQP throughput only
gh workflow run benchmark.yml \
  -f broker_version=2.7.0 \
  -f scenarios=throughput

# MQTT throughput only
gh workflow run benchmark.yml \
  -f broker_version=2.7.0 \
  -f scenarios=mqtt-throughput

# Pin a specific load generator version (leave empty to use latest stable)
gh workflow run benchmark.yml \
  -f broker_version=2.7.0 \
  -f load_generator_version=2.7.0 \
  -f scenarios=all

# Run from a specific branch (e.g. when testing workflow changes)
gh workflow run benchmark.yml \
  -r my-branch \
  -f broker_version=2.7.0 \
  -f scenarios=all

Results are committed to results/v{version}/ and a pull request is created (or updated if one already exists for that version).

Re-running specific broker instance types

All three workflows accept an optional brokers input — a comma-separated list of broker instance types to run. When omitted or empty, all instance types are benchmarked. This is useful for re-running a single instance type that failed without triggering the full suite.

# Re-run latency benchmark for a single instance type
gh workflow run benchmark.yml \
  -f broker_version=2.7.0 \
  -f scenarios=latency \
  -f brokers="c8g.large"

# Re-run latency benchmark for multiple specific instance types
gh workflow run benchmark.yml \
  -f broker_version=2.7.0 \
  -f scenarios=latency \
  -f brokers="c8g.large,t4g.medium"

The individual benchmark-latency.yml, benchmark-throughput.yml, and benchmark-mqtt-throughput.yml workflows can also be triggered directly in the same way if you want to skip the aggregate/PR step.

Manual cleanup after a failed run

Each job runs terraform destroy with if: always(), but if a job is cancelled, times out, or the destroy step itself fails, the AWS resources it created (VPC, subnet, internet gateway, EC2 instances, ENIs, key pair) are left running. Since the Terraform state lives only on the ephemeral runner, it's gone too, so cleanup has to be done directly via the AWS console/CLI rather than by re-running terraform destroy.

Leftover resources are tagged CreatedBy = github-actions. Delete them in this order, since later resources depend on earlier ones being gone:

1. Terminate the 2 EC2 instances (broker + load generator). Their network interfaces are attached to the subnet/VPC and block deletion otherwise.
2. Wait until both show terminated.
3. Detach the Internet Gateway from the VPC.
4. Delete the Internet Gateway.
5. Delete the network interfaces (ENIs) belonging to the two terminated instances, these aren't always cleaned up automatically on termination and will block subnet/VPC deletion if left.
6. Delete the subnet.
7. Delete the VPC, this implicitly removes the default route table entries, default security group rules, and the main route table association created in network.tf. No separate delete needed for those.
8. Delete the SSH key pair (benchmark-ci-latency-<run_id>-<broker_safe> or equivalent for the other scenarios), independent of the above, can be done any time.

Network limits

At larger message sizes the throughput test can outrun an instance's sustained AWS network bandwidth once its burst allowance is spent, giving unreliable numbers. Two configs keep that out of the published charts (raw CSVs stay in git):

• scripts/throughput_skip_sizes.json — sizes skipped per instance (t4g.micro, r7g.medium at 65536 B). The workflow skips them on new runs; build_data.py also redacts them from old data.
• scripts/website_hidden_instances.json — instances still benchmarked but hidden from all charts (z1d.large, whose large-message throughput is bandwidth-bound, not LavinMQ-bound).

Latency is unaffected: its per-size rate limits keep demand below every baseline. See issue #90.

Publishing results

Raw results are stored and available in this repository. These are also available at https://lavinmq.com/benchmark with graphs and a UI to easier compare the numbers.

Aggregated JSONs under results/ are published to benchmark.lavinmq.com and consumed by the charts page in lavinmq-website.

To preview a local copy, start jekyll in lavinmq-website and run the following here:

python3 scripts/build_data.py        # regenerate throughput/latency/mqtt_throughput JSONs
python3 scripts/serve.py             # serve results/ on http://127.0.0.1:8081 with CORS

AI-assisted result comparisons

.github/instructions/benchmark-comparison.instructions.md guides AI coding agents (e.g. GitHub Copilot, Claude Code) through comparing benchmark summaries across releases, checking for missing or invalid runs, accounting for environment differences, and producing a concise writeup suitable for a PR description or comment. Ask your agent to compare results under results/ (e.g. "compare v2.9.3 and v2.10.0 latency results") and it will pick up these instructions automatically.

Logging

Enable detailed Terraform provider logs by setting:

export TF_LOG_PROVIDER=DEBUG

Accepted levels: INFO, DEBUG, WARN, ERROR, TRACE.