Skip to content

Performance Test Plan

mavrovde edited this page Nov 1, 2026 · 1 revision

Performance Test Plan

The test design for performance, capacity and resilience after the stack decision. It refines the Benchmark Plan and does not repeat it: the Benchmark Plan decides the stack in milestone 0.1.0; this page turns its harness into the standing performance suite of milestones 0.5.0 and 0.7.0 and the release gate. Part of the Testing Strategy.

Every figure is a design target from Spec: Non-Functional Requirements or an assumption (A-CAP-1). There are no measurements yet.

1. Objectives

Objective Requirement Scenario
Each endpoint class meets its p95 and p99 budget at design load NFR-PERF-003 load
No request runs past 2 s, and the storage query is cancelled at the deadline NFR-PERF-001, NFR-PERF-002 load, chaos
Repeated polls are 304 or cache hits at ≥ 95 % NFR-PERF-005 polling
50 identical concurrent requests cause exactly one storage read NFR-PERF-006 single-flight
No sequential scan of time-series data NFR-PERF-004 plan tests
Connections never exceed the pool and role limits NFR-CAP-002 load, stress
Above capacity, throughput stays flat and excess is refused within 50 ms NFR-CAP-003 stress, spike
Higher tiers keep their target while bulk is shed RATE-012, RATE-017 overload
Nothing leaks over time TST-007 soak
Faults degrade as specified NFR-RES-002 to NFR-RES-007, QAS-REL-01 to QAS-REL-03 chaos

2. Latency budgets per endpoint class

The budgets are those of NFR-PERF-003; they are repeated here only as the table the load assertions read.

Endpoint class p50 p95 p99 Hard deadline
Conditional hit (304) 5 ms 10 ms 30 ms 2 s
Lookup 40 ms 120 ms 300 ms 2 s
Near and motorway 60 ms 200 ms 500 ms 2 s
Live transit 80 ms 250 ms 600 ms 2 s
Along 120 ms 400 ms 900 ms 2 s
Composed 300 ms 800 ms 1 500 ms 2 s
History 150 ms 400 ms 1 000 ms 2 s

The budget file is one machine-readable file in the repository, generated from the specification table, so the test and the specification cannot disagree.

3. Scenarios

Scenario Load shape (design) Duration Pass criteria Runs
Smoke a few requests per operation under 1 minute no error, budgets not asserted every pull request
Load design load of 50 requests per second, benchmark request mix, open model 10 minutes after warm-up p95 and p99 per class within budget; 0 % errors nightly, release
Polling clients repeating the same request with If-None-Match every 30 s 10 minutes ≥ 95 % 304 or cache hits nightly
Single-flight 50 identical concurrent requests on a cold cache one burst, repeated exactly one storage read per burst nightly
Stress ramp from design load to several times it until the knee plus margin throughput flat after the knee; excess 429 within 50 ms; nothing past 2 s weekly, release
Spike 200 per second for 10 s (A-CAP-1 burst), then four times design load briefly minutes 429 with Retry-After; recovery to budget within one minute after the spike weekly, release
Overload a steady critical stream plus a bulk flood far above capacity, pool reduced 10 minutes critical p95 within budget; bulk shed first (RATE-017) nightly, release
Soak design load 1 hour resident memory grows ≤ 10 % after warm-up; no leaked connections; latency stable nightly (TST-007)

4. Resilience and chaos scenarios

Built in issue #19.

Fault Injection Expected behaviour Requirement
Slow storage latency proxy adds delay to every statement answers by the deadline; 503 deadline-exhausted when the budget is spent; backend idle within the cancel grace plus 100 ms NFR-PERF-002
Hanging query a stub query that sleeps 5 s cancelled at 2 s; 1 000 iterations without a poisoned connection QAS-REL-01, NFR-RES-007
Storage killed stop the database container breaker opens; 503 upstream-unavailable within 50 ms; permitted stale cache served with served_stale; recovery after breaker.open_for plus one probe NFR-RES-002, NFR-RES-004, QAS-REL-02
Pool exhausted hold every connection from outside admission refuses at once when the estimated wait exceeds the budget RATE-011
Access store down stop the access store keys in the key cache keep working; metering buffers up to its limit NFR-RES-005
One composed part times out latency injected for one family 200 with that part timeout, others ok QAS-REL-03, ERR-010
Instance killed mid-request kill the container under load restart and ready within 30 s; at most one metering interval lost QAS-REL-05
Clock skew skew the service clock quotas and freshness use the injected clock correctly; no negative ages RATE-003
Shutdown under load send the stop signal drain within server.shutdown_grace, flush metering, then exit NFR-RES-006

5. Method

Inherited from the Benchmark Plan and TEST-007: a fixed, production-shaped synthetic data set (benchmark profile, Test Data Management); an open-model load generator in its own container (k6 is the working choice); warm-up excluded; percentiles from full histograms, never averages of percentiles; equal container limits; each run writes a dated result file with the commit, image digest, configuration and seed.

Additions after the stack decision:

  • Repeat and variance. Each nightly load scenario runs three times; the median run is compared with the budget, and the spread is reported. A spread above 20 % (design) marks the run as noisy, not as passed.
  • Regression against main. Besides the absolute budget, p95 per class is compared with the previous main run; a regression above 10 % (design) opens an issue even inside the budget (TST-021).
  • Target guard. The runner refuses any host not on the test-environment allow-list. The production API is never a target.
  • Fail closed. A scenario that sent zero requests, wrote no result file or could not read its budget file fails.

6. Environment

The test stack only (Test Environments and Quality Gates): the release image at its design container limits, a throwaway PostGIS with the benchmark profile, the load generator and the fault proxy in separate containers, isolated from any production traffic.

7. Reporting

Each run adds to the release evidence: the scenario list, request counts, percentiles per class against budget, 304 rate, storage reads per single-flight burst, memory and connection curves for the soak, and every chaos assertion with its result. The release's test completion report summarises them.

Home

About

Design

Data Model

Technical specification

Testing

Engineering

Technology choice

Reference

Clone this wiki locally