Repository navigation
Performance Test Plan
The test design for performance, capacity and resilience after the stack decision. It refines the Benchmark Plan and does not repeat it: the Benchmark Plan decides the stack in milestone 0.1.0; this page turns its harness into the standing performance suite of milestones 0.5.0 and 0.7.0 and the release gate. Part of the Testing Strategy.
Every figure is a design target from Spec: Non-Functional Requirements or an assumption (A-CAP-1). There are no measurements yet.
| Objective | Requirement | Scenario |
|---|---|---|
| Each endpoint class meets its p95 and p99 budget at design load | NFR-PERF-003 | load |
| No request runs past 2 s, and the storage query is cancelled at the deadline | NFR-PERF-001, NFR-PERF-002 | load, chaos |
Repeated polls are 304 or cache hits at ≥ 95 % |
NFR-PERF-005 | polling |
| 50 identical concurrent requests cause exactly one storage read | NFR-PERF-006 | single-flight |
| No sequential scan of time-series data | NFR-PERF-004 | plan tests |
| Connections never exceed the pool and role limits | NFR-CAP-002 | load, stress |
| Above capacity, throughput stays flat and excess is refused within 50 ms | NFR-CAP-003 | stress, spike |
| Higher tiers keep their target while bulk is shed | RATE-012, RATE-017 | overload |
| Nothing leaks over time | TST-007 | soak |
| Faults degrade as specified | NFR-RES-002 to NFR-RES-007, QAS-REL-01 to QAS-REL-03 | chaos |
The budgets are those of NFR-PERF-003; they are repeated here only as the table the load assertions read.
| Endpoint class | p50 | p95 | p99 | Hard deadline |
|---|---|---|---|---|
Conditional hit (304) |
5 ms | 10 ms | 30 ms | 2 s |
| Lookup | 40 ms | 120 ms | 300 ms | 2 s |
| Near and motorway | 60 ms | 200 ms | 500 ms | 2 s |
| Live transit | 80 ms | 250 ms | 600 ms | 2 s |
| Along | 120 ms | 400 ms | 900 ms | 2 s |
| Composed | 300 ms | 800 ms | 1 500 ms | 2 s |
| History | 150 ms | 400 ms | 1 000 ms | 2 s |
The budget file is one machine-readable file in the repository, generated from the specification table, so the test and the specification cannot disagree.
| Scenario | Load shape (design) | Duration | Pass criteria | Runs |
|---|---|---|---|---|
| Smoke | a few requests per operation | under 1 minute | no error, budgets not asserted | every pull request |
| Load | design load of 50 requests per second, benchmark request mix, open model | 10 minutes after warm-up | p95 and p99 per class within budget; 0 % errors | nightly, release |
| Polling | clients repeating the same request with If-None-Match every 30 s |
10 minutes | ≥ 95 % 304 or cache hits |
nightly |
| Single-flight | 50 identical concurrent requests on a cold cache | one burst, repeated | exactly one storage read per burst | nightly |
| Stress | ramp from design load to several times it | until the knee plus margin | throughput flat after the knee; excess 429 within 50 ms; nothing past 2 s |
weekly, release |
| Spike | 200 per second for 10 s (A-CAP-1 burst), then four times design load briefly | minutes |
429 with Retry-After; recovery to budget within one minute after the spike |
weekly, release |
| Overload | a steady critical stream plus a bulk flood far above capacity, pool reduced | 10 minutes | critical p95 within budget; bulk shed first (RATE-017) | nightly, release |
| Soak | design load | 1 hour | resident memory grows ≤ 10 % after warm-up; no leaked connections; latency stable | nightly (TST-007) |
Built in issue #19.
| Fault | Injection | Expected behaviour | Requirement |
|---|---|---|---|
| Slow storage | latency proxy adds delay to every statement | answers by the deadline; 503 deadline-exhausted when the budget is spent; backend idle within the cancel grace plus 100 ms |
NFR-PERF-002 |
| Hanging query | a stub query that sleeps 5 s | cancelled at 2 s; 1 000 iterations without a poisoned connection | QAS-REL-01, NFR-RES-007 |
| Storage killed | stop the database container | breaker opens; 503 upstream-unavailable within 50 ms; permitted stale cache served with served_stale; recovery after breaker.open_for plus one probe |
NFR-RES-002, NFR-RES-004, QAS-REL-02 |
| Pool exhausted | hold every connection from outside | admission refuses at once when the estimated wait exceeds the budget | RATE-011 |
| Access store down | stop the access store | keys in the key cache keep working; metering buffers up to its limit | NFR-RES-005 |
| One composed part times out | latency injected for one family |
200 with that part timeout, others ok
|
QAS-REL-03, ERR-010 |
| Instance killed mid-request | kill the container under load | restart and ready within 30 s; at most one metering interval lost | QAS-REL-05 |
| Clock skew | skew the service clock | quotas and freshness use the injected clock correctly; no negative ages | RATE-003 |
| Shutdown under load | send the stop signal | drain within server.shutdown_grace, flush metering, then exit |
NFR-RES-006 |
Inherited from the Benchmark Plan and TEST-007: a fixed, production-shaped synthetic data set (benchmark profile, Test Data Management); an open-model load generator in its own container (k6 is the working choice); warm-up excluded; percentiles from full histograms, never averages of percentiles; equal container limits; each run writes a dated result file with the commit, image digest, configuration and seed.
Additions after the stack decision:
- Repeat and variance. Each nightly load scenario runs three times; the median run is compared with the budget, and the spread is reported. A spread above 20 % (design) marks the run as noisy, not as passed.
- Regression against main. Besides the absolute budget, p95 per class is compared with the previous main run; a regression above 10 % (design) opens an issue even inside the budget (TST-021).
- Target guard. The runner refuses any host not on the test-environment allow-list. The production API is never a target.
- Fail closed. A scenario that sent zero requests, wrote no result file or could not read its budget file fails.
The test stack only (Test Environments and Quality Gates): the release image at its design container limits, a throwaway PostGIS with the benchmark profile, the load generator and the fault proxy in separate containers, isolated from any production traffic.
Each run adds to the release evidence: the scenario list, request counts, percentiles per class against budget, 304 rate, storage reads per single-flight burst, memory and connection curves for the soak, and every chaos assertion with its result. The release's test completion report summarises them.
ViaFrei API: a public, keyed, read-only REST API to live German mobility data. Design phase; nothing on these pages is a measurement of a running service. Data on viafrei.de and in every API answer is attributed to its publishers under their licences.
About
Design
- Domain Model
- API Design
- Architecture Overview
- Security and Access Keys
- Rate Limits and Priority
- Performance and Caching
- Configuration
- Observability
Data Model
Technical specification
-
Technical Specification
- Interface
- Scope, Conventions and References
- Protocol
- Common Envelope
- Resource Model
- Endpoints, Places
- Endpoints, Roads
- Endpoints, Energy and Parking
- Endpoints, Public Transport and Accessibility
- Endpoints, Weather Risk, Driving Rules and Composed Features
- Service Endpoints
- Errors
- Authentication and Authorisation
- Rate Limiting, Quotas and Priority
- Data Licensing and Attribution
- Compatibility and Versioning
- OpenAPI Skeleton
- Quality
- Delivery and operations
- Interface
Testing
- Testing Strategy
- Testing and Quality (design summary)
Engineering
Technology choice
- Technology Evaluation
- Architecture: Go
- Architecture: Rust
- Architecture: TypeScript
- Stack Comparison
- Benchmark Plan
Reference