Repository navigation
Spec Test Strategy
Part of the Technical Specification. Requirement prefix: TST. Extends Testing and Quality; acceptance and conformance are on Spec: Conformance and Acceptance.
A firm rule: tests never call a live data provider and never run against production. All tests use fixtures, throwaway databases and generated keys.
This page is the normative summary of testing: its TST rows are binding, and the pages of the testing document set elaborate them without contradicting them.
| Document (ISO/IEC/IEEE 29119-3) | Page |
|---|---|
| Organisational test strategy | [[Testing Strategy |
| Test plan, per milestone | [[Test Plan |
| Test design, by area | [[Contract and Schema Testing |
| Test data | [[Test Data Management |
| Environments and gates | [[Test Environments and Quality Gates |
| Test case and completion report templates | [[Test Documentation Templates |
| Defect reports | [[Defect Management |
flowchart TB
E2E[End to end: container against a fixture database<br/>few, slow, release gate]
CON[Contract: OpenAPI conformance, MCP parity, executed examples]
INT[Integration: storage mapping, roles, cancellation, query plans]
UNIT[Unit and property: domain, gate, limits, parsers<br/>many, fast]
E2E --- CON --- INT --- UNIT
| Layer | Share of tests (guideline) | Runs |
|---|---|---|
| Unit and property-based | about 70 % | every push |
| Integration (throwaway PostGIS from the contract stub) | about 20 % | every push |
| Contract and parity | about 8 % | every push |
| End to end, load, soak, chaos | about 2 % | pull request smoke, nightly full |
| Technique | Applied to | Target |
|---|---|---|
| Example-based unit tests | every domain rule | each rule has positive and negative cases |
| Property-based tests | kilometre arithmetic, area bounds, price ordering, cursor round-trip, Berlin rendering across daylight-saving changes, the licence gate over plan × family × acceptance | generators cover boundaries; shrinking enabled |
| Contract tests | every recorded response validated against the OpenAPI document; schema-driven generated requests against the running container | zero failures |
| MCP parity | each feature with a counterpart, on shared fixtures | 100 % fact equality |
| Fuzzing | cursor, key, polyline, location, fields, lang parsers and the configuration loader |
no crash; tamper refusal; a short run per pull request, a long run nightly |
| Load | the latency budgets per endpoint class and the overload scenario | NFR-PERF-003, RATE-017 |
| Soak | design load for 1 hour on a test stack | no memory growth beyond 10 % after warm-up, no leaked connections, latency stable |
| Chaos | storage killed, storage slowed, access store down, permits exhausted, clock skew, instance killed mid-request | behaviour per NFR-RES; no request past 2 s |
| Security tests | authentication matrix, gate matrix, log secrets sweep, header checks, TLS configuration scan, schema-driven negative testing | SEC, NFR-SEC |
| Mutation testing | domain, access and admission code | mutation score ≥ 80 % on changed code |
| Accessibility | documentation portal | WCAG 2.2 AA automated checks, plus a manual check per minor |
| ID | Requirement | Verification |
|---|---|---|
| TST-001 | Statement coverage MUST be 100 % in every package of every layer (domain, application, HTTP adapters and middleware, repositories, configuration, commands and the test kit), measured over the merged unit, integration and boot-smoke profiles. Generated code MAY be excluded only through a named exclusion file with a reason per entry; a stale or reasonless entry fails the gate. (Owner rule, 2026-11-02; it replaces the earlier 85 % overall and 95 % for domain, access and admission code.) | Lint |
| TST-002 | Mutation score MUST be at least 80 % on changed domain, access and admission code per pull request, and at least 70 % overall per minor release. | Lint |
| TST-003 | Every bug fix MUST add a test that fails without the fix. | Review |
| TST-004 | Tests MUST be deterministic: no wall-clock dependency (an injectable clock), no network, no shared mutable state between tests, a fixed random seed printed on failure. | Review |
| TST-005 | Tests MUST cover the date boundaries: UTC midnight, Berlin midnight, both daylight-saving transitions, month and year ends. | Unit |
| TST-006 | The test pipeline MUST itself be tested: planted faults (a lint error, a failing test, a removed response member, a leaked key in a log) MUST each turn the pipeline red. | Ops |
| TST-007 | Soak and chaos runs MUST run nightly on a test stack and report into the release evidence. | Ops |
| TST-008 | Flaky tests MUST be quarantined within one day and fixed or deleted within one week; a quarantined test does not count toward coverage. | Ops |
| TST-009 | Fixtures MUST be production-shaped but synthetic or licence-clean, and MUST contain no personal data and no secret. | Review |
| TST-010 | No test MAY reach a live data source, the production API or any host outside the test environment. CI test jobs MUST run with outbound network denied after dependency installation, and load, chaos, fuzzing and DAST runners MUST refuse any target not on the test-environment allow-list. | Security |
| TST-011 | Every quality gate MUST fail closed (an empty input, a missing tool, an unreadable result or a skipped step is a failure) and MUST print what it checked, as counts of the files, tests, operations, requirements or requests it read. Refines DEV-021. | Ops |
| TST-012 | Every test and every CI job MUST have a bounded timeout (design targets: 1 s per unit test, 2 s per handler test, 10 s per integration or end-to-end case). Requests made by tests MUST carry the production deadline of 2 s and MUST NOT raise it; deadline tests assert an answer by 2 s plus 50 ms. | Lint |
| TST-013 | A new or changed test or gate MUST be shown to fail before it is trusted: against the unfixed code for a defect, against at least one planted mutant otherwise. The pull request MUST record the red run. Extends TST-003. | Review |
| TST-014 | Test data MUST come from the seeded synthetic fixture generator ([[Test Data Management | Test-Data-Management]]): reproducible from a printed seed, Germany-wide in shape, fictional in content, with a manifest naming the licence of every input. Refines TST-009. |
| TST-015 | Contract conformance MUST be tested in both directions: every response a test receives MUST validate against the schema for its status, every request the OpenAPI document declares invalid MUST be refused with 400 invalid-parameter listing the violation, and every documented example request MUST be accepted. Refines TEST-003. |
Contract |
| TST-016 | Schema-driven property-based fuzzing of a running test instance from the OpenAPI document MUST run on every pull request with a short budget and nightly with a long budget; a 5xx outside the documented causes, a schema violation or an answer later than 2 s plus 50 ms fails. |
Security |
| TST-017 | Automatic retries of failing tests MUST NOT be configured in any CI job or test runner; an intermittently failing test is handled by quarantine (TST-008). | Lint |
| TST-018 | Every quality gate MUST be declared with its stage and its blocking status ([[Test Environments and Quality Gates | Test-Environments-and-Quality-Gates]]); a blocking gate MUST NOT be made non-blocking without a recorded decision. |
| TST-019 | Every milestone MUST have a test plan section with scope, requirement IDs, entry, exit, suspension and resumption criteria ([[Test Plan | Test-Plan]]), and every release MUST publish a test completion report from the template, referenced by the release acceptance (TEST-011). |
| TST-020 | Every defect fix and every escaped defect MUST get a regression test at the lowest test level able to catch it; the completion report MUST name, for each escaped defect, the level that should have caught it. | Review |
| TST-021 | The nightly load run MUST compare p95 and p99 per endpoint class with the budget of NFR-PERF-003 and with the previous run on main; a budget breach fails the run, and a regression above 10 % (design target) inside the budget opens an issue. | Load |
| TST-022 | CI ephemeral test environments MUST be created from their committed definition for each run and destroyed on every exit path, including failure and cancellation; no state MAY be shared between runs. | Ops |
ViaFrei API: a public, keyed, read-only REST API to live German mobility data. Design phase; nothing on these pages is a measurement of a running service. Data on viafrei.de and in every API answer is attributed to its publishers under their licences.
About
Design
- Domain Model
- API Design
- Architecture Overview
- Security and Access Keys
- Rate Limits and Priority
- Performance and Caching
- Configuration
- Observability
Data Model
Technical specification
-
Technical Specification
- Interface
- Scope, Conventions and References
- Protocol
- Common Envelope
- Resource Model
- Endpoints, Places
- Endpoints, Roads
- Endpoints, Energy and Parking
- Endpoints, Public Transport and Accessibility
- Endpoints, Weather Risk, Driving Rules and Composed Features
- Service Endpoints
- Errors
- Authentication and Authorisation
- Rate Limiting, Quotas and Priority
- Data Licensing and Attribution
- Compatibility and Versioning
- OpenAPI Skeleton
- Quality
- Delivery and operations
- Interface
Testing
- Testing Strategy
- Testing and Quality (design summary)
Engineering
Technology choice
- Technology Evaluation
- Architecture: Go
- Architecture: Rust
- Architecture: TypeScript
- Stack Comparison
- Benchmark Plan
Reference