Skip to content

Spec Test Strategy

mavrovde edited this page Nov 2, 2026 · 3 revisions

Spec: Test Strategy

Part of the Technical Specification. Requirement prefix: TST. Extends Testing and Quality; acceptance and conformance are on Spec: Conformance and Acceptance.

A firm rule: tests never call a live data provider and never run against production. All tests use fixtures, throwaway databases and generated keys.

This page is the normative summary of testing: its TST rows are binding, and the pages of the testing document set elaborate them without contradicting them.

Document (ISO/IEC/IEEE 29119-3) Page
Organisational test strategy [[Testing Strategy
Test plan, per milestone [[Test Plan
Test design, by area [[Contract and Schema Testing
Test data [[Test Data Management
Environments and gates [[Test Environments and Quality Gates
Test case and completion report templates [[Test Documentation Templates
Defect reports [[Defect Management

1. The pyramid

flowchart TB
  E2E[End to end: container against a fixture database<br/>few, slow, release gate]
  CON[Contract: OpenAPI conformance, MCP parity, executed examples]
  INT[Integration: storage mapping, roles, cancellation, query plans]
  UNIT[Unit and property: domain, gate, limits, parsers<br/>many, fast]
  E2E --- CON --- INT --- UNIT
Loading
Layer Share of tests (guideline) Runs
Unit and property-based about 70 % every push
Integration (throwaway PostGIS from the contract stub) about 20 % every push
Contract and parity about 8 % every push
End to end, load, soak, chaos about 2 % pull request smoke, nightly full

2. Techniques

Technique Applied to Target
Example-based unit tests every domain rule each rule has positive and negative cases
Property-based tests kilometre arithmetic, area bounds, price ordering, cursor round-trip, Berlin rendering across daylight-saving changes, the licence gate over plan × family × acceptance generators cover boundaries; shrinking enabled
Contract tests every recorded response validated against the OpenAPI document; schema-driven generated requests against the running container zero failures
MCP parity each feature with a counterpart, on shared fixtures 100 % fact equality
Fuzzing cursor, key, polyline, location, fields, lang parsers and the configuration loader no crash; tamper refusal; a short run per pull request, a long run nightly
Load the latency budgets per endpoint class and the overload scenario NFR-PERF-003, RATE-017
Soak design load for 1 hour on a test stack no memory growth beyond 10 % after warm-up, no leaked connections, latency stable
Chaos storage killed, storage slowed, access store down, permits exhausted, clock skew, instance killed mid-request behaviour per NFR-RES; no request past 2 s
Security tests authentication matrix, gate matrix, log secrets sweep, header checks, TLS configuration scan, schema-driven negative testing SEC, NFR-SEC
Mutation testing domain, access and admission code mutation score ≥ 80 % on changed code
Accessibility documentation portal WCAG 2.2 AA automated checks, plus a manual check per minor

Requirements

ID Requirement Verification
TST-001 Statement coverage MUST be 100 % in every package of every layer (domain, application, HTTP adapters and middleware, repositories, configuration, commands and the test kit), measured over the merged unit, integration and boot-smoke profiles. Generated code MAY be excluded only through a named exclusion file with a reason per entry; a stale or reasonless entry fails the gate. (Owner rule, 2026-11-02; it replaces the earlier 85 % overall and 95 % for domain, access and admission code.) Lint
TST-002 Mutation score MUST be at least 80 % on changed domain, access and admission code per pull request, and at least 70 % overall per minor release. Lint
TST-003 Every bug fix MUST add a test that fails without the fix. Review
TST-004 Tests MUST be deterministic: no wall-clock dependency (an injectable clock), no network, no shared mutable state between tests, a fixed random seed printed on failure. Review
TST-005 Tests MUST cover the date boundaries: UTC midnight, Berlin midnight, both daylight-saving transitions, month and year ends. Unit
TST-006 The test pipeline MUST itself be tested: planted faults (a lint error, a failing test, a removed response member, a leaked key in a log) MUST each turn the pipeline red. Ops
TST-007 Soak and chaos runs MUST run nightly on a test stack and report into the release evidence. Ops
TST-008 Flaky tests MUST be quarantined within one day and fixed or deleted within one week; a quarantined test does not count toward coverage. Ops
TST-009 Fixtures MUST be production-shaped but synthetic or licence-clean, and MUST contain no personal data and no secret. Review
TST-010 No test MAY reach a live data source, the production API or any host outside the test environment. CI test jobs MUST run with outbound network denied after dependency installation, and load, chaos, fuzzing and DAST runners MUST refuse any target not on the test-environment allow-list. Security
TST-011 Every quality gate MUST fail closed (an empty input, a missing tool, an unreadable result or a skipped step is a failure) and MUST print what it checked, as counts of the files, tests, operations, requirements or requests it read. Refines DEV-021. Ops
TST-012 Every test and every CI job MUST have a bounded timeout (design targets: 1 s per unit test, 2 s per handler test, 10 s per integration or end-to-end case). Requests made by tests MUST carry the production deadline of 2 s and MUST NOT raise it; deadline tests assert an answer by 2 s plus 50 ms. Lint
TST-013 A new or changed test or gate MUST be shown to fail before it is trusted: against the unfixed code for a defect, against at least one planted mutant otherwise. The pull request MUST record the red run. Extends TST-003. Review
TST-014 Test data MUST come from the seeded synthetic fixture generator ([[Test Data Management Test-Data-Management]]): reproducible from a printed seed, Germany-wide in shape, fictional in content, with a manifest naming the licence of every input. Refines TST-009.
TST-015 Contract conformance MUST be tested in both directions: every response a test receives MUST validate against the schema for its status, every request the OpenAPI document declares invalid MUST be refused with 400 invalid-parameter listing the violation, and every documented example request MUST be accepted. Refines TEST-003. Contract
TST-016 Schema-driven property-based fuzzing of a running test instance from the OpenAPI document MUST run on every pull request with a short budget and nightly with a long budget; a 5xx outside the documented causes, a schema violation or an answer later than 2 s plus 50 ms fails. Security
TST-017 Automatic retries of failing tests MUST NOT be configured in any CI job or test runner; an intermittently failing test is handled by quarantine (TST-008). Lint
TST-018 Every quality gate MUST be declared with its stage and its blocking status ([[Test Environments and Quality Gates Test-Environments-and-Quality-Gates]]); a blocking gate MUST NOT be made non-blocking without a recorded decision.
TST-019 Every milestone MUST have a test plan section with scope, requirement IDs, entry, exit, suspension and resumption criteria ([[Test Plan Test-Plan]]), and every release MUST publish a test completion report from the template, referenced by the release acceptance (TEST-011).
TST-020 Every defect fix and every escaped defect MUST get a regression test at the lowest test level able to catch it; the completion report MUST name, for each escaped defect, the level that should have caught it. Review
TST-021 The nightly load run MUST compare p95 and p99 per endpoint class with the budget of NFR-PERF-003 and with the previous run on main; a budget breach fails the run, and a regression above 10 % (design target) inside the budget opens an issue. Load
TST-022 CI ephemeral test environments MUST be created from their committed definition for each run and destroyed on every exit path, including failure and cancellation; no state MAY be shared between runs. Ops

Home

About

Design

Data Model

Technical specification

Testing

Engineering

Technology choice

Reference

Clone this wiki locally