Skip to content

Testing Strategy

MAVERICK edited this page Nov 13, 2026 · 3 revisions

Testing Strategy

The organisational test strategy of the ViaFrei API, in the sense of ISO/IEC/IEEE 29119-3, using ISTQB terminology. It says why we test, what each test level and test type proves, who owns it, when it runs and when it is done. The normative requirements stay on Spec: Test Strategy (prefix TST) and Spec: Conformance and Acceptance (prefix TEST); this page and the pages below elaborate them and never contradict them.

Status: design phase. No implementation exists, so nothing here is a measurement. Every figure is a design target or an assumption.

The document set

ISO/IEC/IEEE 29119-3 names a set of test documents. This wiki maps them as follows.

29119-3 document Page Contents
Organisational test strategy this page goals, principles, levels, types, targets, policies, gates, roles
Test plan (per milestone) [[Test Plan Test-Plan]]
Test design specification, by area [[Contract and Schema Testing Contract-and-Schema-Testing]], [[Performance Test Plan
Test data requirements [[Test Data Management Test-Data-Management]]
Test environment requirements and readiness [[Test Environments and Quality Gates Test-Environments-and-Quality-Gates]]
Test case specification, test completion report [[Test Documentation Templates Test-Documentation-Templates]]
Incident (defect) report [[Defect Management Defect-Management]]
Normative requirements [[Spec: Test Strategy Spec-Test-Strategy]], [[Spec: Conformance and Acceptance
Traceability [[Spec: Traceability Matrix Spec-Traceability-Matrix]]

The older overview Testing and Quality stays as the design summary of the test layers; where it and this set differ, this set applies.

1. Goals and risk-based approach

The test effort is spent where a failure hurts most. The priorities follow the quality tree on Spec: Quality Scenarios and the risk register on Spec: Risks, Assumptions and Constraints.

Goal Risk it reduces Primary evidence
No data reaches a key whose plan or licence forbids it RISK-001, THR-007 licence gate matrix, post-check test, mutation score of the gate
No request runs past 2 s, and no query keeps running after it RISK-002, RISK-003, THR-010, THR-015 cancellation tests, chaos runs, overload scenario
The published contract is exactly what the service does RISK-007, THR-005 two-way contract conformance, breaking-change diff, fuzzing
The API and the MCP server answer the same facts RISK-008 parity suite
No secret or personal data leaks RISK-004, THR-008, THR-009, THR-016 planted-secret log sweep, error-detail tests, scans
Stale data is never shown as current RISK-011 freshness tests, lamp unknown scenario
The pipeline itself cannot report green about nothing all planted faults, fail-closed gates

Risk-based depth. Each component is assigned a risk class from likelihood × impact (scale of the risk register). The class sets the test depth.

Risk class Components (planned) Required depth
High (score ≥ 6 or any security, licence or deadline function) authentication, licence gate, admission and deadline, cursor signing, log redaction all applicable levels; property-based tests; mutation score target; second review (SSC-018)
Medium feature use cases, mapping from the read contract, caching, configuration unit, handler, integration, contract; parity where a counterpart exists
Low service endpoints, documentation rendering handler and contract tests; executed examples

Threats from the Spec: Threat Model are turned into abuse cases on the Security Test Plan.

2. Principles

# Principle What it means in practice Requirement
P1 Contract-first The OpenAPI document is written and reviewed before the handler; tests validate behaviour against the document, never the document against the behaviour. LCY-001, VER-006, TEST-003
P2 Tests derive from requirement IDs Every test names the requirement it verifies; CI reports each ID with its tests. A test without an ID is a smell; an automated MUST row without a passing test fails the release. TEST-001, TEST-002
P3 Deterministic Injected clock, seeded randomness with the seed printed on failure, no network, no shared mutable state, no ordering dependence. TST-004
P4 Never a live data source or production No test reaches a live data source, the production API or any host outside the test environment. CI jobs run with outbound network denied after dependencies are installed. CON-009, TST-010
P5 Synthetic fixtures only Test data comes from the seeded fixture generator: production-shaped, fictional, licence-clean, no personal data, no secret. TST-009, TST-014, SSC-017
P6 Every gate fails closed and prints what it checked A gate that found no files, no tests, no operations or no results fails. Its success line states the counts it read. DEV-021, TST-011
P7 Bounded timeouts, and the 2 s deadline is tested Every test has a timeout; requests in tests carry the production deadline and never a raised one. TST-012, NFR-PERF-001
P8 A test must be shown to fail (mutant proof) A new test or gate is shown red against the unfixed code or a planted mutant before it is trusted; the pull request records the red run. TST-003, TST-006, TST-013
P9 Lowest level that can catch it A behaviour is tested at the cheapest level that can observe it; higher levels test integration, not logic again. TST-020
P10 No retry to green A failing test is a signal; it is fixed or quarantined with an issue, never re-run until it passes. TST-008, TST-017

3. Test levels

flowchart TB
  subgraph Static[Static testing]
    S1[Reviews of requirements, OpenAPI and code]
    S2[Lint, type checks, OpenAPI lint, scope lint, SAST]
  end
  subgraph Dynamic[Dynamic testing, cheapest first]
    U[Component level: unit and property-based tests<br/>domain, gate, limits, parsers]
    H[Component integration: handler tests<br/>HTTP in process with stub stores]
    I[Integration: throwaway PostGIS<br/>read-only role, mapping, cancellation, plans]
    C[Contract: OpenAPI both directions, fuzzing, MCP parity]
    E[System: end to end against the built image]
    N[Non-functional system tests<br/>performance, resilience, security scans]
    A[Acceptance: release evidence and conformance claim]
  end
  Static --> U --> H --> I --> C --> E --> N --> A
Loading

Text description of the diagram: static testing comes first; dynamic testing then proceeds from the component level (unit and property-based tests), through handler tests with stub stores, integration tests against a throwaway PostGIS database, contract tests, end-to-end tests against the built image and non-functional system tests, to acceptance, which collects the release evidence.

Level What it proves Scope Test basis Owner When Exit criterion
Static requirements, contract and code are reviewable and free of known defect patterns specification rows, OpenAPI document, code, configuration declaration Spec pages, style guide (LCY), DEV rules author and reviewer every pull request zero lint errors; review approved; OpenAPI lint clean
Component (unit) each rule is right in isolation, including boundaries domain, licence gate logic, limiters, parsers, renderers requirement rows with method Unit author every push all pass; coverage and mutation targets on changed code
Component integration (handler) the HTTP layer maps inputs, errors and headers correctly one handler in process with fixture-backed stub stores rows with method Handler author every push every operation × documented status exercised
Integration the service and its storage agree storage adapter, read-only role, cancellation, query plans, access store rows with method Mapping, Load (plan tests), Security (role) author every push every read-contract view prepared and mapped; a write through the read role refused
Contract the document and the behaviour cannot drift; the API and MCP agree the running service against the OpenAPI document; parity bundle OpenAPI document, rows with method Contract and Parity maintainers every push; long runs nightly zero conformance failures; 100 % parity
System (end to end) the built image works as shipped the image started from its release configuration against a fixture database user journeys on [[Spec: Developer Experience Spec-Developer-Experience]] maintainers pull request smoke; full set on main
Non-functional system budgets, resilience and security hold under load and faults the image on the test stack NFR, RATE, QAS rows maintainers nightly, weekly, release all budgets met; no request past 2 s
Acceptance the release conforms to its specification revision the confirmed release's digest on staging TEST-010, TEST-011 owner each release conformance claim recorded; completion report published

4. Test types

Tooling is named by category. Examples per candidate stack are given where they help; the final choice follows ADR-0001 and is recorded on the stack's architecture page.

4.1 Unit and property-based tests

Aspect Content
Proves each domain rule, positive and negative; invariants over generated inputs (kilometre arithmetic, area bounds, price ordering, cursor round trip, Berlin rendering across daylight-saving changes, the gate over plan × family × acceptance)
Scope domain, application and parser code; no I/O
Tooling test runner with table-driven cases and a property-based library with shrinking. Go: the standard test runner, rapid or gopter. Rust: cargo test or nextest, proptest. TypeScript: Vitest or Jest, fast-check
Owner the author of the change
When every push and pull request
Exit all pass; TST-001 coverage and TST-002 mutation targets on changed code; date boundaries of TST-005 covered

4.2 Component and handler tests with stub stores

Aspect Content
Proves status codes, problem types, headers (security, caching, rate limit, deprecation), 304, fields, lang, the envelope shape
Scope one operation in process; storage and access store replaced by fixture-backed stubs behind the ports
Tooling the framework's in-process HTTP test client; snapshot (golden) files for bodies. Go: httptest. Rust: tower's service test utilities. TypeScript: the framework's inject function or supertest
Owner author
When every push
Exit every operation × every status it can return (TEST-010); every response validated against the document (TEST-003)

4.3 Integration tests against a throwaway PostGIS with a read-only role

Aspect Content
Proves every statement prepares against the read contract; rows round-trip into domain values; code maps are total; the read-only role cannot write or read beyond the contract; storage-side cancellation leaves the backend idle; no sequential scan of time-series data
Scope storage adapter and access-store adapter against a database built from the read-contract stub and seeded by the [[fixture generator
Tooling throwaway database containers (Testcontainers for Go, Rust or Node, or a CI service container), plan assertions over EXPLAIN output
Owner author; maintainers own the harness
When every push
Exit every view and code covered; the planted forbidden row trips the post-check (SEC-032); role write refused (NFR-SEC-004)

4.4 Contract and schema conformance, both directions

Detailed on Contract and Schema Testing.

Aspect Content
Proves responses match the response schema for their status (outbound); requests the document forbids are refused, requests it allows are accepted (inbound)
Scope every operation, status, header and example
Tooling an OpenAPI 3.1 validator in the test harness (Go: kin-openapi. Rust: a JSON Schema 2020-12 validator. TypeScript: Ajv with an OpenAPI wrapper), a negative-request generator, Spectral for the document
Owner maintainers
When every push
Exit zero conformance failures; coverage table of operation × status complete for the milestone

4.5 Property-based API fuzzing from the OpenAPI document

Aspect Content
Proves no generated request yields a 5xx, a schema violation or an answer after 2 s plus 50 ms; parsers survive arbitrary input
Scope the running CI-started instance (schema-driven) and each parser (native fuzz targets)
Tooling Schemathesis against the instance; native fuzzers per stack (Go fuzzing, cargo-fuzz, fast-check or Jazzer.js)
Owner maintainers
When short run on every pull request, long run nightly
Exit no crash, no hang, no schema violation; findings become unit tests

4.6 Breaking-change detection via OpenAPI diff

Aspect Content
Proves no removed operation, member or enum value and no tightened parameter within a major (VER-001, VER-002)
Scope the generated document against main and against the last released document
Tooling an OpenAPI diff tool such as oasdiff
Owner maintainers
When every pull request; release
Exit no breaking change unless the change introduces a new major path

4.7 MCP parity tests

Aspect Content
Proves the same question gives the same facts from the API and from the ViaFrei MCP server (CON-007, FR-GEN-002)
Scope every operation with x-mcp-counterpart (TEST-009)
Tooling the parity harness: a versioned bundle of seed rows, questions and expected facts; a fact projection; a comparator (see [[Contract and Schema Testing
Owner maintainers
When every push; bundle refreshed by the version sync
Exit 100 % fact equality (QAS-FUNC-01)

4.8 End-to-end tests against the built image

Aspect Content
Proves the image as shipped boots, reads its configuration, refuses invalid configuration, serves the journeys of a client (first call, paging, conditional request, refusal, rate limit) and shuts down cleanly
Scope the image, a fixture database, generated test keys
Tooling a container runtime and a scripted HTTP client; the contract validator stays on
Owner maintainers
When smoke on every pull request; full set on main and on each release tag before it publishes
Exit all journeys pass; no secret in the image (scan); /healthz and /readyz per FR-SVC-004

4.9 Security tests

Detailed on Security Test Plan, with the mapping to the OWASP API Security Top 10 (2023) and to every THR row.

Aspect Content
Proves the authentication and authorisation matrix holds; no key or secret appears in any log sink; abuse cases from the threat model are refused; no known vulnerable dependency, image layer or code pattern ships
Scope the service, the image, the dependencies, the repository
Tooling handler and integration tests for the matrices; SAST, dependency and container scans and DAST as listed in issue #12
Owner maintainers; security-relevant changes need a second reviewer (SSC-018)
When every pull request; DAST full scan weekly
Exit matrices complete; no high or critical finding open beyond SSC-015 deadlines

4.10 Licence and attribution tests

Aspect Content
Proves every data response carries attribution and freshness for every contributing family (LIC-001, ENV-003), even when empty; notes are verbatim (LIC-003); fields cannot remove them (LIC-004); a family is served only to plans whose use class its terms permit (SEC-030 to SEC-033)
Scope the licence gate matrix: every key scope × plan × use class × source family × terms acceptance, on the fictional families of the fixture set, which cover every licence class
Tooling generated table-driven tests; a property-based test over the full matrix; a contract rule that meta.attribution is non-empty on every data response
Owner maintainers; owner approves changes to the matrix
When every push
Exit zero forbidden items in the full matrix (QAS-SEC-01); attribution on 100 % of data responses (QAS-FUNC-02)

4.11 Configuration tests

Aspect Content
Proves every key has a default and bounds, is validated at boot, appears in the generated reference, and changes behaviour when set (CFG-001 to CFG-007, NFR-MAINT-002)
Scope the configuration declaration and every consumer of a key
Tooling a declaration-driven test that sets each key to a non-default value and asserts the observable change; boot-refusal tests for unknown keys, out-of-bound values and cross-field rules
Owner author of the key
When every push
Exit every declared key covered (CFG-006); the reference regenerated without diff

4.12 Performance tests

Detailed on Performance Test Plan, which refines the Benchmark Plan.

Type Question Load shape (design)
Benchmark which stack, and does each query meet its budget three steps from light to well above early load (Benchmark Plan)
Load budgets per endpoint class at design load 50 requests per second sustained (A-CAP-1)
Stress where is the knee, and does throughput stay flat above it ramp beyond design load
Spike does a sudden burst shed cleanly 200 per second for 10 s, then four times design load briefly
Soak does anything leak over time design load for 1 hour

Latency budgets per endpoint class are the design targets of NFR-PERF-003 (p50, p95, p99 per class, 2 s hard deadline for all). They are asserted on fixtures, never measured on production in a test.

4.13 Resilience and chaos tests

Aspect Content
Proves slow database, pool exhaustion, deadline cancellation, load shedding and degraded partial responses behave as NFR-RES and QAS-REL specify
Scenarios storage slowed; storage killed; pool exhausted; access store down; instance killed mid-request; clock skew; one part of a composed answer timing out
Tooling a fault-injecting proxy between service and database (for example Toxiproxy), container control, a sleeping stub query
Owner maintainers
When nightly on the test stack (TST-007)
Exit no request past 2 s plus 50 ms; backend idle within the cancel grace plus 100 ms; every scenario's documented response observed

4.14 Observability tests

Aspect Content
Proves the access log line carries the fields of NFR-OBS-003; RED metrics exist per operation and tier (NFR-OBS-001); no personal data and no secret reaches any sink (NFR-OBS-004, NFR-PRIV-002); metrics are not on the public listener (NFR-OBS-002)
Tooling a captured log sink and a metrics scrape in handler and end-to-end tests; a planted-secret sweep over every sink
When every push; the sweep also runs in the end-to-end suite
Exit every declared field present; zero planted secrets or full addresses found

4.15 Documentation tests

Aspect Content
Proves every example in the OpenAPI document and the guide executes against the fixture instance and validates; every problem type URI resolves (ERR-002, FR-SVC-006); internal links resolve
Tooling an example runner driven by the document; a link checker
When every pull request that touches documentation; every release
Exit all examples execute; zero broken links

4.16 Accessibility of the documentation

Aspect Content
Proves WCAG 2.2 level AA for the portal (NFR-A11Y-001); a text description for every diagram (NFR-A11Y-002)
Tooling automated checks (axe-core, pa11y) plus a manual keyboard and screen-reader pass per minor release
When docs build; manual pass per minor
Exit no critical or serious automated finding; the manual pass recorded

4.17 Internationalisation tests

Aspect Content
Proves every human-readable text exists in de and en (NFR-I18N-001); lang overrides Accept-Language (PROT-025); problem title is localised while type and codes are not (ERR-011); numbers in JSON are locale-independent (NFR-I18N-002); condition notes stay verbatim and untranslated (LIC-003); instants are UTC with Europe/Berlin companions (ADR-0004)
Tooling catalogue parity check (every key in every language), handler tests per language, property tests over Berlin rendering
When every push
Exit catalogue parity complete; no locale-dependent number

4.18 Mutation testing

Aspect Content
Proves the tests would notice a wrong answer, not merely execute the line
Scope domain, access and admission code
Tooling Go: Gremlins or go-mutesting. Rust: cargo-mutants. TypeScript: StrykerJS
When changed code on every pull request; full run per minor release
Exit TST-002 targets

5. Coverage and mutation targets

All are design targets set before any code exists.

Measure Target Source
Statement coverage, every package of every layer 100 % TST-001 (owner rule, 2026-11-02)
Branch coverage, licence gate and admission ≥ 90 % design target of this page
Mutation score, changed domain, access and admission code per pull request ≥ 80 % TST-002
Mutation score, overall per minor release ≥ 70 % TST-002
Operation × documented status with a validated response 100 % TEST-003, TEST-010
Operations with a counterpart that have a parity case 100 % TEST-009
Configuration keys with a behaviour test 100 % CFG-006
Automated MUST rows in scope with a passing test 100 % TEST-002

Coverage is measured over three merged profiles: the unit and handler run, the integration run against a throwaway PostGIS container (which reaches the repositories and the generated queries) and a boot smoke that builds every command with coverage instrumentation and runs it once (which reaches main). Generated code may be left out only through the repository's named exclusion file, test/coverage-exclusions.txt, one reason per entry; a stale or reasonless entry fails the gate. A quarantined test does not count (TST-008). Implemented in 0.2.0 as the CI job Coverage gate (100 % per package), whose merged profile is also the one SonarCloud reads.

6. Flaky-test policy

  • A test that fails and then passes without a change is flaky. It is a defect, labelled as such on Defect Management.
  • It is quarantined within one day: moved to a quarantine list that is visible in every run, with an issue linked from the list (TST-008). A quarantine entry without an issue fails the build.
  • It is fixed or deleted within one week. A deleted test whose requirement then has no test turns the requirement report red.
  • No retry to green: CI runners are configured without automatic test retries; a retry flag in the test configuration fails a lint (TST-017).
  • The flake rate is reported per week (section 10).

7. Test data management

Summary; the full rules are on Test Data Management.

  • One synthetic fixture generator, seeded and reproducible, produces production-shaped but fictional data in the shapes of the Data Model, Germany-wide: all 16 Länder, motorways, time partitions, every source family and licence class (issue #13).
  • Profiles tiny, ci and benchmark; a manifest with seed, counts, edge cases and checksum.
  • No copied real data, no personal data, no secret. Any sample data that is not our own carries its licence in the manifest.
  • Keys in tests are generated per run and use the vf_test_ prefix.

8. Environments

Summary; details on Test Environments and Quality Gates.

Environment Used for Data
Local development, all functional levels generator tiny and ci profiles in a throwaway database
CI ephemeral every gate up to end to end; created and destroyed per run generator ci profile
Test stack performance, resilience, DAST; isolated from production generator benchmark profile
Staging release acceptance on the published digest (REL pipeline) production-shaped fixture set

The production API is never a test target. Load, chaos, fuzzing and DAST runners refuse any target not on the test-environment allow-list. Production is observed only through its own telemetry and SLOs (Spec: Operations), which are operations, not tests.

9. Quality gates per stage

flowchart LR
  PC[Pre-commit<br/>format, lint, fast unit] --> PR[Pull request<br/>all functional levels,<br/>contract, scans, smoke]
  PR -->|required checks green<br/>and review| M[Main<br/>full end to end,<br/>requirement report]
  M --> NI[Nightly<br/>long fuzz, load, soak,<br/>chaos, deep schema runs]
  NI --> WK[Weekly<br/>full DAST, scorecard,<br/>scheduled dependency scans]
  WK --> RL[Release<br/>acceptance on staging,<br/>completion report]
  RL --> CL[Conformance claim]
Loading

Text description of the diagram: a change passes the pre-commit checks, then the pull-request gate, which needs every required check and a review; on main the full end-to-end set and the requirement report run; nightly and weekly runs add the long and expensive tests; a release passes acceptance on staging and publishes its completion report and conformance claim.

Check Pre-commit Pull request Main Nightly Weekly Release
Format, lint, type checks runs blocks blocks blocks
Scope lint and configuration lint blocks blocks blocks
Unit and property-based fast subset blocks blocks blocks
Handler and integration blocks blocks blocks
Coverage blocks reports blocks
Mutation, changed code blocks full run blocks
Contract both directions, breaking-change diff blocks blocks blocks
MCP parity blocks blocks blocks
Fuzzing short run blocks long run, opens issue evidence
End to end smoke blocks full blocks full blocks
SAST, dependency, secret, container scans secret scan blocks blocks scheduled scans blocks
DAST baseline blocks full scan, opens issue evidence
Load against budgets short run reports opens issue on breach blocks
Soak, stress, spike, chaos opens issue on breach blocks
Docs examples, links, accessibility blocks when docs change blocks
Requirement report reports blocks for in-scope rows blocks

A nightly or weekly failure does not block a merge that already happened; it opens or updates an issue with priority by Defect Management, and the next release is blocked until it is resolved. Each gate is declared with its stage and its blocking status, and making a blocking gate non-blocking needs a recorded decision (TST-018).

10. Defect management

Summary; details on Defect Management.

  • Severity S1 to S4 describes impact; priority P1 to P3 describes order of work, mapped to the priority: P1 to priority: P3 labels; defects carry type: bug.
  • Every defect fix needs a regression test that fails without the fix (TST-003), at the lowest level able to catch it (TST-020).
  • An open S1 or S2 defect blocks a release.

11. Metrics and reporting

All values below are targets, not results. Results appear in the per-release test completion report.

Metric Definition Target
Pass rate on main passed ÷ executed tests in required checks on main 100 %
Flake rate tests that failed and passed on the same commit ÷ tests executed, per week below 0.5 %
Quarantine age days a test spends in quarantine at most 7
Mutation score killed ÷ (generated − equivalent) section 5
Latency against budget p95 and p99 per endpoint class on the test stack ÷ budget at most 1.0
Requirement coverage automated MUST rows in scope with a passing test 100 %
Escaped defects defects found after a release, per release, with the level that should have caught them trending to zero; each one analysed
Defect removal efficiency defects found before release ÷ all defects of that release at least 90 %
CI time to a green pull request with a warm cache under 15 minutes (REL-014)
DORA: deployment frequency, lead time, change failure rate, time to restore from the deployment record the targets on [[Spec: CI/CD and Release

12. Roles

Role Responsibilities in testing
Owner approves this strategy, the licence matrix and its test matrix, the release conformance claim; decides on blocking-gate changes
Maintainers own the harnesses, the gates, the fixture generator and the non-functional suites; review tests at the right level; triage defects and flaky tests; second review of security-relevant changes
Author (human or AI agent) writes the tests with the change, tags them with requirement IDs, shows each new test failing first, keeps the change within the gates
AI agents may write, run and review tests like any author; work under the same gates and branch protection; never hold production credentials; an agent's review does not replace the required human code-owner approval
Reviewer checks that tests sit at the right level, cite the right IDs and would fail without the change (DEV-011)

Related

Test Plan · Spec: Test Strategy · Spec: Conformance and Acceptance · Spec: Non-Functional Requirements · Spec: Threat Model · Spec: Risks, Assumptions and Constraints · Benchmark Plan · Development Process

Home

About

Design

Data Model

Technical specification

Testing

Engineering

Technology choice

Reference

Clone this wiki locally