Skip to content

Test Plan

MAVERICK edited this page Nov 13, 2026 · 4 revisions

Test Plan

The test plan of the ViaFrei API, one section per milestone of the Roadmap, in the structure of an ISO/IEC/IEEE 29119-3 test plan. It applies the Testing Strategy; the requirements it cites are on the Spec pages and in the Spec: Traceability Matrix.

Status: design phase. Every threshold is a design target. No milestone has started its test execution.

Common to every milestone

Test items and approach

Each milestone tests the increment it delivers plus a regression run of everything delivered before. The test levels and types are those of the Testing Strategy section 3 and 4; which of them a milestone adds is listed in its section.

General entry criteria

  • The previous milestone met its exit criteria, or the owner recorded an exception.
  • The milestone's issues meet the Definition of Ready (DEV-030), and each names the requirement IDs it covers.
  • The in-scope requirement list for the milestone is committed, so the requirement report (TEST-002) knows what must be green.
  • The environments the section needs are available (Test Environments and Quality Gates).

General exit criteria

  • Every blocking gate of the stages the milestone uses is green on main.
  • The requirement report shows every automated MUST row in scope with a passing test.
  • No open defect of severity S1 or S2 (Defect Management); S3 defects are listed in the completion report.
  • No test in quarantine older than seven days.
  • The milestone's test completion report is written from the template and linked from the milestone's release.

General suspension and resumption criteria

Suspend when Resume when
A blocking gate fails on main for a reason other than the change under test (broken harness, broken environment) the harness or environment is fixed and the gate is shown green on an unchanged commit
A test environment cannot be shown isolated from production (allow-list missing or unreadable) isolation is restored and shown by the target guard refusing a non-allowed host
The fixture set is found to contain copied real data, personal data or a secret the data is removed, the generator fixed, the history rewritten if a secret is involved, and the coverage check green
The flake rate exceeds its target for two consecutive weeks the flaky tests are quarantined with issues and the rate is back within target
An S1 defect is open the defect is fixed with a regression test

Test deliverables (every milestone)

Test cases in the repository tagged with requirement IDs; CI run records; the requirement report; the test completion report; for milestones with non-functional tests, the dated result files.

Issues for the test infrastructure

Issue Milestone Delivers
#13 0.1.0 synthetic fixture generator
#14 0.2.0 requirement-ID report and draft completion report
#15 0.3.0 two-way contract-conformance harness, released-document diff
#16 0.3.0 parser fuzz targets, corpus, nightly budget
#17 0.4.0 mutation-testing job
#18 0.6.0 MCP parity harness
#19 0.7.0 soak, stress, spike and fault-injection runs
#20 First public release documentation accessibility and link checks

Already covered by existing issues: the benchmark harness (#2), the CI pipeline with planted-fault proof and the OpenAPI diff against main (#5), SAST, DAST, dependency and container scans and the per-pull-request schema-driven fuzzing job (#12), configuration tests (#11), authentication and licence-gate tests (#7), the overload acceptance (#8), latency budgets and plan tests (#9), executed documentation examples (#4).

0.1.0 Stack decision

Item Content
Scope the benchmark harness and its run over the three candidates (Go, Rust, TypeScript); the fixture generator it needs; the decision record ADR-0001
Features to test the eight checks of the [[Benchmark Plan
Approach the harness, not the candidates, is the test item first: it is verified by planting faults into a candidate (a cancellation that only abandons the query, a pool that admits more than its permits) and showing the corresponding check fail. Then the candidates are run under equal limits with an open-model load generator in its own container, warm-up excluded, percentiles from full histograms
Entry criteria fixture generator benchmark profile available (#13); thresholds of the Benchmark Plan frozen before the first run
Exit criteria every check run for every candidate, results in a dated result file; each check shown able to fail by a planted fault; the decision rule applied and ADR-0001 proposed with the results; the run reproducible by one command
Deliverables the harness, result files, the planted-fault record, ADR-0001 draft, the completion report
Environment the test stack only: a throwaway PostGIS with the benchmark profile, candidate containers at equal limits, separate load-generator container
Risks a check that cannot fail (mitigated by planted faults); noisy neighbours skewing latency (mitigated by repeated runs and reporting variance); thresholds adjusted after seeing results (mitigated by freezing them first)
Suspension and resumption suspend if two runs of the same candidate differ in p95 by more than 20 % (design) on the same host; resume after the noise source is found or the run is moved
Issues #2, #13

0.2.0 Skeleton and pipeline

Item Content
Scope service skeleton, configuration layer, service endpoints, image, CI with every static and security gate, the requirement report
Features to test FR-SVC-003, FR-SVC-004, FR-SVC-005; CFG-001 to CFG-007; NFR-SEC-004 (read-only role); NFR-PORT-001, NFR-PORT-002; SSC-001, SSC-005, SSC-013, SSC-014, SSC-017, SSC-019; DEV-020, DEV-021; TST-006, TST-010, TST-011, TST-017; TEST-001, TEST-002
Approach unit and handler tests for the service endpoints; integration test of the read-only role (a write refused); configuration tests per key; end-to-end smoke of the image; the pipeline itself tested by planted faults: a lint error, a failing test, a breaking OpenAPI change, a leaked test key in a log, a planted secret, a vulnerable dependency, an injection pattern
Entry criteria ADR-0001 accepted
Exit criteria every planted fault turns the pull request red (TST-006); the image boots, answers /healthz and holds no secret; the requirement report runs and fails closed; outbound network denied in CI test jobs
Deliverables CI configuration, planted-fault record, requirement report, completion report
Environment local, CI ephemeral
Risks a gate that reports green over no input (mitigated by TST-011 and DEV-021); tool noise leading to disabled checks (mitigated by TST-018)
Suspension and resumption suspend feature work if a required check cannot be made blocking; resume when branch protection requires it
Issues #3, #5, #11, #12, #14

0.3.0 Contract and version sync

Item Content
Scope the read-contract file and generator, version sync, /v1/meta versions; the two-way contract harness; parser fuzzing
Features to test VER-002, VER-005, VER-006, VER-009; FR-SVC-003; TEST-003, TEST-004, TEST-005, TEST-006; TST-015, TST-016; NFR-SEC-002; ENV-001 to ENV-005; ERR-001, ERR-005
Approach contract tests in both directions on every handler test; the breaking-change diff against main and against the last released document; a sync run against a fixture platform version; readiness refused on an incompatible contract major; native fuzz targets for every parser that exists by then
Entry criteria 0.2.0 exit met; a fixture platform version available
Exit criteria zero conformance failures; each planted contract fault red; a sync bumps version, changelog and tag together; fuzz targets running with a committed corpus
Deliverables contract coverage table, fuzz report, completion report
Environment local, CI ephemeral
Risks validation in tests drifting from validation in the service (mitigated by the negative direction); generated documents differing by formatting only (mitigated by canonical serialisation)
Suspension and resumption suspend the release if the document cannot be regenerated without diff; resume when generation is deterministic
Issues #6, #15, #16

0.4.0 Keys and licence gate

Item Content
Scope access store, key lifecycle, scopes, the licence gate with post-check, audit log, log redaction
Features to test SEC-001 to SEC-052; LIC-001 to LIC-012 where they concern the gate; NFR-OBS-004; QAS-SEC-01, QAS-SEC-03, QAS-SEC-04, QAS-SEC-05; ERR-006; TST-002, TST-013
Approach the authentication matrix and the full licence gate matrix as generated table-driven and property-based tests over the fictional families of the fixture set; a planted forbidden row in storage that the post-check must catch; a planted-secret sweep over every log sink; a timing test of the key check; mutation testing switched on for access code; second review for every change (SSC-018)
Entry criteria 0.3.0 exit met; the fixture set covers every licence class
Exit criteria zero forbidden items in the full matrix; every refusal path has a test; zero planted secrets found in any sink; mutation targets met on access code
Deliverables gate matrix report, log sweep record, mutation report, completion report
Environment local, CI ephemeral
Risks a matrix generated from the same table the code reads, so both are wrong together (mitigated by an independent expected-value table reviewed by the owner); timing tests flaky on shared runners (mitigated by statistical test with many samples)
Suspension and resumption suspend the release on any forbidden item served; resume when the defect is fixed and the full matrix is green
Issues #7, #17

0.5.0 Limits, tiers and deadlines

Item Content
Scope per-IP and per-key limits, quotas, priority admission, the 2 s deadline with storage-side cancellation, metering
Features to test RATE-001 to RATE-017; NFR-PERF-001, NFR-PERF-002; NFR-CAP-002, NFR-CAP-003; NFR-RES-001, NFR-RES-007; ERR-007, ERR-012; QAS-PERF-04, QAS-REL-01; TST-012
Approach unit and property-based tests of buckets and admission with an injected clock; quota reset across both daylight-saving changes; integration tests of cancellation with a sleeping stub query on the timeout and client-disconnect paths; the overload scenario on the test stack
Entry criteria 0.4.0 exit met; test stack available
Exit criteria no request past 2 s plus 50 ms in any test; backend idle within the cancel grace plus 100 ms; 1 000 cancellation iterations without a poisoned connection; the higher tier keeps its target under overload while the lower tier gets 429 with Retry-After
Deliverables overload report, cancellation report, completion report
Environment CI ephemeral for functional tests; test stack for overload
Risks timing-dependent tests flaky (mitigated by injected clocks and tolerance bands); overload results depending on runner size (mitigated by fixed container limits)
Suspension and resumption suspend if a cancellation leak is observed; resume after the fix and 1 000 clean iterations
Issues #8

0.6.0 Feature endpoints

Item Content
Scope the feature endpoints in the planned order, composed features, condition history
Features to test FR-PLACE, FR-ROAD, FR-ENERGY, FR-PARK, FR-TRANSIT, FR-ACCESS, FR-WEATHER, FR-RULES, FR-BRIEF, FR-LOC, FR-GEN; ENV-020 to ENV-033; NFR-FRESH-001 to NFR-FRESH-003; NFR-RES-003; QAS-FUNC-01, QAS-FUNC-02, QAS-REL-03, QAS-SAFE-01; TEST-009, TEST-010; NFR-I18N-001, NFR-I18N-002
Approach the endpoint Definition of Done (TEST-010) per operation; mapping tests over the read-contract fixtures; parity on every operation with a counterpart; composed features tested with one part failing; i18n catalogue parity
Entry criteria 0.5.0 exit met; the endpoint list reviewed by the owner; parity bundle for the pinned platform version available
Exit criteria every operation meets TEST-010; 100 % fact equality in the parity suite; attribution on 100 % of data responses; a stale family produces lamp unknown, never green
Deliverables parity report, contract coverage table, completion report
Environment local, CI ephemeral
Risks parity drift when the platform changes (mitigated by the bundle pinned per platform version); storage names leaking into payloads (mitigated by the scope lint)
Suspension and resumption suspend an endpoint's release if its parity fails; resume when the cause is found on either side and recorded
Issues #10, #18

0.7.0 Performance

Item Content
Scope HTTP caching, result cache with single-flight, compression, query plans, latency budgets in the pipeline; soak, stress, spike and chaos
Features to test NFR-PERF-003 to NFR-PERF-006; NFR-RES-002, NFR-RES-004 to NFR-RES-006; PROT-040 to PROT-044; QAS-PERF-01 to QAS-PERF-03, QAS-REL-02; TST-007, TST-021
Approach per the [[Performance Test Plan
Entry criteria 0.6.0 exit met; test stack with the benchmark profile
Exit criteria every endpoint class within its budget; 304 or cache hit on at least 95 % of repeated polls; exactly one storage read under 50 identical concurrent requests; no sequential scan on time-series data; soak without memory growth beyond 10 % after warm-up
Deliverables dated performance result files, chaos report, completion report
Environment test stack only
Risks budgets met on fixtures but not on production-shaped skew (mitigated by skewed distributions in the generator); result noise (mitigated by repeated runs)
Suspension and resumption suspend if the test stack cannot be isolated from other load; resume when isolation is shown
Issues #9, #19

First public release

Item Content
Scope documentation, the published image, the public API host; full regression; release acceptance and the first conformance claim
Features to test TEST-008, TEST-010, TEST-011, TEST-012; QAS-001; REL-001 to REL-009; SSC-007 to SSC-009; NFR-A11Y-001, NFR-A11Y-002; DX rows; QAS-INT-01, QAS-INT-02
Approach full regression of all levels; security acceptance (OWASP API Top 10 cases per operation, full DAST of staging, header and TLS checks); executed documentation examples; accessibility checks plus a manual pass; a usability check of the getting-started guide; image signature and provenance verified at deploy
Entry criteria 0.7.0 exit met; licence matrix signed off (LIC-009); staging deployed from the confirmed release's digest (a release candidate publishes no image)
Exit criteria every QAS scenario has an automated test or operational measurement; all release gates green on staging; no open S1 or S2; the conformance claim names the specification revision
Deliverables conformance claim, requirement report, security and performance results, completion report, release notes
Environment CI ephemeral, test stack, staging; production is not tested, only observed after deployment
Risks documentation examples drifting (mitigated by execution in CI); unresolved SHOULD deviations (recorded in the release notes)
Suspension and resumption suspend the release on any failed staging gate; resume after a fix forward and a fresh staging run. The staging image is already a published release, so a failed acceptance is answered by yanking it and cutting a new patch release
Issues #4, #20

Home

About

Design

Data Model

Technical specification

Testing

Engineering

Technology choice

Reference

Clone this wiki locally