Repository navigation
Test Plan
The test plan of the ViaFrei API, one section per milestone of the Roadmap, in the structure of an ISO/IEC/IEEE 29119-3 test plan. It applies the Testing Strategy; the requirements it cites are on the Spec pages and in the Spec: Traceability Matrix.
Status: design phase. Every threshold is a design target. No milestone has started its test execution.
Each milestone tests the increment it delivers plus a regression run of everything delivered before. The test levels and types are those of the Testing Strategy section 3 and 4; which of them a milestone adds is listed in its section.
- The previous milestone met its exit criteria, or the owner recorded an exception.
- The milestone's issues meet the Definition of Ready (DEV-030), and each names the requirement IDs it covers.
- The in-scope requirement list for the milestone is committed, so the requirement report (TEST-002) knows what must be green.
- The environments the section needs are available (Test Environments and Quality Gates).
- Every blocking gate of the stages the milestone uses is green on main.
- The requirement report shows every automated MUST row in scope with a passing test.
- No open defect of severity S1 or S2 (Defect Management); S3 defects are listed in the completion report.
- No test in quarantine older than seven days.
- The milestone's test completion report is written from the template and linked from the milestone's release.
| Suspend when | Resume when |
|---|---|
| A blocking gate fails on main for a reason other than the change under test (broken harness, broken environment) | the harness or environment is fixed and the gate is shown green on an unchanged commit |
| A test environment cannot be shown isolated from production (allow-list missing or unreadable) | isolation is restored and shown by the target guard refusing a non-allowed host |
| The fixture set is found to contain copied real data, personal data or a secret | the data is removed, the generator fixed, the history rewritten if a secret is involved, and the coverage check green |
| The flake rate exceeds its target for two consecutive weeks | the flaky tests are quarantined with issues and the rate is back within target |
| An S1 defect is open | the defect is fixed with a regression test |
Test cases in the repository tagged with requirement IDs; CI run records; the requirement report; the test completion report; for milestones with non-functional tests, the dated result files.
| Issue | Milestone | Delivers |
|---|---|---|
| #13 | 0.1.0 | synthetic fixture generator |
| #14 | 0.2.0 | requirement-ID report and draft completion report |
| #15 | 0.3.0 | two-way contract-conformance harness, released-document diff |
| #16 | 0.3.0 | parser fuzz targets, corpus, nightly budget |
| #17 | 0.4.0 | mutation-testing job |
| #18 | 0.6.0 | MCP parity harness |
| #19 | 0.7.0 | soak, stress, spike and fault-injection runs |
| #20 | First public release | documentation accessibility and link checks |
Already covered by existing issues: the benchmark harness (#2), the CI pipeline with planted-fault proof and the OpenAPI diff against main (#5), SAST, DAST, dependency and container scans and the per-pull-request schema-driven fuzzing job (#12), configuration tests (#11), authentication and licence-gate tests (#7), the overload acceptance (#8), latency budgets and plan tests (#9), executed documentation examples (#4).
| Item | Content |
|---|---|
| Scope | the benchmark harness and its run over the three candidates (Go, Rust, TypeScript); the fixture generator it needs; the decision record ADR-0001 |
| Features to test | the eight checks of the [[Benchmark Plan |
| Approach | the harness, not the candidates, is the test item first: it is verified by planting faults into a candidate (a cancellation that only abandons the query, a pool that admits more than its permits) and showing the corresponding check fail. Then the candidates are run under equal limits with an open-model load generator in its own container, warm-up excluded, percentiles from full histograms |
| Entry criteria | fixture generator benchmark profile available (#13); thresholds of the Benchmark Plan frozen before the first run |
| Exit criteria | every check run for every candidate, results in a dated result file; each check shown able to fail by a planted fault; the decision rule applied and ADR-0001 proposed with the results; the run reproducible by one command |
| Deliverables | the harness, result files, the planted-fault record, ADR-0001 draft, the completion report |
| Environment | the test stack only: a throwaway PostGIS with the benchmark profile, candidate containers at equal limits, separate load-generator container |
| Risks | a check that cannot fail (mitigated by planted faults); noisy neighbours skewing latency (mitigated by repeated runs and reporting variance); thresholds adjusted after seeing results (mitigated by freezing them first) |
| Suspension and resumption | suspend if two runs of the same candidate differ in p95 by more than 20 % (design) on the same host; resume after the noise source is found or the run is moved |
| Issues | #2, #13 |
| Item | Content |
|---|---|
| Scope | service skeleton, configuration layer, service endpoints, image, CI with every static and security gate, the requirement report |
| Features to test | FR-SVC-003, FR-SVC-004, FR-SVC-005; CFG-001 to CFG-007; NFR-SEC-004 (read-only role); NFR-PORT-001, NFR-PORT-002; SSC-001, SSC-005, SSC-013, SSC-014, SSC-017, SSC-019; DEV-020, DEV-021; TST-006, TST-010, TST-011, TST-017; TEST-001, TEST-002 |
| Approach | unit and handler tests for the service endpoints; integration test of the read-only role (a write refused); configuration tests per key; end-to-end smoke of the image; the pipeline itself tested by planted faults: a lint error, a failing test, a breaking OpenAPI change, a leaked test key in a log, a planted secret, a vulnerable dependency, an injection pattern |
| Entry criteria | ADR-0001 accepted |
| Exit criteria | every planted fault turns the pull request red (TST-006); the image boots, answers /healthz and holds no secret; the requirement report runs and fails closed; outbound network denied in CI test jobs |
| Deliverables | CI configuration, planted-fault record, requirement report, completion report |
| Environment | local, CI ephemeral |
| Risks | a gate that reports green over no input (mitigated by TST-011 and DEV-021); tool noise leading to disabled checks (mitigated by TST-018) |
| Suspension and resumption | suspend feature work if a required check cannot be made blocking; resume when branch protection requires it |
| Issues | #3, #5, #11, #12, #14 |
| Item | Content |
|---|---|
| Scope | the read-contract file and generator, version sync, /v1/meta versions; the two-way contract harness; parser fuzzing |
| Features to test | VER-002, VER-005, VER-006, VER-009; FR-SVC-003; TEST-003, TEST-004, TEST-005, TEST-006; TST-015, TST-016; NFR-SEC-002; ENV-001 to ENV-005; ERR-001, ERR-005 |
| Approach | contract tests in both directions on every handler test; the breaking-change diff against main and against the last released document; a sync run against a fixture platform version; readiness refused on an incompatible contract major; native fuzz targets for every parser that exists by then |
| Entry criteria | 0.2.0 exit met; a fixture platform version available |
| Exit criteria | zero conformance failures; each planted contract fault red; a sync bumps version, changelog and tag together; fuzz targets running with a committed corpus |
| Deliverables | contract coverage table, fuzz report, completion report |
| Environment | local, CI ephemeral |
| Risks | validation in tests drifting from validation in the service (mitigated by the negative direction); generated documents differing by formatting only (mitigated by canonical serialisation) |
| Suspension and resumption | suspend the release if the document cannot be regenerated without diff; resume when generation is deterministic |
| Issues | #6, #15, #16 |
| Item | Content |
|---|---|
| Scope | access store, key lifecycle, scopes, the licence gate with post-check, audit log, log redaction |
| Features to test | SEC-001 to SEC-052; LIC-001 to LIC-012 where they concern the gate; NFR-OBS-004; QAS-SEC-01, QAS-SEC-03, QAS-SEC-04, QAS-SEC-05; ERR-006; TST-002, TST-013 |
| Approach | the authentication matrix and the full licence gate matrix as generated table-driven and property-based tests over the fictional families of the fixture set; a planted forbidden row in storage that the post-check must catch; a planted-secret sweep over every log sink; a timing test of the key check; mutation testing switched on for access code; second review for every change (SSC-018) |
| Entry criteria | 0.3.0 exit met; the fixture set covers every licence class |
| Exit criteria | zero forbidden items in the full matrix; every refusal path has a test; zero planted secrets found in any sink; mutation targets met on access code |
| Deliverables | gate matrix report, log sweep record, mutation report, completion report |
| Environment | local, CI ephemeral |
| Risks | a matrix generated from the same table the code reads, so both are wrong together (mitigated by an independent expected-value table reviewed by the owner); timing tests flaky on shared runners (mitigated by statistical test with many samples) |
| Suspension and resumption | suspend the release on any forbidden item served; resume when the defect is fixed and the full matrix is green |
| Issues | #7, #17 |
| Item | Content |
|---|---|
| Scope | per-IP and per-key limits, quotas, priority admission, the 2 s deadline with storage-side cancellation, metering |
| Features to test | RATE-001 to RATE-017; NFR-PERF-001, NFR-PERF-002; NFR-CAP-002, NFR-CAP-003; NFR-RES-001, NFR-RES-007; ERR-007, ERR-012; QAS-PERF-04, QAS-REL-01; TST-012 |
| Approach | unit and property-based tests of buckets and admission with an injected clock; quota reset across both daylight-saving changes; integration tests of cancellation with a sleeping stub query on the timeout and client-disconnect paths; the overload scenario on the test stack |
| Entry criteria | 0.4.0 exit met; test stack available |
| Exit criteria | no request past 2 s plus 50 ms in any test; backend idle within the cancel grace plus 100 ms; 1 000 cancellation iterations without a poisoned connection; the higher tier keeps its target under overload while the lower tier gets 429 with Retry-After
|
| Deliverables | overload report, cancellation report, completion report |
| Environment | CI ephemeral for functional tests; test stack for overload |
| Risks | timing-dependent tests flaky (mitigated by injected clocks and tolerance bands); overload results depending on runner size (mitigated by fixed container limits) |
| Suspension and resumption | suspend if a cancellation leak is observed; resume after the fix and 1 000 clean iterations |
| Issues | #8 |
| Item | Content |
|---|---|
| Scope | the feature endpoints in the planned order, composed features, condition history |
| Features to test | FR-PLACE, FR-ROAD, FR-ENERGY, FR-PARK, FR-TRANSIT, FR-ACCESS, FR-WEATHER, FR-RULES, FR-BRIEF, FR-LOC, FR-GEN; ENV-020 to ENV-033; NFR-FRESH-001 to NFR-FRESH-003; NFR-RES-003; QAS-FUNC-01, QAS-FUNC-02, QAS-REL-03, QAS-SAFE-01; TEST-009, TEST-010; NFR-I18N-001, NFR-I18N-002 |
| Approach | the endpoint Definition of Done (TEST-010) per operation; mapping tests over the read-contract fixtures; parity on every operation with a counterpart; composed features tested with one part failing; i18n catalogue parity |
| Entry criteria | 0.5.0 exit met; the endpoint list reviewed by the owner; parity bundle for the pinned platform version available |
| Exit criteria | every operation meets TEST-010; 100 % fact equality in the parity suite; attribution on 100 % of data responses; a stale family produces lamp unknown, never green
|
| Deliverables | parity report, contract coverage table, completion report |
| Environment | local, CI ephemeral |
| Risks | parity drift when the platform changes (mitigated by the bundle pinned per platform version); storage names leaking into payloads (mitigated by the scope lint) |
| Suspension and resumption | suspend an endpoint's release if its parity fails; resume when the cause is found on either side and recorded |
| Issues | #10, #18 |
| Item | Content |
|---|---|
| Scope | HTTP caching, result cache with single-flight, compression, query plans, latency budgets in the pipeline; soak, stress, spike and chaos |
| Features to test | NFR-PERF-003 to NFR-PERF-006; NFR-RES-002, NFR-RES-004 to NFR-RES-006; PROT-040 to PROT-044; QAS-PERF-01 to QAS-PERF-03, QAS-REL-02; TST-007, TST-021 |
| Approach | per the [[Performance Test Plan |
| Entry criteria | 0.6.0 exit met; test stack with the benchmark profile |
| Exit criteria | every endpoint class within its budget; 304 or cache hit on at least 95 % of repeated polls; exactly one storage read under 50 identical concurrent requests; no sequential scan on time-series data; soak without memory growth beyond 10 % after warm-up |
| Deliverables | dated performance result files, chaos report, completion report |
| Environment | test stack only |
| Risks | budgets met on fixtures but not on production-shaped skew (mitigated by skewed distributions in the generator); result noise (mitigated by repeated runs) |
| Suspension and resumption | suspend if the test stack cannot be isolated from other load; resume when isolation is shown |
| Issues | #9, #19 |
| Item | Content |
|---|---|
| Scope | documentation, the published image, the public API host; full regression; release acceptance and the first conformance claim |
| Features to test | TEST-008, TEST-010, TEST-011, TEST-012; QAS-001; REL-001 to REL-009; SSC-007 to SSC-009; NFR-A11Y-001, NFR-A11Y-002; DX rows; QAS-INT-01, QAS-INT-02 |
| Approach | full regression of all levels; security acceptance (OWASP API Top 10 cases per operation, full DAST of staging, header and TLS checks); executed documentation examples; accessibility checks plus a manual pass; a usability check of the getting-started guide; image signature and provenance verified at deploy |
| Entry criteria | 0.7.0 exit met; licence matrix signed off (LIC-009); staging deployed from the confirmed release's digest (a release candidate publishes no image) |
| Exit criteria | every QAS scenario has an automated test or operational measurement; all release gates green on staging; no open S1 or S2; the conformance claim names the specification revision |
| Deliverables | conformance claim, requirement report, security and performance results, completion report, release notes |
| Environment | CI ephemeral, test stack, staging; production is not tested, only observed after deployment |
| Risks | documentation examples drifting (mitigated by execution in CI); unresolved SHOULD deviations (recorded in the release notes) |
| Suspension and resumption | suspend the release on any failed staging gate; resume after a fix forward and a fresh staging run. The staging image is already a published release, so a failed acceptance is answered by yanking it and cutting a new patch release |
| Issues | #4, #20 |
ViaFrei API: a public, keyed, read-only REST API to live German mobility data. Design phase; nothing on these pages is a measurement of a running service. Data on viafrei.de and in every API answer is attributed to its publishers under their licences.
About
Design
- Domain Model
- API Design
- Architecture Overview
- Security and Access Keys
- Rate Limits and Priority
- Performance and Caching
- Configuration
- Observability
Data Model
Technical specification
-
Technical Specification
- Interface
- Scope, Conventions and References
- Protocol
- Common Envelope
- Resource Model
- Endpoints, Places
- Endpoints, Roads
- Endpoints, Energy and Parking
- Endpoints, Public Transport and Accessibility
- Endpoints, Weather Risk, Driving Rules and Composed Features
- Service Endpoints
- Errors
- Authentication and Authorisation
- Rate Limiting, Quotas and Priority
- Data Licensing and Attribution
- Compatibility and Versioning
- OpenAPI Skeleton
- Quality
- Delivery and operations
- Interface
Testing
- Testing Strategy
- Testing and Quality (design summary)
Engineering
Technology choice
- Technology Evaluation
- Architecture: Go
- Architecture: Rust
- Architecture: TypeScript
- Stack Comparison
- Benchmark Plan
Reference