Skip to content

Spec Conformance and Acceptance

mavrovde edited this page Nov 1, 2026 · 2 revisions

Spec: Conformance and Acceptance

Part of the Technical Specification. Requirement prefix: TEST. Defines when an implementation conforms to this specification and how each verification method in the Spec: Traceability Matrix is carried out. The layering and quality bars are in Spec: Test Strategy; the stack-selection measurements are in Benchmark Plan. How each method is carried out in practice is elaborated by the testing document set, starting at Testing Strategy; the per-milestone entry and exit criteria are on Test Plan, and the security acceptance cases of TEST-008 on Security Test Plan.

1. Conformance

An implementation conforms to version N of this specification when every MUST and MUST NOT requirement on the Spec pages is met and verified by its listed method, and every SHOULD that is not met is recorded with its reason in the release notes. MAY items are optional. A conformance claim names the specification revision (the wiki commit) it was verified against.

flowchart LR
  R[Requirement row] --> M[Verification method]
  M --> T[Test or review artefact]
  T --> E[Release evidence]
  E --> C[Conformance claim per release]
Loading

2. Verification methods

Method What it means Where it runs
Unit Pure function or component test with an injected clock and no I/O every pull request
Handler The HTTP handler against a fixture-backed storage stub, asserting status, headers and body every pull request
Contract Responses validated against the published OpenAPI document; the document diffed against the previous release every pull request
Mapping Fixture rows from the read contract mapped to API entities, compared with golden files every pull request
Parity The same fixture question asked of the API and of the MCP server; the facts compared every pull request
Load Load and soak runs on a test stack, never on production nightly and before a release
Security Automated security tests (authentication, authorisation, injection, headers, dynamic scan) every pull request and nightly
Config A configuration key set to a non-default value and the change in behaviour observed every pull request
Lint A static rule over code, OpenAPI document or repository every pull request
Docs A documentation artefact exists and its examples execute every release
Review A recorded human review with a named reviewer as stated
Ops An operational control observed in a running environment, with evidence release and periodically

3. Requirements

ID Requirement Verification
TEST-001 Every requirement row MUST name a verification method from section 2, and every MUST row with an automated method MUST have at least one test that carries the requirement ID in its name or metadata. Lint
TEST-002 CI MUST produce a report listing every requirement ID with its tests and their result; an automated MUST row with no passing test fails the release. Lint
TEST-003 Contract tests MUST validate every response of every handler test against the OpenAPI document, including problem responses and headers. Contract
TEST-004 A breaking-change check MUST compare the generated OpenAPI document with the last released one and fail on a removed operation, member, enum value or a tightened parameter within a major (VER-002). Contract
TEST-005 Property-based tests MUST cover parameter parsing (coordinates, bbox, polyline, instants, lists) and cursor round trips. Unit
TEST-006 Fuzzing MUST run against the request parsers and the query POST body decoder; a crash or a hang over the deadline fails. Security
TEST-007 Performance acceptance MUST use the method of [[Benchmark Plan Benchmark-Plan]]: a fixed, production-shaped synthetic data set, an open-model load generator, warm-up excluded, percentiles from full histograms; the targets are those of NFR-PERF and NFR-CAP.
TEST-008 Security acceptance MUST include: OWASP API Top 10 test cases per operation (broken object and function authorisation, resource consumption, mass assignment of query parameters), a dynamic scan of the sandbox, a header check, and the licence-gate tests of SEC-030 to SEC-032. Security
TEST-009 The parity suite MUST cover every operation that has an MCP counterpart (x-mcp-counterpart). Parity
TEST-010 Endpoint Definition of Done. An operation is done only when all of these hold: the approved OpenAPI fragment (LCY-001); handler tests for every status code it can return; contract validation; mapping tests over the read-contract fixtures; a parity test where a counterpart exists; licence-gate tests for every source family it reads; a configuration test for each of its limits; latency within its class in the nightly load run; documentation with executed examples; a requirement row on its Spec page; an owner in the code-owners file; and an entry in the operation inventory. Review
TEST-011 Release acceptance MUST record the conformance claim, the requirement report (TEST-002), the load and security results, and the open SHOULD deviations. Review
TEST-012 A requirement that cannot be verified by its method MUST be rewritten or downgraded; an unverifiable MUST is a specification defect. Review
TEST-013 Every threat row (THR) and every OWASP API Security Top 10 (2023) risk MUST map to at least one automated test case, listed on [[Security Test Plan Security-Test-Plan]]; a threat or risk without a test fails the release acceptance.

Home

About

Design

Data Model

Technical specification

Testing

Engineering

Technology choice

Reference

Clone this wiki locally