Skip to content

Contract and Schema Testing

mavrovde edited this page Nov 1, 2026 · 1 revision

Contract and Schema Testing

The test design for the API contract: the OpenAPI document checked in both directions, schema-driven fuzzing, breaking-change detection, executed examples and the parity contract with the ViaFrei MCP server. Part of the Testing Strategy. Requirements: TEST-003, TEST-004, TEST-005, TEST-006, TEST-009 on Spec: Conformance and Acceptance; TST-015, TST-016 on Spec: Test Strategy; VER-001, VER-002, VER-006 on Spec: Compatibility and Versioning. The shared components are sketched on Spec: OpenAPI Skeleton.

1. What "the contract" is

Contract Between Checked by
The OpenAPI 3.1 document the API and its clients two-way conformance, fuzzing, breaking-change diff, executed examples
The common envelope and problem details every operation and every client schema rules in the document, contract assertions (ENV, ERR)
The read contract the platform's storage and the API mapping and integration tests, readiness refusal on an incompatible version (VER-009)
Fact parity the API and the MCP server the parity harness

2. Two-way conformance

flowchart LR
  D[OpenAPI document] --> V[Validator]
  D --> NG[Negative request generator]
  D --> EX[Example requests]
  HT[Handler, integration and end-to-end tests] -->|every response| V
  NG -->|one violation each| SVC[Service under test]
  EX --> SVC
  SVC -->|responses| V
  V --> R[Conformance report<br/>operation by status coverage]
Loading

Text description of the diagram: the document feeds a validator, a negative request generator and the set of example requests; every response produced by any test, and every response to generated or example requests, passes through the validator, which writes a conformance report with the coverage of operation by status.

2.1 Outbound: response-schema validation

Every response received in a handler, integration or end-to-end test is validated against the document (TEST-003):

Test condition Expected
Status code is declared for the operation otherwise fail
Every response header is declared, with its schema (rate-limit headers, ETag, Cache-Control, Retry-After, deprecation signals) otherwise fail
Body is valid against the schema for that status and media type otherwise fail
Data responses have exactly data, meta, links (ENV-001) with meta.requestId, generatedAt, language (ENV-002) and non-empty meta.attribution and meta.freshness (ENV-003) otherwise fail
Error responses are application/problem+json with the members of ERR-001 and a type from the catalogue on [[Spec: Errors Spec-Errors]]
additionalProperties is false on response schemas, so an extra member (possibly a storage name) fails otherwise fail

2.2 Inbound: request validation, negative direction

From the document, the generator derives one invalid request per constraint of every parameter and body member: wrong type, wrong format, below minimum, above maximum, too short, too long, pattern mismatch, value outside the enumeration, missing required, unknown parameter, repeated parameter. Each must be answered with 400 invalid-parameter, listing that violation in errors[] (ERR-005), and must not reach storage. Requests combining several violations must list all of them.

2.3 Inbound: request validation, positive direction

Every example request in the document is sent to the service on the ci fixture set and must be answered with a documented status and a valid body. Boundary values (exact minimum and maximum) must be accepted.

2.4 Coverage

The harness prints a table of operation × documented status and marks each pair that has at least one validated response. The endpoint Definition of Done (TEST-010) needs every pair covered. An empty or unparsable document is a failure, not an empty table (TST-011).

3. Property-based API fuzzing

Item Design
Tool category schema-driven property-based testing of a running instance (for example Schemathesis), plus native fuzz targets per parser
Target only a CI-started instance on the ci fixture set, with test keys
Checks no 5xx other than the documented 503 causes; every response conforms (section 2.1); no response later than 2 s plus 50 ms; no response reveals a storage name or stack trace (ERR-004)
Stateful links operations that return cursors are chained, so cursor reuse with other filters is exercised (ENV-023)
Budgets (design) a short run on every pull request (issue #12); a long run nightly (issue #16)
Findings a reproducer is saved and becomes a unit or handler test in the fixing pull request

4. Breaking-change detection

The generated document is compared with:

  1. the document on main, on every pull request (issue #5);
  2. the last released document, on every pull request and on release (issue #15).

A removed operation, member or enum value, a parameter made required, a narrowed range or a changed type fails within a major (VER-002). Additions pass. The rule set is the one on Spec: Compatibility and Versioning section 2. A planted removal proves the check can fail.

5. Document quality

The document itself is linted (Spectral with the ruleset on Spec: API Lifecycle and Governance): every operation has a summary, a description, examples, an attribution note, declared problem responses, an operation id in the house style, and x-mcp-counterpart where one exists. The served document must be byte-identical to the committed one, minus disabled operations (FR-SVC-005).

6. Executed examples

Every example in the document and in the guide is executed against the fixture instance and validated (Docs method, issue #4). A failing example fails the documentation build; examples are never edited by hand to match output.

7. MCP parity

The API must answer the same facts as the MCP server for the same question (CON-007, FR-GEN-002, QAS-FUNC-01). The harness is built in issue #18.

sequenceDiagram
  participant P as MCP side
  participant B as Parity bundle
  participant H as Parity harness
  participant A as API under test
  P->>B: seed rows, questions, expected facts for platform version N
  H->>B: verify checksum and version
  H->>A: load seed into a throwaway database
  H->>A: ask the matching endpoint for each question
  A-->>H: response
  H->>H: project the response to facts and compare
Loading

Text description of the diagram: the MCP side produces a bundle of seed rows, questions and expected facts for a platform version; the harness verifies the bundle, loads the seed into a throwaway database, asks the API each question, projects the answer to facts and compares them with the expected facts.

Projection rule Detail
Identifiers normalised to the public id form
Names exact, Unicode NFC
Positions WGS 84, rounded to five decimals
Prices decimals, no float conversion
States and lamps codes, not labels
Instants UTC
Source families the set of families that contributed
Condition and purpose notes verbatim, original language

The comparison is exact on facts and ignores presentation (ordering only where the operation declares a sort, wording, envelope members). The inventory check fails when an operation marked x-mcp-counterpart has no parity case (TEST-009). The bundle is a file pinned by the version sync; the harness needs no network.

8. Test case examples

Requirement Level Given When Then
ENV-003 Contract the ci set with an area that has no charging station GET charging options near that area 200, data empty, meta.attribution lists every family consulted
ERR-005 Contract — a request with radius_m out of range and an unknown parameter 400 invalid-parameter with two entries in errors[]
VER-002 Contract a branch that removes an enum value the breaking-change check runs the pull request is red, naming the value
TEST-009 Parity the bundle for the pinned platform version the harness runs the departures questions every fact equal

Home

About

Design

Data Model

Technical specification

Testing

Engineering

Technology choice

Reference

Clone this wiki locally