Repository navigation
v1.0.0
First tagged release.
api-vv-toolkit reads an OpenAPI 3.0/3.1 contract and a YAML file of written requirements, generates API tests that each trace to a requirement, runs them against a live service, and reports a requirements-to-test traceability matrix: which requirements are verified, failed, partially covered or not covered.
Included
- CLI
api-vvwithparse,generate,run,report,traceandvalidate-requirements. - Rule-based generators: happy path, schema and parameter boundaries (length, range, item counts, enums, wrong types, unexpected properties), required-field omission, auth-required checks, and status-code coverage.
- Optional LLM-assisted generator (Chat Completions or Messages API over httpx, keys from the environment). Every proposal must match a JSON schema and is validated against the spec before it enters the plan; the prompt, raw reply and accept/reject reason are stored in the plan. A recorded mock provider is used for tests, CI and the demo.
- A reviewable YAML test plan with stable ids and request-fingerprint de-duplication.
- An async executor with bounded concurrency and timeouts that checks status code, response body schema, documented headers and optional latency budgets.
- Reports in Markdown, self-contained HTML, CSV and JUnit XML, plus an exit-code gate (by default, any must-priority requirement that is failed, partially covered or not covered fails the run).
Demo
make demo starts a small demo API with two deliberately seeded defects, generates 31 cases (rule-based plus mock LLM: 1 proposal accepted, 1 rejected), runs them and writes the reports. Result: 29 passed, 2 failed, and the gate fails on exactly the two seeded defects:
- REQ-003: an off-by-one
maxLengthcheck (expected 422, got 201). - REQ-008: the wrong success status (expected 204, got 200).
Verified for this tag
- CI: https://github.com/ai-dev-2024/api-vv-toolkit/actions/runs/36503537583 (lint, format, mypy --strict and tests on Python 3.11 and 3.12; the demo job asserts the gate names exactly REQ-003 and REQ-008; Docker build and smoke test).
- Locally: 111 tests, 98.51% coverage.
Limitations
Response bodies are read fully into memory, and regex patterns from the spec run as written, so a hostile spec can be slow to validate. Remote $refs are rejected. See the README for the full list.