Skip to content

Releases

v1.3.0

Choose a tag to compare

@github-actions github-actions released this 21 Oct 19:47
22c4d28

api-vv-toolkit v1.3.0

OpenAPI contract plus written requirements in; API tests, a requirement-to-verdict
traceability matrix and a CI gate out. This release adds OpenAPI 3.1, seeded negative and
property-based generation, contract drift checks, a reusable GitHub Action and a rebuilt HTML
report.

Highlights

  • Use it from any workflow: uses: muhib-karim/api-vv-toolkit@v1 validates, generates, runs
    and reports, and gates on requirement verdicts. With baseline-spec, it also fails on
    breaking contract drift. Outputs: gate, failed-requirements, breaking-changes,
    report-dir and junit.
  • api-vv diff old.yaml new.yaml: classifies every change as breaking or non-breaking from
    an existing client's point of view, as text, JSON or Markdown. Exits 1 on any breaking change.
    Renamed operationIds, renamed path placeholders and 3.0 → 3.1 migrations aren't reported as
    changes. Changes it can't analyse fail closed.
  • OpenAPI 3.1 alongside 3.0: type arrays, examples maps and webhooks (listed, never
    tested). Fixes a crash on 3.1 type-array parameters and lost null for 3.0 nullable +
    allOf.
  • Auth from environment variable names only: bearer, basic and apiKey in header, query or
    cookie. credentials_env must name variables; otherwise API_VV_AUTH_<SCHEME> is read. Each
    secured operation gets an invalid credentials negative.
  • Seeded negatives: pattern support, and
    generate --property-cases N --seed S for random valid (expect 2xx) and single-violation
    (expect 4xx) requests. The same seed regenerates a byte-identical plan.
  • Reports: an accessible, mobile-friendly HTML matrix (requirement → tests → result with
    "why it failed"), JUnit XML, CSV and Markdown. The Pages site is rebuilt by CI from a live run
    against the bundled demo API.
  • Measured: a live LLM gate re-run on 2026-10-21 with claude-opus-5-5, claude-sonnet-5 and
    gpt-6-sol (results/2026-10-21-llm-gate/). Acceptance was 95%, 76% and 95%.

Quality gates

These run in CI on every push: ruff, mypy --strict on the package, pytest with coverage ≥ 85%
on Python 3.11 and 3.12, CodeQL (security-and-quality), gitleaks, Dependabot, a wheel build and
smoke test, the seeded-defect demo, the Action dogfood workflow and a Docker build.

Compatibility

  • Plans keep version: "1.0". New fields (seed, property_cases, Spec.webhooks,
    auth: invalid) are optional, so 1.2 plans still load in 1.3. Plans written by 1.3 aren't
    readable by 1.2.
  • Generated plans gain one invalid credentials case per secured operation, which changes
    untraced-test counts.

Install

pip install https://github.com/muhib-karim/api-vv-toolkit/releases/download/v1.3.0/api_vv_toolkit-1.3.0-py3-none-any.whl

Verify the download against SHA256SUMS.txt attached to this release. Full details are in
CHANGELOG.md.

v1.2.0

Choose a tag to compare

@github-actions github-actions released this 17 Oct 19:21

api-vv-toolkit v1.2.0: LLM gate queries only opted-in requirements; live four-model study

v1.1.0

Choose a tag to compare

@muhib-karim muhib-karim released this 02 Oct 19:17

What's new in 1.1.0

  • Fix: execution timeouts and per-requirement latency budgets must be finite. NaN and infinity are rejected when the config or requirements file is loaded; an infinite value previously passed validation and effectively switched the limit off. New tests cover it.
  • Live demo report: CI runs the seeded-defect demo (a loopback Starlette API with two deliberate bugs) on every push to main and publishes the HTML traceability report, CSV matrix and JUnit XML to GitHub Pages: https://muhib-karim.github.io/api-vv-toolkit/ — the gate fails on exactly REQ-003 and REQ-008, as designed.
  • Packaging: author and project URLs in pyproject.toml.

CI: ruff, mypy --strict, pytest on Python 3.11 and 3.12, the demo gate check, and a Docker smoke test.

Full changelog: CHANGELOG.md

v1.0.0

Choose a tag to compare

@muhib-karim muhib-karim released this 29 Sep 00:33

First tagged release.

api-vv-toolkit reads an OpenAPI 3.0/3.1 contract and a YAML file of written requirements, generates API tests that each trace to a requirement, runs them against a live service, and reports a requirements-to-test traceability matrix: which requirements are verified, failed, partially covered or not covered.

Included

  • CLI api-vv with parse, generate, run, report, trace and validate-requirements.
  • Rule-based generators: happy path, schema and parameter boundaries (length, range, item counts, enums, wrong types, unexpected properties), required-field omission, auth-required checks, and status-code coverage.
  • Optional LLM-assisted generator (Chat Completions or Messages API over httpx, keys from the environment). Every proposal must match a JSON schema and is validated against the spec before it enters the plan; the prompt, raw reply and accept/reject reason are stored in the plan. A recorded mock provider is used for tests, CI and the demo.
  • A reviewable YAML test plan with stable ids and request-fingerprint de-duplication.
  • An async executor with bounded concurrency and timeouts that checks status code, response body schema, documented headers and optional latency budgets.
  • Reports in Markdown, self-contained HTML, CSV and JUnit XML, plus an exit-code gate (by default, any must-priority requirement that is failed, partially covered or not covered fails the run).

Demo

make demo starts a small demo API with two deliberately seeded defects, generates 31 cases (rule-based plus mock LLM: 1 proposal accepted, 1 rejected), runs them and writes the reports. Result: 29 passed, 2 failed, and the gate fails on exactly the two seeded defects:

  • REQ-003: an off-by-one maxLength check (expected 422, got 201).
  • REQ-008: the wrong success status (expected 204, got 200).

Verified for this tag

Limitations

Response bodies are read fully into memory, and regex patterns from the spec run as written, so a hostile spec can be slow to validate. Remote $refs are rejected. See the README for the full list.