Skip to content

Software Requirements Specification

Jump to bottom
hasancankeles edited this page Mar 20, 2026 · 1 revision

Table of Contents

Introduction

Purpose

This document specifies the functional and non-functional requirements for the Agentic API Testing Platform. The project goal is to deliver an autonomous API testing product that can ingest API specs, generate test suites with LLMs, execute tests, and provide actionable reporting.

Document Conventions

  • Functional requirements are listed under section 1.x and use shall statements.
  • Non-functional requirements are listed under section 2.x and are measurable where possible.

Product Scope

The platform targets backend/API teams that want faster regression coverage with less manual scripting. It currently focuses on:

  • OpenAPI ingestion and normalization.
  • Two-stage agentic test generation (planner + per-test executor).
  • HTTP and WebSocket functional execution.
  • k6-based load testing.
  • Run summaries, filtering, diffs, and generation debug artifacts.

References

  • Internal project planning and implementation notes.

Glossary

Parsed API: Normalized internal representation of an API spec including endpoints, params, and schemas.

Test Suite: A named collection of HTTP and/or WebSocket test cases.

Test Case: A runnable API test definition including endpoint, method, inputs, expected status, and assertions.

Load Scenario: A k6-ready performance test definition with VUs, duration/stages, thresholds, and target URL.

Generation Artifact: Persisted debug object that stores planner output, executor outcomes, fallback usage, and generation metadata.

Fallback: Deterministic conversion path used when LLM executor output fails or is unavailable.

Run: One execution instance of generated suites, producing run-level summary and per-test results.

Environment Profile: Named configuration bundle (e.g., base URL, tokens, variables) used to run tests in different environments.

1. Functional Requirements

1.1 API Specification Ingestion
  • 1.1.1 The system shall accept API specs via one of spec_url, spec_path, or spec_content.
  • 1.1.2 The system shall parse OpenAPI JSON/YAML and extract title, description, version, base URL, endpoints, parameters, responses, and component schemas.
  • 1.1.3 The system shall persist each parsed API as a versioned record with parse timestamp.
  • 1.1.4 The system shall reject parse requests with missing source input using HTTP 400.
  • 1.1.5 The system shall return parse failures as HTTP 400 with error details.
  • 1.1.6 The parser shall detect WebSocket message hints from API description text when present.
  • 1.1.7 The system shall support Postman collection ingestion.
  • 1.1.8 The system shall support cURL ingestion and conversion into normalized endpoint definitions.
  • 1.1.9 The system shall support manual endpoint entry for APIs without formal specs.
1.2 Agentic Test Generation
  • 1.2.1 The system shall generate test artifacts for categories individual, suite, and load.
  • 1.2.2 The generation pipeline shall use a planner stage and an executor stage.
  • 1.2.3 Planner output shall be validated against strict schema contracts before use.
  • 1.2.4 Executor output shall be validated per test case; invalid executor outputs shall not crash generation.
  • 1.2.5 The system shall run per-test executor jobs with bounded concurrency configurable by environment variable.
  • 1.2.6 On per-test executor failure, the system shall apply deterministic fallback and continue generation.
  • 1.2.7 The generation response shall include generation_id, generated suites/scenarios, and generation_meta counters.
  • 1.2.8 Structured generation validation failures shall return HTTP 422.
  • 1.2.9 Upstream model failures shall return HTTP 502 or 503.
  • 1.2.10 When debug artifacts are enabled, the system shall persist planner plan, executor outcomes, fallback case IDs, and final outputs.
  • 1.2.11 Raw LLM output capture shall be optional and disabled by default.
  • 1.2.12 Sensitive keys in debug payloads (e.g., authorization, token, api_key, password, cookie) shall be redacted before persistence.
  • 1.2.13 Users shall be able to add or edit tests with natural language instructions.
1.3 HTTP Test Execution
  • 1.3.1 The system shall execute generated HTTP tests against a target base URL.
  • 1.3.2 The system shall support methods GET, POST, PUT, DELETE, PATCH, OPTIONS, and HEAD.
  • 1.3.3 The system shall resolve path parameters before request dispatch.
  • 1.3.4 The system shall support request headers, query parameters, and JSON body payloads.
  • 1.3.5 Individual HTTP requests shall timeout at 30 seconds.
  • 1.3.6 Assertion operators shall support at least eq, ne, gt, lt, gte, lte, contains, exists, and type.
  • 1.3.7 Each executed test result shall store expected and actual status/body, assertion pass counts, response time, and error message when applicable.
  • 1.3.8 The system shall persist run-level summary metrics (total, passed, failed, errors, average response time).
  • 1.3.9 If suite IDs are omitted, execution shall target the latest generated suite batch by default.
1.4 WebSocket Test Execution
  • 1.4.1 The system shall execute WebSocket tests using step-based actions (send, expect).
  • 1.4.2 Each expect step shall support timeout control in seconds.
  • 1.4.3 WebSocket assertions shall support eq, ne, exists, contains, and type.
  • 1.4.4 WebSocket run results shall include step-level expected/actual details.
  • 1.4.5 WebSocket test failures and connection errors shall be recorded as failed or error results without stopping other tests.
1.5 Load Test Generation and Execution
  • 1.5.1 The system shall materialize load scenarios into executable k6 scripts.
  • 1.5.2 Load scenarios shall support fixed load (vus + duration) and staged ramp definitions.
  • 1.5.3 Load scenarios shall support threshold configuration and custom headers.
  • 1.5.4 The system shall execute k6 and parse output metrics including latency percentiles, RPS, failure rate, and data transfer.
  • 1.5.5 A load run shall timeout after 600 seconds.
  • 1.5.6 Missing k6 installation shall return a structured error payload in results.
  • 1.5.7 If scenario IDs are omitted, the system shall run the latest generated scenario batch by default.
1.6 Results, Dashboard, and Debug Observability
  • 1.6.1 The system shall provide endpoints to list suites, suite details, suite results, and global results.
  • 1.6.2 Global results shall support filtering by status, category, endpoint substring, and run ID.
  • 1.6.3 The results listing API shall support pagination (limit, offset) with upper bound enforcement.
  • 1.6.4 The system shall provide a dashboard summary API for the latest run including pass/fail/error counts, pass rate, average latency, and recent runs.
  • 1.6.5 The system shall expose generation artifact APIs for listing and retrieval by generation_id.
  • 1.6.6 Generation artifact retrieval shall omit raw LLM outputs unless explicitly requested.
  • 1.6.7 The backend shall emit structured lifecycle logs for parse, generate, execute, and load test flows.
1.7 Web Frontend Flows
  • 1.7.1 The frontend shall provide a dashboard flow for parsing spec, generating tests, and executing tests.
  • 1.7.2 The frontend shall provide a suites page with suite-level run action and expandable suite details.
  • 1.7.3 The frontend shall provide a results page with filtering and expected-vs-actual response diff view.
  • 1.7.4 The frontend shall provide a load test page with run single/run all controls and percentile visualization.
  • 1.7.5 The frontend shall support responsive rendering for desktop and mobile form factors.
1.8 CI/CD Automation
  • 1.8.1 The system shall provide a non-interactive test runner for CI/CD usage.
  • 1.8.2 CI/CD execution shall support run modes by suite ID, group ID, and tag filters.
  • 1.8.3 CI/CD execution shall support environment profile selection per pipeline run.
  • 1.8.4 CI/CD execution shall return machine-readable summary output and non-zero exit codes on configured failure thresholds.
  • 1.8.5 The project shall provide reference pipeline templates for GitHub Actions, GitLab CI, and Jenkins.
1.9 Collaboration and Governance
  • 1.9.1 Users shall be able to group test suites and run them as a single operation.
  • 1.9.2 Users shall be able to assign tags to suites/tests and execute by tags.
  • 1.9.3 The system shall support environment profiles with scoped variables and secrets.
  • 1.9.4 The system shall provide role-based access control for admin, editor, and viewer roles.
  • 1.9.5 The system shall maintain an audit trail for test edits, execution triggers, and approval actions.

2. Non-Functional Requirements

2.1 Performance and Scalability
  • 2.1.1 The API shall support at least 30 concurrent users for dashboard, listing, and result-filter queries without service failure.
  • 2.1.2 The results listing endpoint shall return responses within 2 seconds for up to 10,000 stored result rows under local benchmark conditions.
  • 2.1.3 The generation executor concurrency shall be configurable in range 1..32.
  • 2.1.4 The platform shall support at least 100 concurrent users for read-heavy operations.
2.2 Reliability and Availability
  • 2.2.1 The system shall persist parse, generation, execution, and load test records durably in SQLite.
  • 2.2.2 Runtime failures in one test case shall not terminate the entire test run.
  • 2.2.3 Generation failures in one executor case shall not invalidate the whole generation batch when fallback can be applied.
  • 2.2.4 All API errors shall return structured HTTP error payloads with meaningful detail.
  • 2.2.5 The deployed platform shall target 99.0% monthly uptime in managed environments.
2.3 Security and Privacy
  • 2.3.1 Secrets used for model providers and external services shall be provided via environment variables and never hardcoded.
  • 2.3.2 Debug artifact persistence shall redact sensitive key-value pairs before write.
  • 2.3.3 Raw LLM output capture shall be off by default and explicitly opt-in.
  • 2.3.4 Production CORS shall be restricted to approved frontend origins.
  • 2.3.5 Access to generation artifacts and run history shall require authenticated and authorized users.
2.4 Usability and Accessibility
  • 2.4.1 The UI shall keep core actions (parse, generate, execute, run load tests) discoverable within one navigation level.
  • 2.4.2 The UI shall surface user-friendly error messages for parse, generation, execution, and load failures.
  • 2.4.3 The web UI shall meet WCAG 2.1 AA contrast and keyboard navigation criteria.
  • 2.4.4 The platform shall support locale-aware date/time formatting for run history views.
2.5 Maintainability and Quality
  • 2.5.1 Backend request/response contracts shall be defined with typed schema models.
  • 2.5.2 The repository shall include automated tests for generation route behavior and core generator logic.
  • 2.5.3 CI shall enforce unit and integration test execution on each pull request.
  • 2.5.4 The project shall target at least 75% backend unit-test coverage.
2.6 Deployment and Portability
  • 2.6.1 The backend shall run locally with FastAPI/Uvicorn and SQLite without requiring external infrastructure.
  • 2.6.2 The frontend shall run as a Vite React application and communicate with backend APIs over HTTP.
  • 2.6.3 The system shall provide containerized deployment manifests for backend, frontend, and CI test runner components.
  • 2.6.4 The system shall support environment-specific configuration for development, staging, and production.

Clone this wiki locally