OpenAPI Specification and Swagger UI

OpenAPI 3.1 Specification & Interactive Swagger UI 📖

OpenCrawling provides native, platform-wide OpenAPI 3.1 Specification generation and an interactive Swagger UI explorer bundled directly within oc-runtime.

This contract-first foundation standardizes API documentation, guarantees end-to-end type safety across the frontend (oc-admin-ui) and client SDKs (oc-java-client-sdk), enables direct LLM/AI Agent tool integration via function calling, and provides developers with an instant, interactive API exploration portal.

---

🚀 Live Endpoints

When oc-runtime is running (default port 8080), the following OpenAPI endpoints are immediately accessible:

Resource	URL	Description
Interactive Swagger UI	http://localhost:8080/swagger-ui.html	Interactive web interface to browse, inspect schemas, and execute API requests directly from the browser
OpenAPI 3.1 JSON Specification	http://localhost:8080/v3/api-docs	Machine-readable OpenAPI 3.1 specification formatted as JSON
Admin UI Proxied Swagger UI	http://localhost:3000/swagger-ui.html	Proxied through the oc-admin-ui Nginx reverse proxy in Docker environments
Admin UI Proxied API Docs	http://localhost:3000/v3/api-docs	Proxied OpenAPI 3.1 spec through Nginx

---

🎯 Motivation & Ecosystem Value

OpenCrawling is built on Java 25 (Virtual Threads, Structured Concurrency) and Spring Boot 4. As the platform scales across enterprise lakehouses, search engines, and multi-agent AI ecosystems, maintaining a machine-readable, living API contract delivers major strategic advantages:

graph TD
    subgraph OpenCrawling Runtime - Java 25 & Spring Boot 4
        Controllers[REST Controllers: Jobs, Connectors, OIS, Reports, Observability, etc.]
        Springdoc[springdoc-openapi-starter-webmvc-ui v3.1+]
        SwaggerUI[Swagger UI Explorer :8080/swagger-ui.html]
        JsonYaml[OpenAPI 3.1 Spec :8080/v3/api-docs]
        
        Controllers --> Springdoc
        Springdoc --> SwaggerUI
        Springdoc --> JsonYaml
    end

    subgraph Build & CI Automation
        Maven[Maven Build / Reactor]
        SpecArtifact[openapi.yaml / openapi.json Artifact]
        OasDiff[oasdiff / Spectral Contract Linter]
        
        Springdoc -.->|Build Spec Export| Maven
        Maven --> SpecArtifact
        SpecArtifact --> OasDiff
    end

    subgraph Downstream Consumers & AI Ecosystem
        AdminUI[oc-admin-ui: React 19 / TypeScript Client Generator]
        JavaSDK[oc-java-client-sdk: Validated Contract Alignment]
        MultiSDK[Future SDKs: Python / Go / Rust via openapi-generator]
        AIAgents[AI Agents / MCP Tools / LLM Function Calling]
        APIGateway[Enterprise API Gateways: Kong / Envoy / Apigee]
        
        SpecArtifact -->|openapi-typescript| AdminUI
        SpecArtifact -->|Contract Parity| JavaSDK
        SpecArtifact -->|openapi-generator-cli| MultiSDK
        SpecArtifact -->|OpenAPI Tool Definitions| AIAgents
        SpecArtifact -->|Import Spec| APIGateway
    end

Loading

1. Single Source of Truth & Zero Documentation Drift:
   • API endpoints, query parameters, request bodies, and response schemas are declared directly on Spring Boot @RestController classes using standard OpenAPI annotations (@Operation, @ApiResponse, @Parameter, @Schema).
2. Interactive Developer Portal & Exploration:
   • DevOps engineers and developers can test endpoints, submit payloads, and inspect live responses using the Swagger UI "Try it out" feature without needing external tools like Postman or raw curl.
3. End-to-End Type Safety for oc-admin-ui:
   • Eliminates handwritten Axios interfaces in React 19 / Vite. Downstream TypeScript definitions and client stubs can be synthesized directly via openapi-typescript (npm run generate:api), preventing payload mismatches.
4. Client SDK Parity & Multi-Language Generation:
   • Synchronizes oc-java-client-sdk with backend contract changes and enables rapid generation of official client libraries for Python, Go, and Node.js via openapi-generator-cli.
5. LLM Function Calling & AI Agent Integration:
   • Modern LLMs (OpenAI GPT-4o, Anthropic Claude 3.5, Google Gemini 2.0) and agent frameworks (LangChain, LlamaIndex, Semantic Kernel) natively translate OpenAPI 3.1 schemas into function-calling declarations, allowing autonomous agents to operate OpenCrawling pipelines dynamically.
6. Enterprise API Gateway Compatibility:
   • Ingest standardized OpenAPI 3.1 contracts into enterprise API gateways (Kong, Envoy, Apigee, AWS API Gateway, Traefik).

---

🗂️ Logical Domains & REST API Coverage

The OpenAPI specification structures OpenCrawling's REST endpoints into 10 core functional domains, declared in OpenApiConfig:

1. 🗂️ Pipeline & Jobs (/api/jobs)

Full lifecycle orchestration and monitoring of crawling jobs:

• GET /api/jobs: List all configured jobs with pagination (page, size) and optional status filtering (IDLE, RUNNING, PAUSED, FAILED, COMPLETED).
• POST /api/jobs: Create a new ingestion job with repository, transformation, and output connector bindings.
• GET /api/jobs/{id}: Fetch detailed job configuration, pipeline mode (rag or migration), and execution statistics.
• DELETE /api/jobs/{id}: Delete a job and its associated execution state.
• POST /api/jobs/{id}/start: Trigger immediate job crawl execution.
• POST /api/jobs/{id}/stop: Gracefully stop a running job.
• POST /api/jobs/{id}/pause: Pause an active crawling process.
• POST /api/jobs/{id}/resume: Resume a paused crawling job.
• GET /api/jobs/{id}/status: Retrieve real-time execution status, processed document counts, and error counts.
• GET /api/jobs/{id}/ledger: Paginated document-level audit ledger for the specified job.

2. 🔌 Connectors (/api/connectors)

Registry and diagnostic testing for repository, output, and transformation connectors:

• GET /api/connectors: List all registered connectors.
• POST /api/connectors: Register a new connector instance with credentials and configuration.
• GET /api/connectors/{id}: Fetch connector details and parameters.
• DELETE /api/connectors/{id}: Remove a registered connector.
• GET /api/connectors/types: Discover all supported connector types across repository, transformation, and output categories.
• POST /api/connectors/{id}/check: Execute a live connection test against the target data source or vector store, returning status (SUCCESS, FAILED) and diagnostic messages.

3. 📥 OIS Direct Ingestion (/api/v1/ingest/ois)

High-throughput document ingestion adhering to the Open Ingestion Standard (OIS):

• POST /api/v1/ingest/ois/{jobId}: Publishes an OIS document payload (UPSERT or DELETE tombstone) directly into the active job pipeline.
• Security: Protected by OisTokenAuth HTTP Bearer token authentication.
• Responses:
  • 202 Accepted: Document payload accepted and queued onto the Kafka event bus.
  • 400 Bad Request: Malformed JSON or invalid OIS schema structure.
  • 401 Unauthorized: Missing or invalid Bearer token.
  • 404 Not Found: Ingestion endpoint disabled or target job ID not found.
  • 409 Conflict: Job is configured in migration mode (OIS HTTP ingestion supports RAG mode).
  • 503 Service Unavailable: Kafka ingestion event bus unreachable.

4. 📊 Reports & Lakehouse Audit (/api/reports)

Lakehouse post-job audit manifests, ledger querying, and OLAP analytics:

• GET /api/reports: List execution summary reports for all completed jobs.
• GET /api/reports/{jobId}: Retrieve audit summary for a specific job (total documents, success/fail counts, duration, AI token consumption).
• GET /api/reports/{jobId}/ledger: Paginated ledger entries with filtering by status and document URI.
• GET /api/reports/{jobId}/export: Stream audit ledger in columnar or structured formats (?format=parquet|csv|avro|json).
• POST /api/reports/{jobId}/query: Execute serverless DuckDB SQL queries directly against the job's Parquet ledger file.

5. 🤖 Observability & AIOps (/api/observability)

Telemetry, tracing, and automated Root Cause Analysis:

• GET /api/observability/diagnose/{jobId}: Generate an AI-powered Root Cause Analysis (RCA) diagnosis for failed or warning jobs.
• GET /api/observability/traces/{jobId}: Retrieve correlated OpenTelemetry distributed trace spans for the job execution timeline.
• GET /api/observability/errors/{jobId}: Retrieve error logs and virtual thread stack traces.
• GET /api/observability/metrics: Global runtime metrics (virtual thread counts, Kafka consumer lag, memory allocation).

6. ⚙️ System & Settings (/api/system)

System runtime diagnostics and global configuration:

• GET /api/system/status: JVM runtime status, memory consumption, active thread count, and uptime.
• GET /api/system/throughput: Real-time document parsing and vector indexing throughput metrics.
• GET /api/system/logs: Live application log stream with log-level filtering (INFO, WARN, ERROR, DEBUG).
• GET /api/system/settings: Retrieve global system settings (Ollama/OpenAI embedding provider parameters, Apache Ozone storage configuration, Apache Tika process-isolation limits).
• POST /api/system/settings: Update runtime settings dynamically.

7. 🌐 Transport Settings (/api/system/settings/transport, /api/v1/admin/settings/transport)

Configuration and diagnostics for high-performance internal microservice communication:

• GET /api/system/settings/transport: Retrieve current transport settings (operational mode: AUTO, GRPC, or REST; port; message size limits; TLS settings).
• PUT /api/system/settings/transport: Update transport configuration.
• POST /api/system/settings/transport/ping: Trigger a live gRPC ping probe against the configured transport port to verify connectivity and measure latency.

8. 🔄 ManifoldCF Migration (/api/mcf-migration)

Automated migration of legacy Apache ManifoldCF configurations into OpenCrawling:

• POST /api/mcf-migration/plan: Analyze ManifoldCF connection/job snapshots or live API endpoints, generating a structured migration plan with visual compatibility scores and field-level fidelity notes (DROPPED, DEFAULTED, CONVERTED, SCOPE_CHANGE, RUNTIME_RISK).
• POST /api/mcf-migration/apply: Execute the migration plan, provisioning native OpenCrawling connectors and jobs with automatic idempotency.

9. 🔎 Vespa Insights (/api/vespa)

Vespa vector engine diagnostics and schema management:

• GET /api/vespa/health: Verify Vespa configuration and container cluster availability.
• GET /api/vespa/count: Query total indexed document count across schema clusters.
• POST /api/vespa/query: Execute diagnostic BM25, hybrid, or ANN vector search queries directly against Vespa.
• POST /api/vespa/deploy/bundled: Deploy bundled schema application packages (opencrawling-document.sd).
• POST /api/vespa/deploy/custom: Upload and deploy custom multi-vector schemas via multipart ZIP upload.

10. ✍️ Auto-Narrativization Copilot (/api/transformation/copilot)

AI-powered Mustache narrative template generation:

• POST /api/transformation/copilot/generate: Generate Mustache narrative templates and mock datasets from structured tabular dataset schemas (field names, data types, descriptions) using local Ollama or cloud OpenAI models.

---

🔒 Security Schemes & Authentication

OpenCrawling defines the OisTokenAuth security scheme in OpenAPI 3.1:

components:
  securitySchemes:
    OisTokenAuth:
      type: http
      scheme: bearer
      bearerFormat: Token
      description: Bearer token authentication for OIS document direct ingestion (/api/v1/ingest/ois/{jobId})

Using Authentication in Swagger UI

1. Open Swagger UI at http://localhost:8080/swagger-ui.html.
2. Click the Authorize 🔓 button in the top right corner.
3. Enter your bearer token in the OisTokenAuth field:

   your-ois-ingestion-secret-token
   

4. Click Authorize and then Close. All requests to /api/v1/ingest/ois/{jobId} will automatically include the Authorization: Bearer <token> header.

---

⚙️ Configuration Properties

OpenAPI generation and Swagger UI are managed in oc-runtime via standard Spring Boot properties in application.yml:

springdoc:
  api-docs:
    path: ${SPRINGDOC_API_DOCS_PATH:/v3/api-docs}
    enabled: ${SPRINGDOC_API_DOCS_ENABLED:true}
    version: OPENAPI_3_1
  swagger-ui:
    path: ${SPRINGDOC_SWAGGER_UI_PATH:/swagger-ui.html}
    enabled: ${SPRINGDOC_SWAGGER_UI_ENABLED:true}
    display-request-duration: true
    operations-sorter: method
    tags-sorter: alpha
    doc-expansion: none
  show-actuator: false

Environment Variable Overrides

Environment Variable	Description	Default
SPRINGDOC_API_DOCS_PATH	Path to the JSON OpenAPI specification	/v3/api-docs
SPRINGDOC_API_DOCS_ENABLED	Enable or disable OpenAPI JSON generation	true
SPRINGDOC_SWAGGER_UI_PATH	Path to the Swagger UI explorer	/swagger-ui.html
SPRINGDOC_SWAGGER_UI_ENABLED	Enable or disable the Swagger UI web portal	true

---

🌐 Reverse Proxy & SPA Routing Integration

1. Docker Nginx Proxy (oc-admin-ui/nginx.conf)

In containerized environments (such as docker-compose-apps.yml or docker-compose-decoupled.yml), requests hitting the frontend container on port 3000 for Swagger UI or API docs are transparently forwarded to the backend runtime:

# Proxy OpenAPI and Swagger UI calls to the Spring Boot backend
location ~ ^/(v3|swagger-ui) {
    proxy_pass http://opencrawling-backend:8080;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
}

2. Spring WebMVC SPA Route Forwarding (WebConfig.java)

In oc-runtime, single-page application (SPA) fallback routing is configured to preserve Swagger UI and OpenAPI documentation paths:

// Forward all non-api, non-swagger, and non-static-resource routes to index.html for SPA routing
registry.addViewController("/{path:^(?!api|v3|swagger-ui|webjars)[^\\.]*}")
        .setViewName("forward:/index.html");

---

💻 Client SDK & Downstream Generation

TypeScript Types for oc-admin-ui

With OpenAPI 3.1 enabled, frontend types can be generated automatically:

npx openapi-typescript http://localhost:8080/v3/api-docs -o src/lib/api-generated.ts

This guarantees that DTOs in React components are synchronized with Java backend records.

Java Client SDK (oc-java-client-sdk)

The oc-java-client-sdk models align directly with the DTO records documented in OpenAPI (JobDTO, ConnectorDTO, SystemSettingsDTO, MigrationRequest, etc.) and expose a dedicated client.openApi() sub-client (OpenApiClient) for programmatic contract inspection:

OpenApiStatusSummary status = client.openApi().checkStatus();
List<OpenApiEndpointSummary> oisOps = client.openApi().listEndpoints("OIS Direct Ingestion", "POST");
Optional<JsonNode> jobSchema = client.openApi().getComponentSchema("JobDTO");

OpenCrawling CLI (oc openapi)

The OpenCrawling CLI (oc-cli) provides the oc openapi command group for terminal-native contract inspection, YAML/JSON export, strict validation, breaking-change drift detection, and LLM tool export:

# Probe live /v3/api-docs and /swagger-ui.html status
oc openapi status

# Export live OpenAPI 3.1 contract to YAML
oc openapi export --format yaml --output ./openapi.yaml

# Filter endpoints by domain tag or HTTP verb
oc openapi endpoints --tag "OIS Direct Ingestion"

# Inspect a component schema (e.g. JobDTO)
oc openapi schema --name JobDTO

# Validate contract completeness and fail CI on breaking API changes
oc openapi validate --file ./openapi.yaml --strict
oc openapi diff --base ./baseline-openapi.yaml --target ./openapi.yaml --fail-on-breaking

# Export OpenAPI operations as OpenAI / Gemini / Anthropic / MCP function-calling tools
oc openapi tools --provider openai --tag "Pipeline & Jobs" --output ./ai-tools.json

---

🧪 Automated Contract & Documentation Testing

OpenCrawling enforces strict API documentation standards through automated CI tests (OpenApiDocumentationTest.java):

• Tag Verification: Confirms every @RestController carries a valid @Tag annotation with a non-blank description matching the official 10 domain tags.
• Operation & Response Verification: Asserts every public REST mapping method declares @Operation summaries and explicit @ApiResponse or @ApiResponses status codes.
• Schema Annotation Checks: Verifies core DTO records (such as JobDTO, ConnectorDTO, SystemSettingsDTO, QueryRequest) are decorated with @Schema.
• Model Converter Resolution: Validates that Swagger ModelConverters resolves Java 25 records and DTOs into valid JSON schemas without reflection warnings or runtime errors.