AI Powered Observability AIOps

AI-Powered Observability & Log Analysis (AIOps) via OpenTelemetry

OpenCrawling® AI-Powered Observability (AIOps) leverages OpenTelemetry (OTel) correlated distributed spans, Micrometer performance metrics, and Spring AI Model Context Protocol (MCP) tools to automatically translate complex Java 25 Virtual Thread execution stack traces into human-readable Root Cause Analysis (RCA) reports.

When an enterprise data ingestion job fails, experiences backpressure, or suffers throughput degradation, system administrators can trigger instant AI diagnostics directly from the oc-admin-ui administration console or via LLM diagnostic agents.

---

🏗️ Architecture & Component Design

The AIOps observability system integrates telemetry capturing across all major OpenCrawling pipeline stages:

graph TD
    subgraph Engine [OpenCrawling Engine Core - oc-runtime]
        Orchestrator[JobOrchestrator - Scanning]
        IngestConsumer[IngestionConsumer - Extracting & Chunking]
        EmbedConsumer[EmbeddingConsumer - Vectorization]
        WriterConsumer[VectorStoreWriterConsumer - Indexing]
    end

    subgraph Telemetry [OpenTelemetry & Micrometer Core]
        TraceStore[TelemetryTraceStore - Correlated Spans]
        ErrorLog[InMemoryLogAppender - Exception Logs]
        Metrics[Micrometer Performance Counters]
    end

    subgraph AIOps [AIOps Diagnostic Layer]
        McpTools[ObservabilityMcpTools - System MCP Tools]
        DiagnosticService[AIOpsDiagnosticService - RCA Engine]
        RESTController[ObservabilityController - /api/observability]
    end

    subgraph AdminUI [Admin Console - oc-admin-ui]
        DiagnoseBtn[Diagnose with AI Button]
        RCAModal[AIOps Diagnostic Modal]
    end

    Orchestrator -->|Record Span & Errors| TraceStore
    IngestConsumer -->|Record Span & Errors| TraceStore
    EmbedConsumer -->|Record Span & Errors| TraceStore
    WriterConsumer -->|Record Span & Errors| TraceStore

    WriterConsumer -->|Log Event| ErrorLog

    TraceStore --> McpTools
    ErrorLog --> McpTools
    Metrics --> McpTools

    McpTools --> DiagnosticService
    DiagnosticService --> RESTController

    DiagnoseBtn -->|GET /api/observability/diagnose/id| RESTController
    RESTController -->|DiagnosticReport JSON| RCAModal

Loading

---

⏱️ Correlated Pipeline Stage Spans

Every ingestion job execution tracks 5 distinct correlated OpenTelemetry spans:

Stage	Component	Tracked Telemetry & Metrics
Scanning	JobOrchestrator / Repository Connector	Repository discovery duration, total document references found, permissions/ACL resolution.
Extracting	IngestionConsumer / Apache Tika	Text parsing latency, mime-type extraction status, raw document byte sizes.
Chunking	IngestionConsumer / TokenTextSplitter	Token splitting duration, total chunks generated, overlap boundaries.
Embedding	EmbeddingConsumer (oc-embedding-service)	AI model inference latency (Ollama / OpenAI), batch size, vector dimensions (384, 768, 1024).
Indexing	VectorStoreWriterConsumer	Database insertion latency (PgVector / Milvus), SQL batch commit duration, retry counts.

---

🛠️ System-Level MCP Tools (ObservabilityMcpTools)

System-level Model Context Protocol (MCP) tools allow LLMs and the Admin Copilot to query live telemetry without exposing underlying document content:

1. fetch_job_traces(jobId)

Retrieves correlated OpenTelemetry spans and stage timing breakdowns:

{
  "jobId": "1",
  "totalSpans": 5,
  "totalDurationMillis": 4400,
  "overallStatus": "COMPLETED",
  "stageDurationBreakdownMillis": {
    "Scanning": 450,
    "Extracting": 1200,
    "Chunking": 300,
    "Embedding": 1800,
    "Indexing": 650
  }
}

2. get_error_logs(jobId, timeframe)

Retrieves error stack traces and component exception logs for a specific job:

{
  "jobId": "1276647632",
  "errorCount": 1,
  "errors": [
    {
      "jobId": "1276647632",
      "timestamp": "2026-07-21T17:38:00Z",
      "level": "ERROR",
      "component": "VectorStoreWriterConsumer",
      "message": "Vector store batch insert operation timed out after exceeding 30000ms threshold during PostgreSQL pgvector embedding batch flush.",
      "stackTrace": "java.lang.RuntimeException: PostgreSQL connection timeout"
    }
  ]
}

3. query_throughput_metrics(connectorId)

Queries throughput rates, virtual thread concurrency usage, and latency metrics:

{
  "connectorId": "Alfresco_Repository",
  "averageThroughputDocsPerSec": 45.8,
  "p95LatencyMillis": 120.4,
  "activeVirtualThreads": 128.0
}

---

🤖 Automated Root Cause Analysis (RCA) Engine

The AIOpsDiagnosticService evaluates OTel traces, error logs, and metrics against a tuned diagnostic system prompt to synthesize an actionable DiagnosticReport:

public record DiagnosticReport(
    String jobId,
    String jobName,
    String timestamp,
    String status, // "HEALTHY", "WARNING", "FAILED"
    String summary,
    String rootCauseAnalysis,
    Map<String, Long> stageTimingMillis,
    List<String> bottleneckInsights,
    List<String> recommendedActions,
    List<TelemetryTraceStore.ErrorLogRecord> errorLogs
) {}

---

💻 REST API Endpoints

The ObservabilityController (@CrossOrigin(origins = "*")) provides standard REST endpoints at /api/observability:

• GET /api/observability/diagnose/{jobId}: Runs instant AI Root Cause Analysis for a specified job.
• GET /api/observability/traces/{jobId}: Fetches raw OTel trace spans for a job.
• GET /api/observability/errors/{jobId}?timeframe=all: Fetches failure logs and stack traces.
• GET /api/observability/metrics?connectorId={id}: Fetches throughput and latency metrics.

---

🎨 Admin Console Integration (oc-admin-ui)

In oc-admin-ui, every pipeline job row features a "Diagnose with AI" button (Sparkles icon). Clicking the button opens a modern AIOps slide-over modal displaying:

1. Status Banner: HEALTHY (green), WARNING (yellow), or FAILED (red).
2. AI Root Cause Analysis (RCA): Plain language explanation of failure causes or pipeline health.
3. Pipeline Stage Latency Breakdown: Interactive visual progress bars for each OTel span stage.
4. Identified Bottlenecks: Highlights anomalies (e.g. database batch timeouts or Tika text extraction latency).
5. Recommended Actions: Actionable resolution steps for system administrators.