Skip to content

User Guide

Jump to bottom
Piergiorgio Lucidi edited this page Jul 24, 2026 · 3 revisions

Admin UI & Configuration User Guide

The OpenCrawling Admin UI (oc-admin-ui) provides an intuitive, web-based management dashboard to configure repository connectors, manage ingestion job pipelines, monitor real-time activity logs, tune embedding models, and manage Apache Ozone Claim-Check Storage & Garbage Collection parameters.

🎬 Interactive Screencast Walkthrough

Admin UI Screencast Walkthrough

📊 1. Real-Time Telemetry Dashboard

The Telemetry Dashboard gives administrators an instant high-level overview of live ingestion statistics, active virtual threads, Kafka queue metrics, vector database indexing speeds, and system memory consumption.

Telemetry Dashboard

Key Metrics Monitored:

  • Active Crawling Jobs: Total background workers executing scan and extraction tasks.
  • Ingestion Throughput: Real-time document parsing speed (documents/second).
  • Kafka Queue Load: Monitored message count across opencrawling-ingestion, opencrawling-chunks, and opencrawling-embedded topics.
  • Vector Index Count: Total high-dimensional vector embeddings stored across target vector stores.

📋 2. Job Pipeline Scheduler & AIOps Diagnostics

The Job Pipeline Scheduler enables administrators to schedule, monitor, pause, resume, and inspect ingestion crawl jobs across enterprise data repositories.

Pipeline Job Management

Features & Actions:

  • Create & Edit Jobs: Associate repository source connectors with output vector database connectors.
  • Execution Controls: Trigger immediate manual crawls, pause running tasks, or stop execution.
  • Per-Job Narrativization: Inspect and configure Mustache transformation templates attached to specific ingestion pipelines.

Pipeline Job Narrativization

  • AIOps One-Click Diagnosis: Click "Diagnose with AI" on any failed or warning job to launch automated OpenTelemetry (OTel) trace analysis, timeline inspection, and root cause identification.

AIOps Diagnosis with AI

📁 3. Connector Registry Configuration

Manage endpoints, credentials, and execution properties for all repository sources, output vector stores, and authority connectors.

Connector Configuration

Supported Connectors:

  • Repository Connectors: Filesystem, Alfresco Content Services, Apache Iceberg tables.
  • Output Connectors: PostgreSQL (pgvector), Milvus, OpenSearch 2.x, OpenSearch 3.x.
  • Transformation Connectors: Mustache Auto-Narrativization Copilot, Token Text Splitter.

⚙️ 4. Ingestion, Embedding & Storage Settings

The Settings Panel allows administrators to visually configure embedding models, vector dimensions, chunk splitting algorithms, Auto-Narrativization Copilot, and Apache Ozone Claim-Check Storage & Client Strategies.

Ingestion & Embedding Settings

Key Settings Options:

A. AI Core Provider & Embedding Models

  • AI Core Provider: Select local Ollama or cloud OpenAI / HuggingFace providers.
  • Host URL & Model: Configure connection endpoints (e.g. http://127.0.0.1:11434) and model selection (mxbai-embed-large, nomic-embed-text, all-minilm, bge-large-en).
  • Vector Dimensions: Automatic mapping to target pgvector table index dimensions (384d, 768d, 1024d).

B. Apache Ozone & Claim-Check Storage Configuration

Configure binary payload offloading and client protocol strategy directly from the UI:

Apache Ozone Client Settings

  • Claim Check Store Provider: Switch between Apache Ozone Distributed Storage (Recommended for scaling heavy binary files) and Local Shared Disk.
  • Ozone Client Protocol Strategy:
    • Native Ozone Client (ofs / o3fs): High-performance gRPC/RPC binary transport directly to DataNodes & Ozone Manager (om port 9862). Bypasses HTTP translation overhead for maximum throughput.
    • S3 Gateway Client (s3g): Standard AWS S3 SDK integration hitting Ozone's S3 Gateway endpoint (http://localhost:9878).
  • Connection Parameters: Configure Ozone Manager Host, OM Port, S3 Endpoint URL, Target Volume, and Target Bucket.

C. Lifecycle & Explicit Garbage Collection (GC)

  • Post-ACK Explicit Deletion: Immediately purges offloaded claim-check objects upon receiving a post-ingestion ACK from Kafka chunk consumers.
  • Background Garbage Collector: Runs an automated background sweep (ClaimCheckGarbageCollector) based on a cron schedule (gc-cron) to purge orphaned claim-check payloads older than ttl-hours (default 24h).

D. Auto-Narrativization Copilot Settings

Configure AI schema transformation and Mustache narrative template generation settings:

Settings Auto-Narrativization

🪵 5. Real-Time Activity Logs & Telemetry

Inspect live streaming application logs, trace spans, and consumer lag across OpenCrawling microservices.

Real-Time Activity Logs

Diagnostic Features:

  • Log Level Filter: Filter log output by INFO, WARN, ERROR, or DEBUG.
  • Kafka Consumer Lag: Track lag offset metrics per topic to identify pipeline bottlenecks.
  • Virtual Thread Stack Traces: View full exception stack traces for instant troubleshooting.

🗺️ Documentation

🧪 Testing

🤝 Community

Clone this wiki locally