Repository Connector Execution Modes

Repository Connector Execution Modes

OpenCrawling offers a flexible architecture designed to run repository connectors under two different operational modes: Dynamic UI-Driven (Long-Lived Service) or Standalone/Headless (One-Shot Task).

This guide explains both architectures and details how to configure them for local and production environments.

---

🔄 Execution Architectures At-A-Glance

Feature	Dynamic UI-Driven Mode	Standalone/Headless Mode
Container Lifespan	Long-running daemon	Ephemeral (runs one-shot crawl and exits)
Connector Resolution	Instantitated at runtime via API/UI inputs	Pre-selected on boot via environment variables
Multi-Tenancy	Single container runs multiple jobs in parallel	One container runs one specific job
Typical Target	Web applications, Admin console dashboards	Kubernetes CronJobs, Airflow, CI/CD pipelines

---

🖥️ 1. Dynamic UI-Driven Mode (Default Production Deployment)

In this mode, you deploy a single, long-running instance of the OpenCrawling backend (oc-mcp-server or oc-runtime container).

All connector classes (e.g. FileSystemRepositoryConnector, AlfrescoRepositoryConnector, etc.) are loaded on the classpath. The backend acts as a job manager.

How It Works:

1. Dynamic Registration: Through the Admin UI (or REST API), you register repository and output connectors. Configurations (like URLs, credentials, paths) are saved as JSON metadata.
2. Runtime Instantiation: When you hit "Start" on a Job, the orchestrator retrieves the connector class name (e.g., org.opencrawling.alfresco.AlfrescoRepositoryConnector), dynamically instantiates it with the saved configuration map, and triggers the crawl thread.
3. Isolation: Because instantiation happens dynamically inside virtual threads, you can run multiple crawl jobs simultaneously targeting completely different repositories (e.g., crawling local directories, Alfresco nodes, and SharePoint folders) on the same running container without needing restarts.

---

⚙️ 2. Standalone / Headless Mode (Automated / One-Shot)

In this mode, you run a container configured to automatically boot, crawl a preconfigured path, write the claims/documents, and exit. This is ideal for cron schedulers or scheduled job workflows.

You control this behavior using Spring Boot properties.

Property Keys

• spring.opencrawling.crawl-on-startup: Enable/disable automated crawl on boot.
• spring.opencrawling.repository-connector-type: Choose which repository connector class to bind (file or alfresco).
• spring.opencrawling.scan-path: The starting folder path or node ID to crawl.
• spring.opencrawling.transformation-connector: The target vector/embedding connector key (e.g., Ollama local).

Docker Compose Example

To configure a headless container for crawling an Alfresco Content Services server:

services:
  oc-crawler-alfresco:
    build:
      context: .
      dockerfile: docker/Dockerfile.crawler
    container_name: oc-crawler-alfresco-job
    environment:
      - SPRING_KAFKA_BOOTSTRAP_SERVERS=kafka:9094
      - SPRING_OPENCRAWLING_CRAWL_ON_STARTUP=true
      - SPRING_OPENCRAWLING_REPOSITORY_CONNECTOR_TYPE=alfresco
      - SPRING_OPENCRAWLING_SCAN_PATH=-root-
      - SPRING_OPENCRAWLING_TRANSFORMATION_CONNECTOR=Ollama local
    volumes:
      - ./oc-runtime/data:/data
    networks:
      - opencrawling-network

On container startup:

1. The runtime boots up and detects SPRING_OPENCRAWLING_CRAWL_ON_STARTUP=true.
2. It filters the loaded beans to find a connector matching alfresco (instantiating AlfrescoRepositoryConnector).
3. It crawls from -root- recursively, sending claims to the Kafka queue.
4. Once completed, the container exits successfully.

---

🧩 Connector Class Map reference

If you are registering connectors via the REST API or adding them to the connectors database (connectors.json), reference these fully qualified class names:

Connector	Class Name
Local File System	org.opencrawling.crawler.connectors.filesystem.FileConnector
Alfresco Content Services	org.opencrawling.alfresco.AlfrescoRepositoryConnector
Apache Iceberg	org.opencrawling.iceberg.IcebergRepositoryConnector
PGVector Output	org.opencrawling.vector.VectorOutputConnector