Configuration Guide

Configuration Guide

OpenCrawling is configured using standard Spring Boot property profiles. The main configuration parameters specify repository connections, message brokers, caching, and target vector databases.

---

🛠️ Dev Profile Properties (application-dev.yml)

The runtime behavior is controlled under oc-runtime via YAML configurations. Here is a typical deployment setup:

spring:
  profiles:
    active: dev
  
  datasource:
    url: jdbc:postgresql://localhost:5432/opencrawling
    username: opencrawling
    password: croom-secure-password
    driver-class-name: org.postgresql.Driver
    
  kafka:
    bootstrap-servers: localhost:9092
    consumer:
      group-id: opencrawling-group
      auto-offset-reset: earliest
      
  redis:
    host: localhost
    port: 6379

# OpenCrawling Custom Settings
spring:
  opencrawling:
    # Trigger a scan path immediately on application boot
    crawl-on-startup: false
    scan-path: /tmp/opencrawling-mount
    
    # Configure default transformation embedding models
    embedding:
      engine: ollama # Options: ollama, openai
      model-name: mxbai-embed-large
      dimensions: 1024
      url: http://localhost:11434

---

📁 Local Filesystem Claim Check Store Configuration

For lightweight or single-node deployments where object storage (Apache Ozone or AWS S3) is not required, OpenCrawling provides a high-performance Local File Claim Check Store (LocalFileClaimCheckStore).

Raw document payloads are written to local disk space, and file:// scheme URIs (e.g., file:///data/claims/doc-uuid.txt) are circulated through Kafka topics.

1. Spring Boot Properties (application.yml)

spring:
  opencrawling:
    claim-check:
      store: local                      # Set store type to "local"
      cleanup-on-consume: true          # Delete claim file after Kafka ingestion
      local:
        dir: target/claims             # Claims directory ("target/claims" or "/data/claims")

2. Environment Variables

Environment Variable	Description	Default Value
SPRING_OPENCRAWLING_CLAIM_CHECK_STORE	Set active store type	local
SPRING_OPENCRAWLING_CLAIM_CHECK_LOCAL_DIR	Local disk folder for claim payloads	target/claims (Docker: /data/claims)
SPRING_OPENCRAWLING_CLAIM_CHECK_CLEANUP_ON_CONSUME	Delete file payload after processing	true

---

🏭 Production PostgreSQL Configuration (application-prod.yml)

For production environments, activate the prod profile (SPRING_PROFILES_ACTIVE=prod). OpenCrawling connects to PostgreSQL (with the pgvector extension installed) for Spring Batch metadata, state persistence, and vector index storage.

spring:
  config:
    activate:
      on-profile: prod

  # 1. Main Datasource (Spring Batch & Job Metadata)
  datasource:
    url: ${SPRING_DATASOURCE_URL:jdbc:postgresql://postgres-host:5432/opencrawling}
    username: ${SPRING_DATASOURCE_USERNAME:opencrawling}
    password: ${SPRING_DATASOURCE_PASSWORD:opencrawling_password}
    driver-class-name: org.postgresql.Driver
    hikari:
      maximum-pool-size: ${SPRING_DATASOURCE_HIKARI_MAXIMUM_POOL_SIZE:20}
      minimum-idle: ${SPRING_DATASOURCE_HIKARI_MINIMUM_IDLE:5}
      idle-timeout: 300000
      max-lifetime: 1800000

  jpa:
    hibernate:
      ddl-auto: ${SPRING_JPA_HIBERNATE_DDL_AUTO:update}
    show-sql: false

  # 2. PgVector Store (Vector Index & Document Security ACLs)
  ai:
    vectorstore:
      pgvector:
        url: ${SPRING_AI_VECTORSTORE_PGVECTOR_URL:jdbc:postgresql://postgres-host:5432/opencrawling}
        username: ${SPRING_AI_VECTORSTORE_PGVECTOR_USERNAME:opencrawling}
        password: ${SPRING_AI_VECTORSTORE_PGVECTOR_PASSWORD:opencrawling_password}
        driver-class-name: org.postgresql.Driver
        dimensions: ${SPRING_AI_VECTORSTORE_PGVECTOR_DIMENSIONS:1024}
        initialize-schema: true

Required PostgreSQL Extension

Ensure the vector extension is enabled in your production PostgreSQL database:

CREATE EXTENSION IF NOT EXISTS vector;

---

🔌 Core Connector Types

OpenCrawling uses a Service Provider Interface (SPI) structure. To run a job, you map a Repository Ingestion Source to a Vector Output Destination.

1. Repository Connectors (Sources)

• Filesystem Connector: Scans local or mounted directories recursively.
• S3 Connector: Pulls document objects from AWS S3 or MinIO buckets.
• SharePoint Connector: Utilizes Microsoft Graph delta queries to scan directories incrementally while extracting Active Directory SIDs.

2. Output Connectors (Destinations)

• PgVector Store: Connects to PostgreSQL using pgvector extension.
• Elasticsearch / OpenSearch: Leverages KNN indexing vector engines.
• Qdrant / Milvus: Connects to standard cloud-native vector indexes.

---

📈 Horizon Scaling of Consumers

For high ingestion jobs, adjust Kafka topic partition numbers:

• Configure the opencrawling-chunks and opencrawling-embedded topics with 4 to 8 partitions.
• Run multiple instances of oc-embedding-service in the same consumer group. Kafka will distribute chunk embedding calculations evenly across instances.