Skip to content

VM and Bare Metal Distribution Installation

Piergiorgio Lucidi edited this page Oct 27, 2026 · 1 revision

📦 VM & Bare-Metal Distribution ZIP Installation Guide

While OpenCrawling provides official container images (ghcr.io/opencrawling/*) and Docker Compose / Kubernetes topologies, many regulated enterprise environments, government agencies, financial institutions, and air-gapped data centers operate on dedicated Virtual Machines (VMs) or bare-metal servers where container runtimes (Docker, Podman, containerd, or Kubernetes pods) are restricted or prohibited by corporate security policies.

To support native, container-free deployments across Linux (RHEL, Ubuntu, Debian, SLES), Windows Server, and macOS, OpenCrawling publishes self-contained Distribution ZIP Packages automatically on every GitHub Release via the Publish VM Distribution ZIP Packages (dist-release-publish.yml) workflow.


🏗️ Deployment Topologies on VMs

Every OpenCrawling Distribution ZIP supports both All-in-One Single-VM execution and Horizontally Scaled Multi-VM Decoupled architectures using the exact same pre-extracted binaries and role-specific startup wrappers in bin/.

flowchart TB
    subgraph OptionA["Topology A: All-in-One Single VM"]
        direction TB
        AIO["bin/start-runtime.sh (or .bat)<br/>• REST Control Plane & Swagger UI (:8080)<br/>• Embedded React 19 Admin UI (:8080)<br/>• Crawler + Tika ForkParser + Embedding + Writer<br/>• Secure MCP Server (/mcp)<br/>• gRPC Transport (:9095) & gRPC Push Gateway (:9096)"]
    end

    subgraph OptionB["Topology B: Decoupled Multi-VM Fleet"]
        direction LR
        VM1["Crawler VM<br/>bin/start-crawler.sh"]
        VM2["Ingestion VM (Apache Tika 4)<br/>bin/start-ingestion-consumer.sh"]
        VM3["Embedding VM (Spring AI)<br/>bin/start-embedding-service.sh"]
        VM4["Vector Writer VM<br/>bin/start-writer-consumer.sh"]
        VM5["Control Plane & Secure MCP VM<br/>bin/start-mcp-server.sh"]
        VM6["Reverse Proxy / Web VM<br/>Nginx + admin-ui/dist"]

        VM1 -->|Kafka + Claim Check| VM2
        VM2 -->|Kafka| VM3
        VM3 -->|Kafka| VM4
        VM6 -->|/api/* proxy| VM5
    end

    subgraph Backing["Enterprise Backing Infrastructure (Managed Services or Dedicated VMs)"]
        PG[("Vector Store & State DB<br/>(PostgreSQL + pgvector / Solr / OpenSearch / Qdrant / Milvus / Vespa)")]
        KF[["Apache Kafka<br/>(Event Streaming)"]]
        RD[("Redis<br/>(Distributed State & Cache)")]
        OZ[("Claim Check Store<br/>(Local NAS / NFS or Apache Ozone)")]
    end

    OptionA --> Backing
    OptionB --> Backing
Loading

📦 Published Distribution ZIP Packages

Each GitHub Release attaches 5 ZIP distributions plus cryptographic SHA-256 checksums (SHA256SUMS.txt and individual .sha256 files):

Distribution Archive Approx. Size Target Audience / VM Role Included Artifacts
opencrawling-platform-dist-<version>.zip ~745 MB Complete Platform Bundle (All-in-One VM or master Golden Image for multi-VM provisioning) Extracted runtime/app.jar + runtime/lib/ (for Apache Tika ForkParser), embedding-service/oc-embedding-service.jar, admin-ui/dist/ + nginx.conf, cli/oc-cli.jar, mcf-migrator/oc-mcf-migrator.jar, all .sh/.bat scripts, config/, and systemd/ service units
opencrawling-runtime-dist-<version>.zip ~459 MB Runtime / Worker / Control Plane VMs Extracted app.jar + lib/ directory, bin/start-runtime.sh (.bat), decoupled role scripts (start-crawler.sh, start-ingestion-consumer.sh, start-writer-consumer.sh, start-mcp-server.sh), config/, and systemd/ units
opencrawling-embedding-service-dist-<version>.zip ~192 MB Stateless Embedding Worker VMs oc-embedding-service.jar, bin/start-embedding-service.sh (.bat), config/opencrawling.env.example, and systemd/opencrawling-embedding-service.service
opencrawling-admin-ui-dist-<version>.zip ~275 KB Frontend Web / Reverse Proxy VMs (Nginx, Apache HTTP Server, IIS) Pre-built React 19 + Vite static bundle (dist/index.html, dist/assets/*) and production nginx.conf
opencrawling-cli-dist-<version>.zip ~91 MB Bastion Hosts, Admin Workstations & CI/CD Runners Standalone oc-cli.jar plus bin/oc (Linux/macOS) and bin/oc.bat (Windows) wrappers
SHA256SUMS.txt & *.zip.sha256 < 1 KB Security & Compliance Verification SHA-256 digests for all 5 ZIP archives

📂 Directory Structure (opencrawling-platform-dist-<version>.zip)

When you extract opencrawling-platform-dist-<version>.zip, the directory layout is structured for standard /opt/opencrawling (Linux/macOS) or C:\opencrawling (Windows Server) installations:

opencrawling-platform-dist-<version>/
├── INSTALL.md                                  # Offline quick-start & operational manual
├── LICENSE                                     # Apache License 2.0
├── bin/
│   ├── start-runtime.sh                        # All-in-One Runtime launcher (Linux/macOS)
│   ├── start-runtime.bat                       # All-in-One Runtime launcher (Windows Server)
│   ├── start-crawler.sh                        # Decoupled Role: Crawler Node only
│   ├── start-ingestion-consumer.sh             # Decoupled Role: Apache Tika Ingestion Consumer only
│   ├── start-embedding-service.sh              # Decoupled Role: Stateless Embedding Microservice (Linux/macOS)
│   ├── start-embedding-service.bat             # Decoupled Role: Stateless Embedding Microservice (Windows)
│   ├── start-writer-consumer.sh                # Decoupled Role: Vector Store Writer Consumer only
│   ├── start-mcp-server.sh                     # Decoupled Role: Secure MCP Server & REST Control Plane
│   ├── oc                                      # OpenCrawling CLI wrapper (Linux/macOS)
│   └── oc.bat                                  # OpenCrawling CLI wrapper (Windows)
├── config/
│   ├── opencrawling.env.example                # Central environment variable template
│   └── application-vm.yml                      # Reference Spring Boot YAML configuration
├── runtime/
│   ├── app.jar                                 # Extracted Spring Boot application entry point
│   └── lib/                                    # Extracted dependency JARs (required for Apache Tika ForkParser)
├── embedding-service/
│   └── oc-embedding-service.jar                # Standalone Spring AI Embedding Microservice
├── admin-ui/
│   ├── dist/                                   # Pre-compiled React 19 SPA static bundle
│   └── nginx.conf                              # Reverse proxy & SPA fallback configuration for Nginx
├── cli/
│   └── oc-cli.jar                              # Standalone OpenCrawling CLI executable JAR
├── mcf-migrator/
│   └── oc-mcf-migrator.jar                     # Apache ManifoldCF migration tool
├── systemd/
│   ├── opencrawling-runtime.service            # Linux systemd unit for Runtime / Control Plane
│   └── opencrawling-embedding-service.service  # Linux systemd unit for Embedding Microservice
├── data/
│   ├── claims/                                 # Default local Claim-Check payload directory
│   └── reports/                                # Default Lakehouse Audit Report (Parquet/CSV) directory
└── logs/                                       # Application log output directory

Why runtime/app.jar and runtime/lib/ Are Pre-Extracted

During packaging, OpenCrawling runs Spring Boot's layered JAR extractor:

java -Djarmode=tools -jar oc-runtime-<version>.jar extract --destination extracted-runtime

This separates runtime/app.jar from its third-party libraries in runtime/lib/. This is critical on bare-metal VMs because Apache Tika 4's out-of-process PipesForkParser spawns isolated child JVMs to parse untrusted PDFs, Office files, and archives safely; those child JVMs require a flat filesystem directory of JARs (-Dtika.extras.dir=$BASE_DIR/runtime/lib, automatically configured by bin/start-runtime.sh and bin/start-runtime.bat) rather than nested Spring Boot fat-JAR entries.


⚙️ Automated GitHub Release Workflow & Local Packaging

1. Automated GitHub Actions Workflow (dist-release-publish.yml)

The workflow .github/workflows/dist-release-publish.yml is triggered automatically on:

  • GitHub Release published (on: release: types: [published])
  • Version tag push (on: push: tags: ['v*'])
  • Manual dispatch (on: workflow_dispatch with input release_version, e.g., 1.0.4)

Pipeline Stages:

  1. build-vm-distributions (ubuntu-24.04, Eclipse Temurin JDK 25):
    • Resolves the release version and tag name.
    • Synchronizes the version banner in oc-cli.
    • Executes ./scripts/package-dist.sh "<version>" to compile oc-runtime, oc-embedding-service, oc-admin-ui, oc-cli, and oc-mcf-migrator, extract the Spring Boot runtime layers, generate startup scripts and systemd units, and assemble the 5 .zip archives plus SHA256SUMS.txt.
    • Runs an automated smoke test on opencrawling-platform-dist-<version>.zip verifying the presence of runtime/app.jar, runtime/lib/, embedding-service/oc-embedding-service.jar, admin-ui/dist/index.html, cli/oc-cli.jar, and executing ./bin/oc --version.
    • Uploads all .zip and .sha256 files as workflow artifacts (opencrawling-vm-dist-packages).
  2. publish-dist-release-assets:
    • Downloads the verified distribution artifacts and attaches all 5 .zip archives, individual .sha256 files, and SHA256SUMS.txt directly to the GitHub Release page via softprops/action-gh-release.

2. Building the Distribution ZIPs Locally

If you are building custom connectors or air-gapped internal releases from source, you can generate all 5 ZIP archives on any machine with JDK 25 and Apache Maven 3.9+:

# Full Maven build + ZIP packaging (outputs to ./dist/)
./scripts/package-dist.sh 1.0.4

# Fast packaging if Maven targets are already compiled in target/ directories
SKIP_BUILD=true ./scripts/package-dist.sh 1.0.4

🐧 Step-by-Step Installation on Linux VMs

1. Prerequisites

  • Operating System: Linux x86_64 or ARM64 (RHEL 9+, Ubuntu 22.04/24.04 LTS, Debian 12+, SLES 15+, Amazon Linux 2023).
  • Java Runtime: Java 25 (JDK or JRE 25) installed and available on PATH or specified via JAVA_HOME:
    java -version
    # openjdk version "25" 2025-09-16 LTS
  • Network Connectivity to Backing Services:
    • PostgreSQL 17+ with pgvector (default port 5432) or your chosen Vector Database.
    • Apache Kafka (default port 9092) — required when running asynchronous pipelines or decoupled consumers.
    • Redis 7+ (default port 6379) — distributed crawl state, throttling gates, and caching.
    • Ollama (11434) or OpenAI-compatible endpoint — if generating vector embeddings or using AIOps / Auto-Narrativization Copilot.

2. Download & Verify SHA-256 Checksum

export OC_VERSION="1.0.4"

# Download the full platform distribution and checksum manifest
curl -LO "https://github.com/opencrawling/opencrawling/releases/download/v${OC_VERSION}/opencrawling-platform-dist-${OC_VERSION}.zip"
curl -LO "https://github.com/opencrawling/opencrawling/releases/download/v${OC_VERSION}/SHA256SUMS.txt"

# Verify archive integrity
sha256sum --ignore-missing -c SHA256SUMS.txt

3. Extract to /opt/opencrawling & Create Dedicated Service User

# Create dedicated non-root system user
sudo groupadd --system opencrawling || true
sudo useradd --system --gid opencrawling --home-dir /opt/opencrawling --shell /sbin/nologin opencrawling || true

# Extract archive and set ownership
sudo mkdir -p /opt/opencrawling
sudo unzip -q "opencrawling-platform-dist-${OC_VERSION}.zip" -d /opt/opencrawling
sudo ln -sfn "/opt/opencrawling/opencrawling-platform-dist-${OC_VERSION}" /opt/opencrawling/current
sudo chown -R opencrawling:opencrawling /opt/opencrawling

4. Configure Environment Variables (config/opencrawling.env)

Copy the template environment file and adjust endpoints and credentials for your VM infrastructure:

cd /opt/opencrawling/current
sudo -u opencrawling cp config/opencrawling.env.example config/opencrawling.env
sudo -u opencrawling vi config/opencrawling.env

Example config/opencrawling.env:

# Java 25 Runtime Options
JAVA_OPTS="-Xms1g -Xmx4g --enable-preview --enable-native-access=ALL-UNNAMED"

# PostgreSQL / pgvector Connection
SPRING_AI_VECTORSTORE_PGVECTOR_URL=jdbc:postgresql://pg-vm.internal:5432/opencrawling
SPRING_AI_VECTORSTORE_PGVECTOR_USERNAME=opencrawling
SPRING_AI_VECTORSTORE_PGVECTOR_PASSWORD=strong_production_password

# Apache Kafka Bootstrap Servers
SPRING_KAFKA_BOOTSTRAP_SERVERS=kafka-vm-1.internal:9092,kafka-vm-2.internal:9092

# Redis Cache / State Store
SPRING_DATA_REDIS_HOST=redis-vm.internal
SPRING_DATA_REDIS_PORT=6379
SPRING_DATA_REDIS_PASSWORD=strong_redis_password

# Ollama / OpenAI Embedding Provider
SPRING_AI_OLLAMA_BASE_URL=http://ollama-gpu-vm.internal:11434
SPRING_AI_OPENAI_API_KEY=your-api-key

# Claim-Check & Lakehouse Audit Report Directories (use shared NFS/NAS mount in multi-VM setups)
SPRING_OPENCRAWLING_CLAIM_CHECK_LOCAL_DIR=/mnt/shared-nas/opencrawling/claims
SPRING_OPENCRAWLING_REPORTING_LOCAL_DIR=/mnt/shared-nas/opencrawling/reports

# Apache Tika Out-of-Process ForkParser Settings
OPENCRAWLING_TIKA_FORK_ENABLED=true
OPENCRAWLING_TIKA_TIMEOUT_MS=30000
OPENCRAWLING_TIKA_JVM_ARGS=-Xmx1g

# Server Ports
SERVER_PORT=8080
OPENCRAWLING_GRPC_PORT=9095
OPENCRAWLING_GRPC_PUSH_ENABLED=true
OPENCRAWLING_GRPC_PUSH_PORT=9096

Important

Multi-VM Claim Check Storage: When running decoupled Crawler and Ingestion Consumer roles across separate VMs, ensure both VMs can access the same Claim Check storage — either by pointing SPRING_OPENCRAWLING_CLAIM_CHECK_LOCAL_DIR to a shared NFS/SMB mount or by configuring the Apache Ozone Claim Check Store (SPRING_OPENCRAWLING_CLAIM_CHECK_STORE_TYPE=ozone).


🚀 Running OpenCrawling on VMs

Option A: All-in-One Mode (Single VM)

Starts the entire platform in a single JVM process (REST API, Embedded React Admin UI on port 8080, Swagger UI, Crawler, Tika Ingestion, Vector Writer, Secure MCP Server, and gRPC servers on 9095/9096):

cd /opt/opencrawling/current
./bin/start-runtime.sh

Option B: Decoupled Microservice Roles (Multi-VM Fleet)

On dedicated worker VMs, use the specialized launcher scripts in bin/ (available in both opencrawling-platform-dist and opencrawling-runtime-dist / opencrawling-embedding-service-dist):

VM Role Command Description
1. Crawler VM ./bin/start-crawler.sh Runs repository connectors in non-web mode (SPRING_MAIN_WEB_APPLICATION_TYPE=none), offloads raw binaries to Claim Check storage, and publishes crawl events to Kafka.
2. Ingestion Consumer VM ./bin/start-ingestion-consumer.sh Consumes raw document claims from Kafka, extracts text/metadata via Apache Tika 4 ForkParser (runtime/lib), applies transformation & narrativization pipelines, and publishes chunks to Kafka.
3. Embedding Service VM ./bin/start-embedding-service.sh Runs oc-embedding-service.jar (default port 8082), consuming text chunks from Kafka and computing dense vector embeddings via Spring AI (Ollama / OpenAI).
4. Vector Writer VM ./bin/start-writer-consumer.sh Consumes embedded document chunks and tombstones from Kafka and writes them into the target Output Connector (PgVector, Solr, OpenSearch, Vespa, Qdrant, Milvus, Luxir, Hyland OpenArch, or Apache Ozone).
5. Control Plane & MCP VM ./bin/start-mcp-server.sh Exposes the REST API, Embedded Admin UI (:8080), OpenAPI Swagger UI, and the Zero-Trust Secure Model Context Protocol (MCP) Server (/mcp) without running background crawl workers.

🛡️ Production Daemon Management with systemd (Linux)

Ready-to-use systemd unit files are included in the systemd/ directory of the distribution archives.

1. Install and Enable opencrawling-runtime.service

If you created the /opt/opencrawling/current symlink, update WorkingDirectory, EnvironmentFile, and ExecStart (or extract directly into /opt/opencrawling):

# Copy unit files to /etc/systemd/system/
sudo cp /opt/opencrawling/current/systemd/opencrawling-runtime.service /etc/systemd/system/
sudo cp /opt/opencrawling/current/systemd/opencrawling-embedding-service.service /etc/systemd/system/

# Reload systemd daemon and enable service on boot
sudo systemctl daemon-reload
sudo systemctl enable --now opencrawling-runtime.service

# Check service status and follow logs
sudo systemctl status opencrawling-runtime.service
sudo journalctl -u opencrawling-runtime.service -f

Tip

Running a Decoupled Role via systemd: To dedicate a VM to a specific worker role (for example, an Ingestion Consumer VM), simply change ExecStart in /etc/systemd/system/opencrawling-runtime.service from /opt/opencrawling/bin/start-runtime.sh to /opt/opencrawling/bin/start-ingestion-consumer.sh.


🌐 Hosting the Standalone Admin UI on Nginx (opencrawling-admin-ui-dist)

Note that oc-runtime already embeds the Admin UI directly on http://<vm-host>:8080. However, if your enterprise architecture requires serving the frontend SPA from a dedicated DMZ web server or reverse proxy tier using opencrawling-admin-ui-dist-<version>.zip:

# 1. Extract the Admin UI distribution
unzip opencrawling-admin-ui-dist-1.0.4.zip

# 2. Copy static assets to Nginx web root
sudo mkdir -p /usr/share/nginx/html
sudo cp -R opencrawling-admin-ui-dist-1.0.4/dist/* /usr/share/nginx/html/

# 3. Install the provided Nginx configuration (and update proxy_pass upstream if needed)
sudo cp opencrawling-admin-ui-dist-1.0.4/nginx.conf /etc/nginx/conf.d/opencrawling.conf
sudo nginx -t && sudo systemctl reload nginx

In /etc/nginx/conf.d/opencrawling.conf, point the /api/ proxy_pass directive to the internal IP/hostname of your OpenCrawling Control Plane VM (http://<control-plane-vm>:8080/api/).


🪟 Step-by-Step Installation on Windows Server VMs

  1. Install Java 25: Install Eclipse Temurin JDK 25 (.msi installer) and ensure JAVA_HOME is set or java.exe is on the system PATH.
  2. Extract the Distribution Archive: Extract opencrawling-platform-dist-<version>.zip to C:\opencrawling.
  3. Configure Environment Variables: Set environment variables in PowerShell or System Properties (or customize bin\start-runtime.bat):
    cd C:\opencrawling\opencrawling-platform-dist-1.0.4
    
    $env:JAVA_OPTS = "-Xms1g -Xmx4g"
    $env:SPRING_AI_VECTORSTORE_PGVECTOR_URL = "jdbc:postgresql://127.0.0.1:5432/opencrawling"
    $env:SPRING_KAFKA_BOOTSTRAP_SERVERS = "127.0.0.1:9092"
    $env:SPRING_DATA_REDIS_HOST = "127.0.0.1"
  4. Launch Runtime, Embedding Service, or CLI:
    # Start All-in-One Runtime
    .\bin\start-runtime.bat
    
    # Start Stateless Embedding Microservice (in a separate terminal or VM)
    .\bin\start-embedding-service.bat
    
    # Run the OpenCrawling CLI
    .\bin\oc.bat --version
    .\bin\oc.bat system status

✅ Post-Installation Verification

Once the runtime service is running on your VM, verify health and connectivity using the bundled oc CLI or curl:

# 1. Verify CLI binary
/opt/opencrawling/current/bin/oc --version

# 2. Check Spring Boot Actuator Health Endpoint
curl -s http://localhost:8080/actuator/health | jq .

# 3. Query Platform Status via CLI
/opt/opencrawling/current/bin/oc system status --url http://localhost:8080
  • Admin UI Dashboard: http://<vm-hostname>:8080/
  • Interactive OpenAPI 3.1 / Swagger UI: http://<vm-hostname>:8080/swagger-ui.html
  • Secure MCP Server Endpoint: http://<vm-hostname>:8080/mcp
  • gRPC Internal Transport & Push Gateway: <vm-hostname>:9095 and <vm-hostname>:9096

🗺️ Documentation


🧪 Testing


🤝 Community

Clone this wiki locally