Repository navigation
VM and Bare Metal Distribution Installation
While OpenCrawling provides official container images (ghcr.io/opencrawling/*) and Docker Compose / Kubernetes topologies, many regulated enterprise environments, government agencies, financial institutions, and air-gapped data centers operate on dedicated Virtual Machines (VMs) or bare-metal servers where container runtimes (Docker, Podman, containerd, or Kubernetes pods) are restricted or prohibited by corporate security policies.
To support native, container-free deployments across Linux (RHEL, Ubuntu, Debian, SLES), Windows Server, and macOS, OpenCrawling publishes self-contained Distribution ZIP Packages automatically on every GitHub Release via the Publish VM Distribution ZIP Packages (dist-release-publish.yml) workflow.
Every OpenCrawling Distribution ZIP supports both All-in-One Single-VM execution and Horizontally Scaled Multi-VM Decoupled architectures using the exact same pre-extracted binaries and role-specific startup wrappers in bin/.
flowchart TB
subgraph OptionA["Topology A: All-in-One Single VM"]
direction TB
AIO["bin/start-runtime.sh (or .bat)<br/>• REST Control Plane & Swagger UI (:8080)<br/>• Embedded React 19 Admin UI (:8080)<br/>• Crawler + Tika ForkParser + Embedding + Writer<br/>• Secure MCP Server (/mcp)<br/>• gRPC Transport (:9095) & gRPC Push Gateway (:9096)"]
end
subgraph OptionB["Topology B: Decoupled Multi-VM Fleet"]
direction LR
VM1["Crawler VM<br/>bin/start-crawler.sh"]
VM2["Ingestion VM (Apache Tika 4)<br/>bin/start-ingestion-consumer.sh"]
VM3["Embedding VM (Spring AI)<br/>bin/start-embedding-service.sh"]
VM4["Vector Writer VM<br/>bin/start-writer-consumer.sh"]
VM5["Control Plane & Secure MCP VM<br/>bin/start-mcp-server.sh"]
VM6["Reverse Proxy / Web VM<br/>Nginx + admin-ui/dist"]
VM1 -->|Kafka + Claim Check| VM2
VM2 -->|Kafka| VM3
VM3 -->|Kafka| VM4
VM6 -->|/api/* proxy| VM5
end
subgraph Backing["Enterprise Backing Infrastructure (Managed Services or Dedicated VMs)"]
PG[("Vector Store & State DB<br/>(PostgreSQL + pgvector / Solr / OpenSearch / Qdrant / Milvus / Vespa)")]
KF[["Apache Kafka<br/>(Event Streaming)"]]
RD[("Redis<br/>(Distributed State & Cache)")]
OZ[("Claim Check Store<br/>(Local NAS / NFS or Apache Ozone)")]
end
OptionA --> Backing
OptionB --> Backing
Each GitHub Release attaches 5 ZIP distributions plus cryptographic SHA-256 checksums (SHA256SUMS.txt and individual .sha256 files):
| Distribution Archive | Approx. Size | Target Audience / VM Role | Included Artifacts |
|---|---|---|---|
opencrawling-platform-dist-<version>.zip |
~745 MB | Complete Platform Bundle (All-in-One VM or master Golden Image for multi-VM provisioning) | Extracted runtime/app.jar + runtime/lib/ (for Apache Tika ForkParser), embedding-service/oc-embedding-service.jar, admin-ui/dist/ + nginx.conf, cli/oc-cli.jar, mcf-migrator/oc-mcf-migrator.jar, all .sh/.bat scripts, config/, and systemd/ service units |
opencrawling-runtime-dist-<version>.zip |
~459 MB | Runtime / Worker / Control Plane VMs | Extracted app.jar + lib/ directory, bin/start-runtime.sh (.bat), decoupled role scripts (start-crawler.sh, start-ingestion-consumer.sh, start-writer-consumer.sh, start-mcp-server.sh), config/, and systemd/ units |
opencrawling-embedding-service-dist-<version>.zip |
~192 MB | Stateless Embedding Worker VMs |
oc-embedding-service.jar, bin/start-embedding-service.sh (.bat), config/opencrawling.env.example, and systemd/opencrawling-embedding-service.service
|
opencrawling-admin-ui-dist-<version>.zip |
~275 KB | Frontend Web / Reverse Proxy VMs (Nginx, Apache HTTP Server, IIS) | Pre-built React 19 + Vite static bundle (dist/index.html, dist/assets/*) and production nginx.conf
|
opencrawling-cli-dist-<version>.zip |
~91 MB | Bastion Hosts, Admin Workstations & CI/CD Runners | Standalone oc-cli.jar plus bin/oc (Linux/macOS) and bin/oc.bat (Windows) wrappers |
SHA256SUMS.txt & *.zip.sha256
|
< 1 KB | Security & Compliance Verification | SHA-256 digests for all 5 ZIP archives |
When you extract opencrawling-platform-dist-<version>.zip, the directory layout is structured for standard /opt/opencrawling (Linux/macOS) or C:\opencrawling (Windows Server) installations:
opencrawling-platform-dist-<version>/
├── INSTALL.md # Offline quick-start & operational manual
├── LICENSE # Apache License 2.0
├── bin/
│ ├── start-runtime.sh # All-in-One Runtime launcher (Linux/macOS)
│ ├── start-runtime.bat # All-in-One Runtime launcher (Windows Server)
│ ├── start-crawler.sh # Decoupled Role: Crawler Node only
│ ├── start-ingestion-consumer.sh # Decoupled Role: Apache Tika Ingestion Consumer only
│ ├── start-embedding-service.sh # Decoupled Role: Stateless Embedding Microservice (Linux/macOS)
│ ├── start-embedding-service.bat # Decoupled Role: Stateless Embedding Microservice (Windows)
│ ├── start-writer-consumer.sh # Decoupled Role: Vector Store Writer Consumer only
│ ├── start-mcp-server.sh # Decoupled Role: Secure MCP Server & REST Control Plane
│ ├── oc # OpenCrawling CLI wrapper (Linux/macOS)
│ └── oc.bat # OpenCrawling CLI wrapper (Windows)
├── config/
│ ├── opencrawling.env.example # Central environment variable template
│ └── application-vm.yml # Reference Spring Boot YAML configuration
├── runtime/
│ ├── app.jar # Extracted Spring Boot application entry point
│ └── lib/ # Extracted dependency JARs (required for Apache Tika ForkParser)
├── embedding-service/
│ └── oc-embedding-service.jar # Standalone Spring AI Embedding Microservice
├── admin-ui/
│ ├── dist/ # Pre-compiled React 19 SPA static bundle
│ └── nginx.conf # Reverse proxy & SPA fallback configuration for Nginx
├── cli/
│ └── oc-cli.jar # Standalone OpenCrawling CLI executable JAR
├── mcf-migrator/
│ └── oc-mcf-migrator.jar # Apache ManifoldCF migration tool
├── systemd/
│ ├── opencrawling-runtime.service # Linux systemd unit for Runtime / Control Plane
│ └── opencrawling-embedding-service.service # Linux systemd unit for Embedding Microservice
├── data/
│ ├── claims/ # Default local Claim-Check payload directory
│ └── reports/ # Default Lakehouse Audit Report (Parquet/CSV) directory
└── logs/ # Application log output directory
During packaging, OpenCrawling runs Spring Boot's layered JAR extractor:
java -Djarmode=tools -jar oc-runtime-<version>.jar extract --destination extracted-runtimeThis separates runtime/app.jar from its third-party libraries in runtime/lib/. This is critical on bare-metal VMs because Apache Tika 4's out-of-process PipesForkParser spawns isolated child JVMs to parse untrusted PDFs, Office files, and archives safely; those child JVMs require a flat filesystem directory of JARs (-Dtika.extras.dir=$BASE_DIR/runtime/lib, automatically configured by bin/start-runtime.sh and bin/start-runtime.bat) rather than nested Spring Boot fat-JAR entries.
The workflow .github/workflows/dist-release-publish.yml is triggered automatically on:
-
GitHub Release published (
on: release: types: [published]) -
Version tag push (
on: push: tags: ['v*']) -
Manual dispatch (
on: workflow_dispatchwith inputrelease_version, e.g.,1.0.4)
Pipeline Stages:
-
build-vm-distributions(ubuntu-24.04, Eclipse Temurin JDK 25):- Resolves the release version and tag name.
- Synchronizes the version banner in
oc-cli. - Executes
./scripts/package-dist.sh "<version>"to compileoc-runtime,oc-embedding-service,oc-admin-ui,oc-cli, andoc-mcf-migrator, extract the Spring Boot runtime layers, generate startup scripts andsystemdunits, and assemble the 5.ziparchives plusSHA256SUMS.txt. - Runs an automated smoke test on
opencrawling-platform-dist-<version>.zipverifying the presence ofruntime/app.jar,runtime/lib/,embedding-service/oc-embedding-service.jar,admin-ui/dist/index.html,cli/oc-cli.jar, and executing./bin/oc --version. - Uploads all
.zipand.sha256files as workflow artifacts (opencrawling-vm-dist-packages).
-
publish-dist-release-assets:- Downloads the verified distribution artifacts and attaches all 5
.ziparchives, individual.sha256files, andSHA256SUMS.txtdirectly to the GitHub Release page viasoftprops/action-gh-release.
- Downloads the verified distribution artifacts and attaches all 5
If you are building custom connectors or air-gapped internal releases from source, you can generate all 5 ZIP archives on any machine with JDK 25 and Apache Maven 3.9+:
# Full Maven build + ZIP packaging (outputs to ./dist/)
./scripts/package-dist.sh 1.0.4
# Fast packaging if Maven targets are already compiled in target/ directories
SKIP_BUILD=true ./scripts/package-dist.sh 1.0.4- Operating System: Linux x86_64 or ARM64 (RHEL 9+, Ubuntu 22.04/24.04 LTS, Debian 12+, SLES 15+, Amazon Linux 2023).
-
Java Runtime: Java 25 (JDK or JRE 25) installed and available on
PATHor specified viaJAVA_HOME:java -version # openjdk version "25" 2025-09-16 LTS -
Network Connectivity to Backing Services:
-
PostgreSQL 17+ with
pgvector(default port5432) or your chosen Vector Database. -
Apache Kafka (default port
9092) — required when running asynchronous pipelines or decoupled consumers. -
Redis 7+ (default port
6379) — distributed crawl state, throttling gates, and caching. -
Ollama (
11434) or OpenAI-compatible endpoint — if generating vector embeddings or using AIOps / Auto-Narrativization Copilot.
-
PostgreSQL 17+ with
export OC_VERSION="1.0.4"
# Download the full platform distribution and checksum manifest
curl -LO "https://github.com/opencrawling/opencrawling/releases/download/v${OC_VERSION}/opencrawling-platform-dist-${OC_VERSION}.zip"
curl -LO "https://github.com/opencrawling/opencrawling/releases/download/v${OC_VERSION}/SHA256SUMS.txt"
# Verify archive integrity
sha256sum --ignore-missing -c SHA256SUMS.txt# Create dedicated non-root system user
sudo groupadd --system opencrawling || true
sudo useradd --system --gid opencrawling --home-dir /opt/opencrawling --shell /sbin/nologin opencrawling || true
# Extract archive and set ownership
sudo mkdir -p /opt/opencrawling
sudo unzip -q "opencrawling-platform-dist-${OC_VERSION}.zip" -d /opt/opencrawling
sudo ln -sfn "/opt/opencrawling/opencrawling-platform-dist-${OC_VERSION}" /opt/opencrawling/current
sudo chown -R opencrawling:opencrawling /opt/opencrawlingCopy the template environment file and adjust endpoints and credentials for your VM infrastructure:
cd /opt/opencrawling/current
sudo -u opencrawling cp config/opencrawling.env.example config/opencrawling.env
sudo -u opencrawling vi config/opencrawling.envExample config/opencrawling.env:
# Java 25 Runtime Options
JAVA_OPTS="-Xms1g -Xmx4g --enable-preview --enable-native-access=ALL-UNNAMED"
# PostgreSQL / pgvector Connection
SPRING_AI_VECTORSTORE_PGVECTOR_URL=jdbc:postgresql://pg-vm.internal:5432/opencrawling
SPRING_AI_VECTORSTORE_PGVECTOR_USERNAME=opencrawling
SPRING_AI_VECTORSTORE_PGVECTOR_PASSWORD=strong_production_password
# Apache Kafka Bootstrap Servers
SPRING_KAFKA_BOOTSTRAP_SERVERS=kafka-vm-1.internal:9092,kafka-vm-2.internal:9092
# Redis Cache / State Store
SPRING_DATA_REDIS_HOST=redis-vm.internal
SPRING_DATA_REDIS_PORT=6379
SPRING_DATA_REDIS_PASSWORD=strong_redis_password
# Ollama / OpenAI Embedding Provider
SPRING_AI_OLLAMA_BASE_URL=http://ollama-gpu-vm.internal:11434
SPRING_AI_OPENAI_API_KEY=your-api-key
# Claim-Check & Lakehouse Audit Report Directories (use shared NFS/NAS mount in multi-VM setups)
SPRING_OPENCRAWLING_CLAIM_CHECK_LOCAL_DIR=/mnt/shared-nas/opencrawling/claims
SPRING_OPENCRAWLING_REPORTING_LOCAL_DIR=/mnt/shared-nas/opencrawling/reports
# Apache Tika Out-of-Process ForkParser Settings
OPENCRAWLING_TIKA_FORK_ENABLED=true
OPENCRAWLING_TIKA_TIMEOUT_MS=30000
OPENCRAWLING_TIKA_JVM_ARGS=-Xmx1g
# Server Ports
SERVER_PORT=8080
OPENCRAWLING_GRPC_PORT=9095
OPENCRAWLING_GRPC_PUSH_ENABLED=true
OPENCRAWLING_GRPC_PUSH_PORT=9096Important
Multi-VM Claim Check Storage: When running decoupled Crawler and Ingestion Consumer roles across separate VMs, ensure both VMs can access the same Claim Check storage — either by pointing SPRING_OPENCRAWLING_CLAIM_CHECK_LOCAL_DIR to a shared NFS/SMB mount or by configuring the Apache Ozone Claim Check Store (SPRING_OPENCRAWLING_CLAIM_CHECK_STORE_TYPE=ozone).
Starts the entire platform in a single JVM process (REST API, Embedded React Admin UI on port 8080, Swagger UI, Crawler, Tika Ingestion, Vector Writer, Secure MCP Server, and gRPC servers on 9095/9096):
cd /opt/opencrawling/current
./bin/start-runtime.shOn dedicated worker VMs, use the specialized launcher scripts in bin/ (available in both opencrawling-platform-dist and opencrawling-runtime-dist / opencrawling-embedding-service-dist):
| VM Role | Command | Description |
|---|---|---|
| 1. Crawler VM | ./bin/start-crawler.sh |
Runs repository connectors in non-web mode (SPRING_MAIN_WEB_APPLICATION_TYPE=none), offloads raw binaries to Claim Check storage, and publishes crawl events to Kafka. |
| 2. Ingestion Consumer VM | ./bin/start-ingestion-consumer.sh |
Consumes raw document claims from Kafka, extracts text/metadata via Apache Tika 4 ForkParser (runtime/lib), applies transformation & narrativization pipelines, and publishes chunks to Kafka. |
| 3. Embedding Service VM | ./bin/start-embedding-service.sh |
Runs oc-embedding-service.jar (default port 8082), consuming text chunks from Kafka and computing dense vector embeddings via Spring AI (Ollama / OpenAI). |
| 4. Vector Writer VM | ./bin/start-writer-consumer.sh |
Consumes embedded document chunks and tombstones from Kafka and writes them into the target Output Connector (PgVector, Solr, OpenSearch, Vespa, Qdrant, Milvus, Luxir, Hyland OpenArch, or Apache Ozone). |
| 5. Control Plane & MCP VM | ./bin/start-mcp-server.sh |
Exposes the REST API, Embedded Admin UI (:8080), OpenAPI Swagger UI, and the Zero-Trust Secure Model Context Protocol (MCP) Server (/mcp) without running background crawl workers. |
Ready-to-use systemd unit files are included in the systemd/ directory of the distribution archives.
If you created the /opt/opencrawling/current symlink, update WorkingDirectory, EnvironmentFile, and ExecStart (or extract directly into /opt/opencrawling):
# Copy unit files to /etc/systemd/system/
sudo cp /opt/opencrawling/current/systemd/opencrawling-runtime.service /etc/systemd/system/
sudo cp /opt/opencrawling/current/systemd/opencrawling-embedding-service.service /etc/systemd/system/
# Reload systemd daemon and enable service on boot
sudo systemctl daemon-reload
sudo systemctl enable --now opencrawling-runtime.service
# Check service status and follow logs
sudo systemctl status opencrawling-runtime.service
sudo journalctl -u opencrawling-runtime.service -fTip
Running a Decoupled Role via systemd: To dedicate a VM to a specific worker role (for example, an Ingestion Consumer VM), simply change ExecStart in /etc/systemd/system/opencrawling-runtime.service from /opt/opencrawling/bin/start-runtime.sh to /opt/opencrawling/bin/start-ingestion-consumer.sh.
Note that oc-runtime already embeds the Admin UI directly on http://<vm-host>:8080. However, if your enterprise architecture requires serving the frontend SPA from a dedicated DMZ web server or reverse proxy tier using opencrawling-admin-ui-dist-<version>.zip:
# 1. Extract the Admin UI distribution
unzip opencrawling-admin-ui-dist-1.0.4.zip
# 2. Copy static assets to Nginx web root
sudo mkdir -p /usr/share/nginx/html
sudo cp -R opencrawling-admin-ui-dist-1.0.4/dist/* /usr/share/nginx/html/
# 3. Install the provided Nginx configuration (and update proxy_pass upstream if needed)
sudo cp opencrawling-admin-ui-dist-1.0.4/nginx.conf /etc/nginx/conf.d/opencrawling.conf
sudo nginx -t && sudo systemctl reload nginxIn /etc/nginx/conf.d/opencrawling.conf, point the /api/ proxy_pass directive to the internal IP/hostname of your OpenCrawling Control Plane VM (http://<control-plane-vm>:8080/api/).
-
Install Java 25: Install Eclipse Temurin JDK 25 (
.msiinstaller) and ensureJAVA_HOMEis set orjava.exeis on the systemPATH. -
Extract the Distribution Archive: Extract
opencrawling-platform-dist-<version>.ziptoC:\opencrawling. -
Configure Environment Variables: Set environment variables in PowerShell or System Properties (or customize
bin\start-runtime.bat):cd C:\opencrawling\opencrawling-platform-dist-1.0.4 $env:JAVA_OPTS = "-Xms1g -Xmx4g" $env:SPRING_AI_VECTORSTORE_PGVECTOR_URL = "jdbc:postgresql://127.0.0.1:5432/opencrawling" $env:SPRING_KAFKA_BOOTSTRAP_SERVERS = "127.0.0.1:9092" $env:SPRING_DATA_REDIS_HOST = "127.0.0.1"
-
Launch Runtime, Embedding Service, or CLI:
# Start All-in-One Runtime .\bin\start-runtime.bat # Start Stateless Embedding Microservice (in a separate terminal or VM) .\bin\start-embedding-service.bat # Run the OpenCrawling CLI .\bin\oc.bat --version .\bin\oc.bat system status
Once the runtime service is running on your VM, verify health and connectivity using the bundled oc CLI or curl:
# 1. Verify CLI binary
/opt/opencrawling/current/bin/oc --version
# 2. Check Spring Boot Actuator Health Endpoint
curl -s http://localhost:8080/actuator/health | jq .
# 3. Query Platform Status via CLI
/opt/opencrawling/current/bin/oc system status --url http://localhost:8080-
Admin UI Dashboard:
http://<vm-hostname>:8080/ -
Interactive OpenAPI 3.1 / Swagger UI:
http://<vm-hostname>:8080/swagger-ui.html -
Secure MCP Server Endpoint:
http://<vm-hostname>:8080/mcp -
gRPC Internal Transport & Push Gateway:
<vm-hostname>:9095and<vm-hostname>:9096
- Home
- User & Installation Guide
- Quick Start
- VM & Bare-Metal Distribution Installation
- Local Development & Fast Docker Builds
- User Guide
- Architecture
- Zero-Trust Security & ACL Management
- Document Lifecycle Actions & Tombstones
- Apache Tika 4 Text Extraction
- Configuration Guide
- Connectors Reference
- Alfresco Content Services (ACS) Connector
- Alfresco Process Services (APS) Connector
- Flowable Repository Connector
- Camunda Repository Connector
- OASIS CMIS Repository Connector
- Doxis Repository Connector
- M-Files Repository Connector
- SharePoint Online Repository Connector
- Hyland OpenArch Integration (Repository, Output & MCP)
- Relational Database (JDBC) Connector
- Apache StormCrawler Repository Connector
- gRPC Push Repository Connector
- Apache Solr Output Connector
- Luxir Output Connector
- Apache SeaTunnel Output Connector
- Apache Ozone Output Connector (Migration Mode)
- Connector Execution Modes
- Model Context Protocol
- Apache Ozone Claim Check Store
- Lakehouse Audit Reporting & Ledger
- AI-Powered Observability (AIOps)
- Auto-Narrativization Copilot
- Maven Connector Archetypes
- Java Client SDK
- OpenCrawling CLI
- OpenAPI Specification & Swagger UI
- gRPC Internal Transport
- Migration Guide (ManifoldCF)
- Integration Test — Decoupled Pipeline
- M-Files Integration Test
- SharePoint Online Integration Test
- Hyland OpenArch Integration Test
- Local Docker Build Test
- Secure MCP Server Integration Test
- Narrativization Integration Test
- Reporting Integration Test