Quick Start

This page explains every Docker Compose file included in the project root and shows the exact commands needed to start each deployment mode.

Prerequisites

Requirement	Minimum version	Notes
Docker	24+	Docker Desktop or Docker Engine
Docker Compose	Plugin v2	`docker compose` (not `docker-compose`)
RAM	8 GB allocated to Docker	Kafka + Postgres + Ollama run concurrently
Disk	5 GB free in Docker VM	Ollama model cache ~2 GB per model
Java / Maven	JDK 25, Maven 3.9+	Only needed for source builds

Caution

If Postgres fails to start with No space left on device, the Docker VM disk is full. Free space before retrying:

docker system prune -af --volumes

Compose File Overview

File	Purpose	Who should use it
`docker-compose.yml`	Infrastructure only (Postgres, Redis, Ollama, Kafka)	Developers running the Java app locally with Maven
`docker-compose-apps.yml`	Application services only — requires the infrastructure stack to be running first	Developers who want to add the apps on top of the infra stack
`docker-compose-decoupled.yml`	Full decoupled stack — builds images from local source	Contributors & CI — integration tests, local development of the decoupled pipeline
`docker-compose-decoupled-dist.yml`	Full decoupled stack — uses pre-built `ghcr.io` images	End users — production-style deployment without needing Java or Maven

Option 1 — Infrastructure Only (`docker-compose.yml`)

Use this when you want to run the OpenCrawling Java application directly on your host machine (e.g. via mvn spring-boot:run) while the infrastructure services run in Docker.

Services started

Container	Image	Port	Purpose
`postgres-vector`	`pgvector/pgvector:pg17`	`5432`	Vector database (PGVector)
`redis-stack`	`redis/redis-stack:7.4.0-v8`	`6379`, `8001`	Cache / claim-check store + RedisInsight UI
`ollama`	`ollama/ollama:0.23.4`	`11434`	Local embedding model server
`ollama-model-puller`	`ollama/ollama:0.23.4`	—	One-shot: pulls `mxbai-embed-large` model
`kafka`	`apache/kafka:3.7.0`	`9092`	Message broker (KRaft mode, no Zookeeper)

Commands

# Start infrastructure
docker compose -f docker-compose.yml up -d

# Run the application locally (in a separate terminal)
mvn spring-boot:run -pl oc-runtime

# Stop infrastructure
docker compose -f docker-compose.yml down

Exposed endpoints

Service	URL
PGVector (psql)	`localhost:5432` — user `opencrawling`, password `opencrawling_password`
Redis	`localhost:6379` — password `redis_password`
RedisInsight UI	http://localhost:8001
Ollama API	http://localhost:11434
Kafka broker	`localhost:9092`

Option 2 — Infrastructure + Apps (`docker-compose.yml` + `docker-compose-apps.yml`)

Use this when you want to run all services in Docker — both the infrastructure and the OpenCrawling application containers — built from local source.

Important

You must build the project JAR files first:

mvn clean install -DskipTests

The Dockerfiles copy the .jar from the target/ directory.

Services added by `docker-compose-apps.yml`

Container	Image	Port	Purpose
`opencrawling-backend`	Built from `oc-runtime/Dockerfile`	`8080`	Runtime + MCP server + Admin API
`opencrawling-embedding`	Built from `docker/Dockerfile.embedding-consumer`	`8082`	Embedding consumer service
`opencrawling-frontend`	Built from `oc-admin-ui/Dockerfile`	`3000`	Admin web UI (Nginx)

Commands

# Step 1: Build JARs
mvn clean install -DskipTests

# Step 2: Start infrastructure
docker compose -f docker-compose.yml up -d

# Step 3: Start application services (uses the existing network)
docker compose -f docker-compose-apps.yml up -d

# View logs
docker compose -f docker-compose-apps.yml logs -f

# Stop apps only
docker compose -f docker-compose-apps.yml down

# Stop everything including infrastructure
docker compose -f docker-compose.yml down

Exposed endpoints

Service	URL
Admin UI	http://localhost:3000
Backend API / MCP Server	http://localhost:8080
Embedding Service	http://localhost:8082

Option 3 — Full Decoupled Stack from Source (`docker-compose-decoupled.yml`)

The recommended mode for contributors and CI pipelines. Builds all microservice Docker images from your local source code and runs the complete decoupled pipeline in a single compose project.

Important

Build the JARs first:

mvn clean install -DskipTests

Architecture

oc-crawler ──Kafka──► oc-ingestion-consumer ──Kafka──► oc-embedding-consumer ──Kafka──► oc-writer-consumer ──► PGVector
                                                                                                       ↑
                                                                                               oc-mcp-server
                                                                                               oc-admin-ui

Services

Infrastructure

Container	Image	Port
`postgres-vector-decoupled`	`pgvector/pgvector:pg17`	`5432`
`redis-stack-decoupled`	`redis/redis-stack:7.4.0-v8`	`6379`, `8001`
`ollama-decoupled`	`ollama/ollama:0.23.4`	`11434`
`ollama-model-puller-decoupled`	`ollama/ollama:0.23.4`	— (pulls `mxbai-embed-large`, `nomic-embed-text`, `all-minilm`)
`kafka-decoupled`	`apache/kafka:3.7.0`	`9092`

OpenCrawling Microservices

Container	Dockerfile	Port	Role
`oc-crawler-service`	`docker/Dockerfile.crawler`	—	Scans `/data`, publishes to Kafka
`oc-ingestion-consumer-service`	`docker/Dockerfile.ingestion-consumer`	—	Tika parsing + chunking
`oc-embedding-consumer-service`	`docker/Dockerfile.embedding-consumer`	`8082`	Ollama/OpenAI embedding
`oc-writer-service`	`docker/Dockerfile.writer-consumer`	—	PGVector persistence
`oc-mcp-server-service`	`docker/Dockerfile.mcp-server`	`8080`	MCP Streamable HTTP server + API
`oc-admin-ui-service`	`oc-admin-ui/Dockerfile`	`3000`	Admin web UI

Commands

# Build JARs
mvn clean install -DskipTests

# Start the full stack (builds images first)
docker compose -f docker-compose-decoupled.yml up -d --build

# Follow all logs
docker compose -f docker-compose-decoupled.yml logs -f

# Follow a specific service
docker compose -f docker-compose-decoupled.yml logs -f oc-embedding-consumer

# Scan a specific directory (override SCAN_PATH)
docker compose -f docker-compose-decoupled.yml up -d \
  --scale oc-crawler=0   # stop existing crawler
docker compose -f docker-compose-decoupled.yml run --rm \
  -e SPRING_OPENCRAWLING_SCAN_PATH=/data/my-folder oc-crawler

# Stop and remove containers (keeps volumes)
docker compose -f docker-compose-decoupled.yml down

# Full teardown including volumes
docker compose -f docker-compose-decoupled.yml down -v

Placing files to crawl

The crawler mounts ./oc-runtime/data as /data inside the container. Place files there before starting:

mkdir -p oc-runtime/data
cp ~/Documents/my-report.pdf oc-runtime/data/

Note

The oc-runtime/data/ directory is listed in .gitignore — local data is not committed.

Automated integration test

A shell script is available to run and verify the full decoupled pipeline automatically:

chmod +x scripts/test-docker-decoupled.sh
./scripts/test-docker-decoupled.sh

See Integration Test — Decoupled Pipeline for full documentation.

Option 4 — Full Decoupled Stack from Registry (`docker-compose-decoupled-dist.yml`)

The easiest way to run OpenCrawling. Uses pre-built images published to the GitHub Container Registry (ghcr.io/opencrawling/...). No Java, Maven, or source code required.

Commands

# Pull and start everything
docker compose -f docker-compose-decoupled-dist.yml up -d

# Follow logs
docker compose -f docker-compose-decoupled-dist.yml logs -f

# Stop
docker compose -f docker-compose-decoupled-dist.yml down

Available images

Image	Tag	Service
`ghcr.io/opencrawling/oc-crawler`	`latest`	Crawler
`ghcr.io/opencrawling/oc-ingestion-consumer`	`latest`	Ingestion consumer
`ghcr.io/opencrawling/oc-embedding-consumer`	`latest`	Embedding consumer
`ghcr.io/opencrawling/oc-writer-consumer`	`latest`	Vector writer
`ghcr.io/opencrawling/oc-mcp-server`	`latest`	MCP server
`ghcr.io/opencrawling/oc-admin-ui`	`latest`	Admin UI

Note

The dist file also includes build: sections, so contributors can rebuild and tag production images locally with:

docker compose -f docker-compose-decoupled-dist.yml build

Connector Type Configuration

Both decoupled compose files support selecting the repository connector via the CONNECTOR_TYPE environment variable on the oc-crawler service.

Filesystem connector (default)

oc-crawler:
  environment:
    - CONNECTOR_TYPE=filesystem
    - SPRING_OPENCRAWLING_SCAN_PATH=/data

Place files in ./oc-runtime/data/ — they are mounted to /data inside the container.

Alfresco connector

oc-crawler:
  environment:
    - CONNECTOR_TYPE=alfresco
    - SCAN_PATH=/Company Home
    - ALFRESCO_URL=http://alfresco:8080/alfresco/api/-default-/public/alfresco/versions/1
    - ALFRESCO_USERNAME=admin
    - ALFRESCO_PASSWORD=admin

Apache Iceberg connector

Use the Iceberg connector to ingest structured data from Apache Iceberg table catalogs (REST, Hive Metastore, Hadoop, or AWS Glue). The SCAN_PATH is an Iceberg table identifier in <namespace>.<table> format.

oc-crawler:
  environment:
    - CONNECTOR_TYPE=iceberg
    - SCAN_PATH=analytics.events           # <namespace>.<table>
    - ICEBERG_CATALOG_TYPE=rest            # rest | hive | hadoop | glue | in-memory
    - ICEBERG_CATALOG_URI=http://iceberg-rest:8181
    - ICEBERG_WAREHOUSE=s3://warehouse/

Start the companion Iceberg infrastructure (REST catalog + MinIO) using the dedicated compose file:

docker compose -f oc-iceberg-repository-connector/docker-compose.yml up -d

See Connectors Reference for catalog type details and metadata extraction behaviour.

Tip

See Connectors Reference for the full list of available repository connectors, configuration properties, and Docker Compose examples for all connector types.

Useful Commands Reference

# Check health of all containers
docker compose -f docker-compose-decoupled.yml ps

# Check vector records ingested
docker exec -i postgres-vector-decoupled psql -U opencrawling -d opencrawling -c \
  "SELECT 'vector_store' AS tbl, count(*) FROM vector_store
   UNION ALL SELECT 'vector_store_384', count(*) FROM vector_store_384
   UNION ALL SELECT 'vector_store_768', count(*) FROM vector_store_768
   UNION ALL SELECT 'vector_store_1024', count(*) FROM vector_store_1024;"

# Free Docker disk space if running low
docker system prune -af --volumes

Choosing the Right Option

Are you a contributor or running CI?
├── Yes → Option 3: docker-compose-decoupled.yml  (source build)
└── No  → Do you have Java/Maven installed?
          ├── Yes, and want to debug the Java app →
          │     Option 1: docker-compose.yml  (infra only, run app with mvn)
          │     Option 2: docker-compose.yml + docker-compose-apps.yml  (all in Docker)
          └── No → Option 4: docker-compose-decoupled-dist.yml  (pre-built images)

🗺️ Documentation

🧪 Testing

Integration Test — Decoupled Pipeline

Uh oh!

Quick Start

Quick Start

Prerequisites

Compose File Overview

Option 1 — Infrastructure Only (docker-compose.yml)

Services started

Commands

Exposed endpoints

Option 2 — Infrastructure + Apps (docker-compose.yml + docker-compose-apps.yml)

Services added by docker-compose-apps.yml

Commands

Exposed endpoints

Option 3 — Full Decoupled Stack from Source (docker-compose-decoupled.yml)

Architecture

Services

Infrastructure

OpenCrawling Microservices

Commands

Placing files to crawl

Automated integration test

Option 4 — Full Decoupled Stack from Registry (docker-compose-decoupled-dist.yml)

Commands

Available images

Connector Type Configuration

Filesystem connector (default)

Alfresco connector

Apache Iceberg connector

Useful Commands Reference

Choosing the Right Option

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

🗺️ Documentation

🧪 Testing

🤝 Community

Clone this wiki locally

Option 1 — Infrastructure Only (`docker-compose.yml`)

Option 2 — Infrastructure + Apps (`docker-compose.yml` + `docker-compose-apps.yml`)

Services added by `docker-compose-apps.yml`

Option 3 — Full Decoupled Stack from Source (`docker-compose-decoupled.yml`)

Option 4 — Full Decoupled Stack from Registry (`docker-compose-decoupled-dist.yml`)