Local Development and Docker Builds

Local Development & Fast Docker Builds ⚡

This guide explains how to build and develop OpenCrawling locally using dedicated configurations that leverage your local host Maven repository cache (<USER_HOME>/.m2), providing fast, near-instant container builds without modifying the production Dockerfiles used in GitHub Actions CI workflows.

---

🎯 The Problem & Architecture

In continuous integration (GitHub Actions), container images are built from scratch to guarantee clean, hermetic, and isolated builds. In these production Dockerfiles (docker/Dockerfile.*), Maven dependencies are downloaded over the network inside the container (mvn dependency:go-offline / mvn clean package).

For local development across the decoupled microservices architecture (oc-crawler, oc-ingestion-consumer, oc-embedding-consumer, oc-writer-consumer, oc-mcp-server), building from scratch creates severe friction:

• Repeatedly downloading hundreds of megabytes over the network.
• Taking 4 to 10+ minutes per rebuild.
• Ignoring dependencies already cached on the developer's host machine in ~/.m2/repository.

OpenCrawling Local Development Solution

OpenCrawling resolves this with a two-tier local acceleration architecture that guarantees zero modifications to CI Dockerfiles:

flowchart TD
    subgraph Host["Developer Workstation"]
        M2["Host Maven Cache<br/>(~/.m2/repository)"]
        SRC["OpenCrawling Source<br/>(./)"]
    end

    subgraph FastBuildPaths["Fast Local Build Mechanisms"]
        direction TB
        subgraph BuildKit["1. BuildKit Cache Mounts"]
            BK["Local Dockerfiles (docker/local/Dockerfile.*)<br/>RUN --mount=type=cache,target=/root/.m2,id=opencrawling-m2-cache"]
        end
        subgraph BuilderService["2. Containerized Maven Builder"]
            OMB["oc-maven-builder (docker-compose-maven-builder.yml)<br/>Volumes: ${HOME}/.m2:/root/.m2 and .:/app"]
        end
    end

    subgraph Targets["Container Stacks & Runtimes"]
        DC["Decoupled Microservices<br/>(docker-compose-decoupled-local.yml)"]
        APP["Monolith Apps Stack<br/>(docker-compose-apps-local.yml)"]
        CONN["Connector Stacks<br/>(Vespa, Solr, Luxir, Qdrant, Milvus)"]
    end

    M2 -.->|"Persistent Cache"| BK
    M2 <==>|"Bind Mount Volume"| OMB
    SRC <==>|"Workspace Volume"| OMB
    BK --> DC
    BK --> APP
    OMB --> Targets

Loading

1. BuildKit Persistent Cache Mounts (docker/local/Dockerfile.*): Dedicated local Dockerfiles use RUN --mount=type=cache,target=/root/.m2,id=opencrawling-m2-cache,sharing=shared. All services share the same persistent cache volume on the local Docker engine. Redundant downloads (dependency:go-offline) are eliminated, reducing rebuild times from minutes to seconds.
2. Containerized Maven Builder (oc-maven-builder): Runs the official maven:3-eclipse-temurin-25 container with ${HOME}/.m2:/root/.m2 directly mounted, allowing developers to build and test code without requiring JDK 25 or Maven installed on their host machine.

---

🛠 Available Local Compose Files

File	Purpose	Usage Command
docker-compose-decoupled-local.yml	Standalone decoupled stack using local fast Dockerfiles	docker compose -f docker-compose-decoupled-local.yml up -d --build
docker-compose-decoupled.local.yml	Override overlay for docker-compose-decoupled.yml	docker compose -f docker-compose-decoupled.yml -f docker-compose-decoupled.local.yml up -d --build
docker-compose-apps-local.yml	Standalone apps stack (backend, embedding, UI) with local Dockerfiles	docker compose -f docker-compose-apps-local.yml up -d --build
docker-compose-apps.local.yml	Override overlay for docker-compose-apps.yml	docker compose -f docker-compose.yml -f docker-compose-apps.yml -f docker-compose-apps.local.yml up -d --build
docker-compose-maven-builder.yml	Containerized Maven builder mounting host ~/.m2 volume	docker compose -f docker-compose-maven-builder.yml run --rm oc-maven-builder

---

🚀 Step-by-Step Usage

1. Fast Builds for Decoupled Microservices

To build and start the complete decoupled stack (Kafka, Postgres pgvector, Redis, Ollama, OpenTelemetry, Crawler, Ingestion Consumer, Embedding Consumer, Writer Consumer, Secure MCP Server, and Admin UI):

# Build and start using the standalone local compose file
docker compose -f docker-compose-decoupled-local.yml up -d --build

Or using the compose override:

docker compose -f docker-compose-decoupled.yml -f docker-compose-decoupled.local.yml up -d --build

2. Fast Builds for Apps Stack (Backend + UI)

To build and run the backend runtime and React Admin UI:

docker compose -f docker-compose-apps-local.yml up -d --build

---

☕ The oc-maven-builder Service

The oc-maven-builder service compiles OpenCrawling inside an official Eclipse Temurin JDK 25 container while mapping your host machine's ~/.m2 repository and workspace:

services:
  oc-maven-builder:
    image: maven:3-eclipse-temurin-25
    container_name: oc-maven-builder
    working_dir: /app
    volumes:
      - ${HOME}/.m2:/root/.m2   # Host Maven repository cache
      - .:/app                 # Host workspace directory
    command: mvn clean package -DskipTests
    profiles: ["build"]

Note

oc-maven-builder is protected by profiles: ["build"]. When you run docker compose up -d, it is not started as a daemon process. It executes on-demand only when invoked with run --rm.

Benefits:

• Zero Host Tooling Required: Developers without Java 25 or Maven installed on their host machine can compile, package, and test OpenCrawling in an identical environment.
• Bi-Directional Cache Sharing: Any dependency downloaded by oc-maven-builder is saved directly into the host's ~/.m2 folder. Any existing dependency already in ~/.m2 is immediately available to the container.
• Host Artifact Output: Compiled JAR files (target/*.jar) are written directly to your host workspace, making them accessible to IDEs and CLI tools.

Running with Any Compose Stack:

# 1. Compile all project modules
docker compose -f docker-compose-maven-builder.yml run --rm oc-maven-builder

# 2. Start any specific specialized connector stack (e.g. Vespa, Solr, Luxir)
docker compose -f oc-vespa-output-connector/docker/docker-compose-decoupled-with-vespa.yml up -d

Running Custom Maven Goals:

# Run unit tests for oc-runtime only
docker compose -f docker-compose-maven-builder.yml run --rm oc-maven-builder mvn test -pl oc-runtime

# Fast offline build
docker compose -f docker-compose-maven-builder.yml run --rm oc-maven-builder mvn clean package -o -DskipTests

# Inspect dependency hierarchy
docker compose -f docker-compose-maven-builder.yml run --rm oc-maven-builder mvn dependency:tree -pl oc-vector-output-connector

---

⚡ Speed Benchmark Comparison

Build Scenario	First Build (Clean)	Warm Rebuild (Source Change)	Layer-Cached Build
Production CI Dockerfiles (docker/Dockerfile.*)	~4m 30s	~4m 15s (re-downloads dependencies)	~1s
Local Cache Dockerfiles (docker/local/Dockerfile.*)	~4m 15s	~3 seconds (reads from cache mount)	~0.5s
Host Compilation + oc-maven-builder	~35s	~5 seconds	N/A

---

🧪 Integration Testing with Local Cache

Existing integration test scripts under scripts/ (e.g., test-mcp-decoupled.sh, test-ois-ingest-decoupled.sh) remain completely untouched to preserve CI reproducibility.

To run existing integration test scripts with the local fast build configuration, use the Docker Compose COMPOSE_FILE environment variable:

# Transparently overlay local cache Dockerfiles onto existing test scripts
export COMPOSE_FILE="docker-compose-decoupled.yml:docker-compose-decoupled.local.yml"

# Run integration test without re-downloading dependencies
./scripts/test-ois-ingest-decoupled.sh

Running the Dedicated Local Build Test Suite

OpenCrawling provides an automated integration test script to verify local build acceleration:

chmod +x scripts/test-docker-local-dev.sh
./scripts/test-docker-local-dev.sh

This test script validates:

1. Production CI Dockerfile integrity (ensuring zero changes).
2. Dedicated local Dockerfiles and BuildKit cache mount syntax.
3. Compose configuration syntax across all local standalone and override files.
4. CLI container image build with warm cache timing verification (~3s).
5. Container runtime execution with Java 25 preview features.