Skip to content

Apache Ozone Claim Check Store

Jump to bottom
Piergiorgio Lucidi edited this page Jul 20, 2026 · 1 revision

Apache Ozone Claim Check Store

OpenCrawling supports Apache Ozone (v2.2.0+) as a first-class distributed object store provider for the Claim Check Pattern.

In large-scale enterprise deployments, crawling millions of files (PDFs, office documents, CAD files) can produce hundreds of gigabytes of raw binary data. Sending raw file payloads through messaging channels like Apache Kafka causes topic partition congestion, high memory pressure, and consumer degradation. OpenCrawling solves this by saving raw document payloads to a Claim Check store and circulating lightweight reference URIs through Kafka.

🏗️ Architecture & Component Integration

Apache Ozone is a scalable, redundant, distributed object store optimized for Big Data and enterprise AI pipelines.

┌──────────────────────────────┐
│  OpenCrawling Repository     │
│       Connector              │
└──────────────┬───────────────┘
               │ (1) Upload Raw Payload
               ▼
┌──────────────────────────────┐       ┌────────────────────────────────┐
│  OzoneClaimCheckStore        ├──────>│  Apache Ozone S3 Gateway (s3g) │
│  (AWS S3 SDK v2 Client)      │       │          (Port 9878)           │
└──────────────┬───────────────┘       └───────────────┬────────────────┘
               │                                       │ (2) Store Payload
               │ (3) Publish s3:// URI                 ▼
               ▼                       ┌────────────────────────────────┐
┌──────────────────────────────┐       │  Ozone Manager (om)            │
│   Apache Kafka Topic         │       │  & Storage Container Manager   │
│  (opencrawling-ingestion)    │       └────────────────────────────────┘
└──────────────┬───────────────┘
               │ (4) Consume Message & Fetch Content
               ▼
┌──────────────────────────────┐
│  IngestionConsumer           │
│  (Apache Tika Extractor)     │
└──────────────────────────────┘

Supported URIs

OzoneClaimCheckStore supports and parses all standard Apache Ozone URI formats:

  • S3 Protocol URIs: s3://claims/doc-uuid.pdf
  • Ozone File System (OFS) URIs: ofs://s3v/claims/doc-uuid.pdf
  • HTTP/S Gateway URIs: http://s3g:9878/s3v/claims/doc-uuid.pdf

⚙️ Configuration Guide

1. Spring Boot Properties (application.yml)

To activate Apache Ozone as the active Claim Check store, configure spring.opencrawling.claim-check:

spring:
  opencrawling:
    claim-check:
      store: ozone                       # Values: "local", "ozone", or "s3"
      cleanup-on-consume: true           # Delete object from Ozone after consumption
      ozone:
        s3-endpoint: http://localhost:9878
        volume: s3v                      # Ozone Volume name
        bucket: claims                   # Ozone Bucket name
        access-key: ozone
        secret-key: ozone-secret
        path-style-access: true          # Required for Ozone S3 Gateway
        region: us-east-1
        auto-create-bucket: true         # Automatically initialize volume/bucket if missing

2. Environment Variables

In containerized or Kubernetes environments, you can configure Ozone via environment variables:

Environment Variable Description Default Value
SPRING_OPENCRAWLING_CLAIM_CHECK_STORE Active store type (local or ozone) local
SPRING_OPENCRAWLING_CLAIM_CHECK_OZONE_S3_ENDPOINT Apache Ozone S3 Gateway URL http://s3g:9878
SPRING_OPENCRAWLING_CLAIM_CHECK_OZONE_VOLUME Target Ozone Volume s3v
SPRING_OPENCRAWLING_CLAIM_CHECK_OZONE_BUCKET Target Ozone Bucket claims
OZONE_ACCESS_KEY_ID Ozone S3 Access Key ozone
OZONE_SECRET_ACCESS_KEY Ozone S3 Secret Key ozone-secret

🐳 Docker Compose Stack

OpenCrawling provides built-in Docker Compose configurations (docker-compose-decoupled.yml) with a complete 4-node Apache Ozone cluster:

services:
  ozone-scm:
    image: apache/ozone:2.2.0
    container_name: ozone-scm
    hostname: scm
    environment:
      OZONE-SITE.XML_ozone.scm.client.address: scm
      OZONE-SITE.XML_ozone.om.address: om
    command: ["ozone", "scm"]

  ozone-om:
    image: apache/ozone:2.2.0
    container_name: ozone-om
    hostname: om
    environment:
      OZONE-SITE.XML_ozone.scm.client.address: scm
      OZONE-SITE.XML_ozone.om.address: om
    command: ["ozone", "om"]
    depends_on:
      - ozone-scm

  ozone-datanode:
    image: apache/ozone:2.2.0
    container_name: ozone-datanode
    environment:
      OZONE-SITE.XML_ozone.scm.client.address: scm
      OZONE-SITE.XML_ozone.om.address: om
    command: ["ozone", "datanode"]
    depends_on:
      - ozone-scm
      - ozone-om

  ozone-s3g:
    image: apache/ozone:2.2.0
    container_name: ozone-s3g
    hostname: s3g
    ports:
      - "9878:9878"
    environment:
      OZONE-SITE.XML_ozone.scm.client.address: scm
      OZONE-SITE.XML_ozone.om.address: om
    command: ["ozone", "s3g"]
    depends_on:
      - ozone-scm
      - ozone-om

🔍 Payload Storage Location & Verification Methods

1. Storage Location Hierarchy

Level Location Explanation
S3 Gateway (API) http://localhost:9878/claims/<file-key> Public HTTP S3 endpoint exposed on port 9878
Apache Ozone Namespace /s3v/claims/<file-key> Native volume (s3v) and bucket (claims) managed by Ozone Manager (om)
Datanode Physical Storage /data/hdds/ Replicated block containers stored physically inside ozone-datanode

2. How to Verify Stored Content

A. Verification via HTTP curl (S3 Gateway Endpoint)

Fetch the raw content of any claim check payload directly from the S3 Gateway endpoint:

curl -s http://localhost:9878/claims/data_ozone-claim-check-test.txt

B. Verification via AWS CLI

List and retrieve objects using standard S3 commands:

# List all claim check keys in the Ozone bucket
AWS_ACCESS_KEY_ID=ozone AWS_SECRET_ACCESS_KEY=ozone-secret \
aws --endpoint-url=http://localhost:9878 s3 ls s3://claims/

# Read the document content directly
AWS_ACCESS_KEY_ID=ozone AWS_SECRET_ACCESS_KEY=ozone-secret \
aws --endpoint-url=http://localhost:9878 s3 cp s3://claims/data_ozone-claim-check-test.txt -

C. Verification via Native Apache Ozone CLI

Execute the native Ozone CLI inside the ozone-om container to list keys and inspect metadata in /s3v/claims:

# List all keys in the s3v volume / claims bucket
docker exec -it ozone-om ozone sh key list /s3v/claims

# Inspect key metadata
docker exec -it ozone-om ozone sh key info /s3v/claims/data_ozone-claim-check-test.txt

🧪 Automated Integration Testing

An automated end-to-end integration test script is included in the project:

# Execute from project root
./scripts/test-ozone-decoupled.sh

What the test performs:

  1. Boots PostgreSQL (with pgvector), Ollama (mxbai-embed-large), Apache Kafka, and Apache Ozone 2.2.0.
  2. Waits for Apache Ozone S3 Gateway on port 9878 to report HTTP readiness.
  3. Triggers oc-crawler-service with SPRING_OPENCRAWLING_CLAIM_CHECK_STORE=ozone.
  4. Confirms that documents are stored in Ozone, consumed from Kafka, embedded into 1024-dimension vectors, and stored in pgvector.
  5. Verifies Secure MCP Server connectivity.
  6. Performs automated teardown.

💡 Operational Notes & Troubleshooting

Transient RPC Connection Retry Messages During Startup

When launching the stack with docker compose up -d, you may observe log entries in ozone-s3g:

INFO retry.RetryInvocationHandler: ... java.net.UnknownHostException: Invalid host name: local host is: "s3g/172.18.0.9"; destination host is: "om":9862 ... Trying to failover after sleeping for 4000ms.
  • Explanation: This is a normal transient startup message. When ozone-s3g starts, ozone-om takes a few seconds to initialize RocksDB and bind port 9862. Hadoop's internal RetryInvocationHandler retries automatically until ozone-om is ready.

Local Development Directory Fallback

For host-based Maven builds (mvn clean install), claim check files default to target/claims inside the project build folder to avoid root system permission errors on macOS and Linux host machines. Inside Docker containers, set SPRING_OPENCRAWLING_CLAIM_CHECK_LOCAL_DIR=/data/claims.

🗺️ Documentation

🧪 Testing

🤝 Community

Clone this wiki locally