Apache StormCrawler Repository Connector

Apache StormCrawler Repository Connector & OpenCrawling Bolt

Modules:

• Repository Connector: oc-stormcrawler-repository-connector (Class: org.opencrawling.stormcrawler.StormCrawlerRepositoryConnector)
• Apache Storm Bolt: oc-stormcrawler-bolt (Class: org.opencrawling.stormcrawler.bolt.OpenCrawlingBolt)

---

📌 Overview

The Apache StormCrawler Integration provides an enterprise-grade distributed web crawling solution for OpenCrawling. It bridges Apache StormCrawler 3.7.0 running atop Apache Storm 3.1.0 clusters on Java 25 with the OpenCrawling ingestion and vector pipeline.

The integration is architected in two complementary modules:

1. oc-stormcrawler-repository-connector: A Spring Boot RepositoryConnector implementing the OpenCrawling connector SPI. It interacts with the Apache Storm Nimbus REST API to monitor cluster health, query and supervise topologies, enforce web crawling politeness, and stream crawled documents into the pipeline using Java 25 Structured Task Scope.
2. oc-stormcrawler-bolt: An Apache Storm bolt library (OpenCrawlingBolt extends BaseRichBolt) that runs directly inside distributed Storm worker processes. It intercepts parsed web documents and URL lifecycle deletion signals, normalizes them into Open Ingestion Standard (OIS) UPSERT and DELETE payloads, and dispatches them to OpenCrawling.

graph TD
    subgraph Apache Storm 3.1.0 Cluster
        Spout["Crawl Spout (URLs)"] --> Fetcher["FetcherBolt"]
        Fetcher --> Parser["ParserBolt (HTML / Text)"]
        Parser -->|"Default Stream (url, content, metadata)"| OCBolt["OpenCrawlingBolt"]
        Fetcher -.->|"Status Stream (404 / 410 / FETCH_ERROR)"| OCBolt
        Nimbus["Storm Nimbus REST API (:8080 / :6627)"]
    end

    subgraph OpenCrawling Ingestion Runtime
        OCBolt -->|"HTTP POST /api/v1/ingest/ois/{jobId} (action: UPSERT)"| OIS_Ingest["OIS Ingestion Endpoint"]
        OCBolt -->|"HTTP POST /api/v1/ingest/ois/{jobId} (action: DELETE)"| OIS_Delete["OIS Tombstone Purge"]
        Connector["StormCrawlerRepositoryConnector"] -->|"Cluster Supervision & Health Check"| Nimbus
        OIS_Ingest --> Kafka["Kafka (opencrawling-raw-documents)"]
        Kafka --> Embedding["oc-embedding-consumer (Ollama / OpenAI)"]
        Embedding --> Writer["Vector Store (PGVector / Milvus / Solr / etc.)"]
    end

Loading

---

🚀 Key Features

• Apache Storm 3.1.0 & StormCrawler 3.7.0 on Java 25:
  • Fully tested and compatible with Temurin Java 25.
  • Storm 3.1.0 resolves JEP 486 Security Manager removal (Subject.getSubject deprecation) while supporting modern Java 25 classfile bytecode.
• Dual-Stream Processing (OpenCrawlingBolt):
  • Default Content Stream: Receives (url, content, metadata) emitted by ParserBolt, generates SHA-256 content hashes, and constructs standard OIS UPSERT documents.
  • Status & Deletion Stream: Subscribes to Constants.StatusStreamName (status). When a crawled URL returns HTTP 404, 410, or Status.FETCH_ERROR, it automatically issues an OIS DELETE tombstone to purge stale document vectors from downstream vector stores.
• Content Deduplication & Hash Tracking:
  • Computes cryptographic hashes (SHA-256 or MD5) over document content bytes to prevent redundant downstream embedding and indexing.
• High Concurrency with Java 25 Structured Concurrency:
  • Utilizes Java 25 Virtual Threads and StructuredTaskScope within StormCrawlerRepositoryConnector for asynchronous seed URL validation and parallelized target scanning.
• Enterprise Web Politeness & Filtering:
  • Configurable crawl delay (delayMs), custom user-agent headers (OpenCrawling-StormCrawler-Bot/1.0), robots.txt compliance, and regex-based URL inclusion/exclusion patterns.
• Nimbus Cluster REST Supervision:
  • Real-time monitoring of cluster summary (/api/v1/cluster/summary), topology health (/api/v1/topology/summary), worker assignments, and lifecycle actions.
• Multiple Transport Modes:
  • REST: Asynchronous HTTP client dispatching JSON OIS payloads to the OpenCrawling ingestion endpoint.
  • MEMORY: In-memory dispatcher designed for integration testing and embedded topologies.
• Shaded Topology Fat JAR:
  • The oc-stormcrawler-bolt module builds both a standard library JAR (for reactor dependencies) and a shaded fat JAR (oc-stormcrawler-bolt-*-topology.jar) bundling all dependencies for deployment directly via storm jar.

---

⚙️ Configuration Properties

1. Spring Boot Properties (application.yml)

Configured under spring.opencrawling.connector.stormcrawler.*:

Property	Environment Variable	Default	Description
spring.opencrawling.connector.type	CONNECTOR_TYPE	stormcrawler	Activates the StormCrawler repository connector bean
spring.opencrawling.connector.stormcrawler.nimbus-host	STORMCRAWLER_NIMBUS_HOST	localhost	Apache Storm Nimbus hostname
spring.opencrawling.connector.stormcrawler.nimbus-port	STORMCRAWLER_NIMBUS_PORT	6627	Nimbus Thrift service RPC port
spring.opencrawling.connector.stormcrawler.nimbus-rest-url	STORMCRAWLER_NIMBUS_REST_URL	http://localhost:8080	Storm UI / Nimbus REST API base URL
spring.opencrawling.connector.stormcrawler.topology-name	STORMCRAWLER_TOPOLOGY_NAME	opencrawling-web-crawler	Target StormCrawler topology name
spring.opencrawling.connector.stormcrawler.seeds	STORMCRAWLER_SEEDS	https://docs.example.com	Comma-separated list of seed URLs to crawl
spring.opencrawling.connector.stormcrawler.concurrency	STORMCRAWLER_CONCURRENCY	8	Number of virtual threads for parallel crawl tasks
spring.opencrawling.connector.stormcrawler.delay-ms	STORMCRAWLER_DELAY_MS	1000	Politeness delay between HTTP requests in milliseconds
spring.opencrawling.connector.stormcrawler.ignore-robots-txt	STORMCRAWLER_IGNORE_ROBOTS_TXT	false	Whether to ignore robots.txt disallow rules
spring.opencrawling.connector.stormcrawler.custom-user-agent	STORMCRAWLER_USER_AGENT	OpenCrawling-StormCrawler-Bot/1.0	Custom HTTP User-Agent header sent by crawler
spring.opencrawling.connector.stormcrawler.exclude-patterns	—	[]	Regex patterns for URLs to exclude (e.g., .*\\.(pdf|zip)$)

Example application.yml:

spring:
  opencrawling:
    connector:
      type: stormcrawler
      stormcrawler:
        nimbus-host: "storm-nimbus"
        nimbus-port: 6627
        nimbus-rest-url: "http://storm-ui:8080"
        topology-name: "opencrawling-web-crawler"
        concurrency: 8
        delay-ms: 1000
        seeds:
          - "https://docs.opencrawling.org"
          - "https://developer.opencrawling.org"
        custom-user-agent: "OpenCrawling-StormCrawler-Bot/1.0"
        exclude-patterns:
          - ".*\\.(pdf|zip|gz|exe)$"

---

2. StormCrawler Topology Configuration (crawler-conf.yaml)

Configured for OpenCrawlingBolt inside the StormCrawler topology YAML:

Property	Default	Description
opencrawling.target.endpoint	http://localhost:8080/api/v1/ingest/ois/1	Target OpenCrawling REST ingestion endpoint URL (ends with the runtime jobId determining the embedding model)
opencrawling.http.auth.header	null	Optional HTTP Authorization header (Bearer <token>, required when opencrawling.ingest.ois.token is configured on the runtime)
opencrawling.transport.mode	REST	Payload dispatcher mode: REST or MEMORY
opencrawling.emit.deletions	true	Whether to process status stream 404/410 and emit OIS DELETE tombstones
opencrawling.hash.algorithm	SHA-256	Content hashing algorithm (SHA-256 or MD5)
opencrawling.instance.id	stormcrawler-cluster-01	Cluster / crawler instance identifier stamped into document metadata

Example crawler-conf.yaml:

config:
  topology.workers: 2
  topology.message.timeout.secs: 30
  
  # OpenCrawling Bolt Configuration
  opencrawling.target.endpoint: "http://oc-runtime:8080/api/v1/ingest/ois/1" # ends with the runtime job id
  opencrawling.http.auth.header: "Bearer change-me" # oc-runtime's opencrawling.ingest.ois.token
  opencrawling.transport.mode: "REST"
  opencrawling.emit.deletions: true
  opencrawling.hash.algorithm: "SHA-256"
  opencrawling.instance.id: "storm-production-cluster-01"

---

💻 Java Topology Integration

To integrate OpenCrawlingBolt into an Apache StormCrawler topology, register the bolt in your TopologyBuilder subscribing to both the default parsed content stream and the status deletion stream:

import org.apache.storm.topology.TopologyBuilder;
import org.apache.stormcrawler.Constants;
import org.opencrawling.stormcrawler.bolt.OpenCrawlingBolt;

TopologyBuilder builder = new TopologyBuilder();

// 1. Spout & Parser Bolts
builder.setSpout("spout", new MemoryCrawlSpout());
builder.setBolt("fetcher", new FetcherBolt()).shuffleGrouping("spout");
builder.setBolt("parser", new JSoupParserBolt()).shuffleGrouping("fetcher");

// 2. OpenCrawling Bolt
OpenCrawlingBolt openCrawlingBolt = new OpenCrawlingBolt()
        .withTargetEndpoint("http://oc-runtime:8080/api/v1/ingest/ois/1")
        .withTransportMode("REST")
        .withEmitDeletions(true)
        .withInstanceId("storm-crawler-prod");

// 3. Connect content stream and status deletion stream
builder.setBolt("opencrawling-bolt", openCrawlingBolt, 2)
        .shuffleGrouping("parser")                                  // Default content stream (UPSERT)
        .shuffleGrouping("fetcher", Constants.StatusStreamName);    // Status stream 404/410 (DELETE)

---

🐳 Docker Deployment

1. Standalone Apache Storm 3.1.0 Cluster

Spin up a standalone cluster with ZooKeeper, Nimbus, Supervisor, and Storm UI:

docker compose -f oc-stormcrawler-repository-connector/docker/docker-compose.yml up -d

Access the Storm UI at: http://localhost:8088

2. Full Decoupled Pipeline with StormCrawler

Launch the complete OpenCrawling microservices stack with an Apache Storm cluster, Kafka message broker, Redis cache, PostgreSQL pgvector store, and Ollama embeddings:

docker compose -f oc-stormcrawler-repository-connector/docker/docker-compose-decoupled-with-stormcrawler.yml up -d

---

🧪 Integration Testing

OpenCrawling provides three automated test suites for the StormCrawler connector and bolt:

1. Dedicated StormCrawler Bolt Cluster Integration Test

Builds the shaded topology fat JAR, launches a distributed Apache Storm 3.1.0 cluster, deploys OpenCrawlingTestTopology via Nimbus, and verifies that opencrawling-bolt executes and processes crawl and deletion tuples:

./scripts/test-stormcrawler-bolt.sh

2. StormCrawler Connector REST Integration Test

Validates connection and topology discovery through the Apache Storm UI and Nimbus REST API:

./scripts/test-stormcrawler-connector.sh

3. Decoupled Pipeline End-to-End Test

Executes a full crawl across the decoupled microservices infrastructure:

./scripts/test-stormcrawler-decoupled.sh

All three suites are integrated into the master test runner:

./run-integration-tests.sh stormcrawler