Skip to content

Web Repository Connector

Piergiorgio Lucidi edited this page Nov 6, 2026 · 2 revisions

Web Repository Connector

Java Spring Boot Jsoup Crawler Commons RE2/J OIS Compliance

Module: oc-web-repository-connector
Connector Name: WebCrawler (SPI registration name)
Main Class: org.opencrawling.web.WebRepositoryConnector
Factory Class: org.opencrawling.web.WebRepositoryConnectorFactory


📌 Overview

The OpenCrawling Web Repository Connector (oc-web-repository-connector) provides an in-process, lightweight, enterprise-ready web and intranet crawler. It crawls websites from one or more seed URLs, discovers links, respects standard web politeness rules, enforces enterprise network security guards (SSRF protection), and emits normalized documents into the OpenCrawling ingestion pipeline (claim-check storage, Apache Tika text extraction, vector embedding, and search indexation).

How it relates to Apache StormCrawler

OpenCrawling offers two complementary web crawling solutions depending on the crawl volume and infrastructure requirements:

Feature Web Repository Connector (oc-web-repository-connector) Apache StormCrawler (oc-stormcrawler-repository-connector)
Architecture In-process embedded crawler running inside OpenCrawling Distributed cluster running atop Apache Storm 3.1.0
Infrastructure Zero additional infrastructure required Requires an Apache Storm Nimbus / Supervisor cluster
Ideal Use Cases Product documentation sites, corporate intranets, knowledge bases, blogs, portals (up to tens of thousands of pages per crawl) Large-scale, continuous, multi-tenant web crawling (millions of URLs)
Concurrency Java 25 Virtual Threads (Thread.ofVirtual()) Distributed Storm Worker tasks & Spouts / Bolts
Security Guards Built-in strict SSRF Guard, DNS rebinding protection, and Zero-Trust ACL modes Network-level VPC isolation

🏗️ Architecture & Processing Workflow

The connector uses Java 25 Virtual Threads to coordinate crawling workers while enforcing host-level politeness and strict backpressure towards downstream consumers:

flowchart TD
    subgraph Web_Repository_Connector["Web Repository Connector (oc-web-repository-connector)"]
        Seeds["Seed URLs / Sitemaps"] --> HostReg["HostRegistry (Shared Politeness Gate)"]
        HostReg --> Fetcher["HttpFetcher (Apache HttpClient 5)"]
        Fetcher --> SSRF["SsrfGuard (DNS Resolution & IP Filter)"]
        SSRF --> Net["Target Web Server"]
        Net --> Response["HTTP Response (HTML / PDF / Office)"]
        Response --> Parser["HtmlPageParser (Jsoup)"]
        Parser --> Links["Discovered Links (Queue)"]
        Links --> Policy["UrlPolicy & UrlNormalizer"]
        Policy --> HostReg
    end

    subgraph OpenCrawling_Pipeline["OpenCrawling Ingestion Pipeline"]
        Response --> Doc["RepositoryDocument (Reactive Flux)"]
        Doc --> ClaimCheck["Claim-Check Store (Local / Ozone / S3)"]
        ClaimCheck --> Tika["Apache Tika 4 (Isolated Fork Parser)"]
        Tika --> Embedding["Embedding Service (Ollama / OpenAI)"]
        Embedding --> VectorStore["Vector Store (PGVector / Milvus / Solr / etc.)"]
    end
Loading

1. Host Politeness & Distributed Delays (HostRegistry)

All crawl jobs in the OpenCrawling runtime share a singleton HostRegistry. It enforces a minimum delay (delayMs, default 1000ms) between consecutive HTTP requests directed to the same hostname, ensuring the crawler never overwhelms target servers even when multiple jobs run simultaneously. If a website's robots.txt specifies a higher Crawl-delay, that delay is respected up to maxCrawlDelayMs.

  • Double-Checked robots.txt Caching: Concurrent virtual threads crawling pages from the same host serialize cleanly on the host politeness gate. The first thread caches parsed rules before releasing permits, preventing duplicate robots.txt HTTP requests.
  • Instant Suspended Host Drainage: When a host encounters consecutive network or HTTP 5xx errors exceeding hostErrorThreshold (default 20), the host is suspended for the crawl run. All remaining queued URLs for that host are drained immediately in $O(1)$ without waiting for politeness delays ($N \times \text{delayMs}$), allowing the job to finish without idle waiting. Any subsequent links discovered for the suspended host are dropped immediately upon enqueue.

2. RFC 9309 Robots.txt & Sitemap Compliance

Using crawler-commons 1.5, robots.txt rules are retrieved and cached per host:

  • Evaluates Disallow and Allow rules according to RFC 9309.
  • Automatically discovers sitemaps from Sitemap: directives in robots.txt or parses explicit sitemapUrls.
  • Honors page-level <meta name="robots" content="noindex, nofollow"> directives and X-Robots-Tag HTTP response headers.

3. Spider Trap Prevention & URL Filtering

To prevent infinite loops, calendar traps, and session-ID cycles:

  • Depth Limiting: maxDepth restricts link distance from seeds (default 3, 0 to 50).
  • Spider Trap Guards: Automatically rejects URLs exceeding maxUrlLength (default 2048), query parameter counts exceeding maxQueryParams (default 32), or path structures exceeding maxRepeatedSegments (default 3).
  • Linear-Time Regex Matching: includePatterns and excludePatterns use Google's RE2/J regular expression engine to guarantee linear execution time and eliminate Regular Expression Denial of Service (ReDoS) vulnerabilities.
  • URL Normalization: Strips URL fragments (#...), normalizes default HTTP/HTTPS ports, lowercases hostnames, and strips tracking parameters (utm_*, fbclid, gclid, msclkid, jsessionid, phpsessid, sid, and user-defined stripQueryParams).
  • Canonical URLs & Deduplication: Follows <link rel="canonical"> and computes cryptographic SHA-256 content hashes to discard redundant pages.

4. Enterprise SSRF Guard (SsrfGuard)

The connector incorporates an enterprise-grade Server-Side Request Forgery (SsrfGuard) defense mechanism operating directly at the HTTP client DNS resolution layer:

  • Blocked Ranges: By default, blocks all connections to loopback (127.0.0.0/8, ::1), RFC 1918 private subnets (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16), link-local addresses (169.254.0.0/16, fe80::/10), cloud instance metadata endpoints (169.254.169.254), RFC 8215 local-use NAT64 translation prefix (64:ff9b:1::/48), and IPv6 unique-local networks (fc00::/7).
  • IPv4-Mapped IPv6 Defense-in-Depth: Automatically unwraps IPv4-mapped IPv6 addresses (::ffff:0:0/96, e.g. ::ffff:127.0.0.1 or ::ffff:10.0.0.1) and RFC 6052 NAT64 (64:ff9b::/96) at the DNS level, evaluating the embedded IPv4 address directly against IPv4 CIDR rules to prevent intranet bypasses in custom resolver environments.
  • DNS Rebinding Protection: Re-validates every IP address resolved by DNS on initial requests and on every subsequent HTTP redirect hop.
  • Intranet Crawling Support: Internal intranets can be safely crawled by enabling allowPrivateNetworks: true combined with an explicit privateAllowlist specifying permitted hostnames (e.g. intranet.company.com, .corp.local) or CIDR ranges (10.20.0.0/16).

5. Zero-Trust Security & ACL Modes

  • PUBLIC Mode: Applies the public read key (security_read_keys: [public]). When allowPrivateNetworks is enabled, setting aclMode: public requires explicit confirmation via confirmPublicExposure: true to prevent accidental public indexing of internal systems.
  • STATIC Mode: Assigns static security identities (aclIdentity) to all ingested pages, enabling role-based or group-based access control (e.g., role:engineering, group:internal-docs).

6. Reactive Streaming with Dynamic Backpressure

The connector emits a reactive Flux<RepositoryDocument>. Crawling virtual threads use a thread-safe Semaphore demand gate (sink.onRequest(...)), ensuring the crawler pauses fetching whenever downstream components (Tika extraction, AI embedding, vector writing) are saturated.


⚙️ Configuration Reference

Settings are resolved in the following priority order (first match wins):

  1. Job crawlSpec (JSON payload in POST /api/jobs)
  2. Connector configuration (Admin UI connector form or connectors.json)
  3. Spring Boot property (spring.opencrawling.connector.web.<kebab-case-name>)
  4. Built-in default

Configuration Properties Table

Setting Spring Property Default Description
seeds spring.opencrawling.connector.web.seeds (Job path) Start URLs (space- or newline-separated in job path)
maxDepth spring.opencrawling.connector.web.max-depth 3 Maximum link hops from seed URLs (0 to 50)
maxPages spring.opencrawling.connector.web.max-pages 1000 Maximum page requests executed per crawl run
delayMs spring.opencrawling.connector.web.delay-ms 1000 Minimum wait time (ms) between requests to the same host
maxCrawlDelayMs spring.opencrawling.connector.web.max-crawl-delay-ms 30000 Maximum politeness delay to honor if robots.txt specifies a high Crawl-delay
scope spring.opencrawling.connector.web.scope DOMAIN (if stayOnDomain=true) Crawl boundary: HOST, DOMAIN, PREFIX (directory path), or ANY
stayOnDomain spring.opencrawling.connector.web.stay-on-domain true Restricts crawling to seeds' registrable domain (including subdomains)
userAgent spring.opencrawling.connector.web.user-agent OpenCrawlingBot/1.0 (+https://opencrawling.github.io) HTTP User-Agent header (must contain a valid URL or email)
robotsAgentName spring.opencrawling.connector.web.robots-agent-name derived from userAgent Custom product token matched against User-agent: lines in robots.txt
respectRobots spring.opencrawling.connector.web.respect-robots true When false, bypasses robots.txt (only for owned sites; logs warnings)
sitemapUrls spring.opencrawling.connector.web.sitemap-urls none Explicit list of sitemap URLs to parse
discoverSitemaps spring.opencrawling.connector.web.discover-sitemaps true Automatically parse Sitemap: lines declared in robots.txt
includePatterns spring.opencrawling.connector.web.include-patterns none RE2/J regex list; URLs must match at least one to be indexed
excludePatterns spring.opencrawling.connector.web.exclude-patterns none RE2/J regex list; matching URLs are rejected (takes precedence over includes)
stripQueryParams spring.opencrawling.connector.web.strip-query-params utm_*, fbclid, gclid, msclkid, *sessid* Additional query parameters to remove during URL normalization
maxContentBytes spring.opencrawling.connector.web.max-content-bytes 52428800 (50 MB) Maximum allowed response body size; larger responses are skipped
acceptMimeTypes spring.opencrawling.connector.web.accept-mime-types HTML, text, Markdown, PDF, Office, OpenDocument, RTF Allowed MIME types for indexing
maxThreads spring.opencrawling.connector.web.max-threads 16 Number of concurrent virtual threads allocated per crawl
connectTimeoutMs spring.opencrawling.connector.web.connect-timeout-ms 10000 (10s) HTTP socket connection timeout
readTimeoutMs spring.opencrawling.connector.web.read-timeout-ms 30000 (30s) HTTP socket read timeout
maxRedirects spring.opencrawling.connector.web.max-redirects 5 Maximum redirect hops followed per URL
maxRetries spring.opencrawling.connector.web.max-retries 2 Retry attempts for transient 5xx, 429, or network errors
allowPrivateNetworks spring.opencrawling.connector.web.allow-private-networks false When true, permits crawling private/intranet hosts
privateAllowlist spring.opencrawling.connector.web.private-allowlist none Permitted hostnames (intranet.corp.com, .corp.local) or CIDRs (10.0.0.0/8)
aclMode spring.opencrawling.connector.web.acl-mode PUBLIC Document access control mode: PUBLIC or STATIC
aclIdentity spring.opencrawling.connector.web.acl-identity none Identity key (user/group) attached when aclMode=STATIC
maxUrlLength spring.opencrawling.connector.web.max-url-length 2048 Maximum URL length before triggering spider trap rejection
maxQueryParams spring.opencrawling.connector.web.max-query-params 32 Maximum query parameters before triggering spider trap rejection
maxRepeatedSegments spring.opencrawling.connector.web.max-repeated-segments 3 Maximum repeated directory path segments (e.g. /a/b/a/b/a/b)
hostErrorThreshold spring.opencrawling.connector.web.host-error-threshold 20 Consecutive host errors before aborting requests to that host

📄 Ingested Document Metadata

Each crawled page emits an OpenCrawling RepositoryDocument with the following metadata:

Metadata Field Type Description Example
name String Filename extracted from path or host features.html
mimeType String Response Content-Type (clean MIME) text/html, application/pdf
source String Ingestion source identifier web
title String HTML <title> or document title OpenCrawling Documentation
description String <meta name="description"> content Enterprise Data Ingestion Framework
language String <html lang="..."> or Content-Language en
keywords String <meta name="keywords"> content search, rag, crawler
web_host String Fully qualified domain name docs.opencrawling.org
web_canonical_url String Resolved canonical URL https://docs.opencrawling.org/intro
web_depth Integer Link hop distance from seed 1
web_referrer String URL of the page that provided the link https://docs.opencrawling.org/
web_seed String Seed URL that initiated this discovery https://docs.opencrawling.org/
web_anchor_text String Anchor text on referring link Getting Started Guide
http_status Integer HTTP response status code 200
http_server String Server response header nginx/1.24
http_etag String HTTP ETag header W/"65432-1"
http_last_modified String Last-Modified header Wed, 21 Oct 2026 07:28:00 GMT
http_content_length Long Content length in bytes 45230
web_content_hash String SHA-256 hash of response body a3b7...
web_fetched_at Long Epoch milliseconds when fetched 1762356000000
jobId String OpenCrawling Job Identifier job-web-docs-01

Note

The document ID is deterministically computed as web:<sha256(jobId:normalizedUrl)>. This ensures document IDs remain stable across crawl executions, enabling accurate deduplication and downstream updates.


💡 Job Configuration Examples

1. Public Documentation Crawl (PGVector Destination)

Crawls an official documentation portal up to 4 hops deep, restricting links to /docs/ and discarding images and binaries:

{
  "name": "OpenCrawling Documentation Crawl",
  "repositoryConnector": "WebCrawler",
  "outputConnector": "PGVector_Output",
  "path": "https://docs.opencrawling.org/",
  "crawlSpec": {
    "maxDepth": 4,
    "maxPages": 5000,
    "delayMs": 1000,
    "scope": "PREFIX",
    "includePatterns": [
      "^/docs/"
    ],
    "excludePatterns": [
      "\\.(png|jpe?g|gif|svg|webp|css|js|woff2?)$",
      "/print/"
    ],
    "discoverSitemaps": true
  }
}

2. Secure Internal Intranet Crawl (Static ACLs)

Crawls a private enterprise portal behind RFC 1918 addressing, tagging all documents with an internal employee group security key:

{
  "name": "Internal Engineering Portal",
  "repositoryConnector": "WebCrawler",
  "outputConnector": "PGVector_Output",
  "path": "https://wiki.corp.internal/",
  "crawlSpec": {
    "maxDepth": 5,
    "maxPages": 10000,
    "allowPrivateNetworks": true,
    "privateAllowlist": [
      "wiki.corp.internal",
      "10.50.0.0/16"
    ],
    "aclMode": "STATIC",
    "aclIdentity": "group:engineering-staff",
    "maxThreads": 8,
    "delayMs": 500
  }
}

🧪 Testing & Verification

The connector comes with comprehensive unit and integration test coverage:

mvn test -pl oc-web-repository-connector

Key test suites:

  • WebRepositoryConnectorTest: Spins up an embedded JDK HTTP server (127.0.0.1) verifying robots.txt parsing, sitemaps, depth boundaries, redirect loops, canonical URL handling, duplicate content skips, noindex/nofollow, and host politeness delay sharing across concurrent crawl jobs.
  • SsrfGuardTest: Verifies comprehensive rejection of loopback, private IPv4/IPv6, cloud metadata (169.254.169.254), and DNS rebinding attacks while honoring allowlist rules.
  • UrlNormalizerTest: Verifies tracking parameter removal, case normalization, port stripping, and path canonicalization.
  • UrlPolicyTest: Verifies spider trap detection, query parameter caps, and RE2/J pattern evaluation.

🗺️ Documentation


🎓 Tutorials


🧪 Testing


🤝 Community

Clone this wiki locally