Repository navigation
Web Repository Connector
Module: oc-web-repository-connector
Connector Name: WebCrawler (SPI registration name)
Main Class: org.opencrawling.web.WebRepositoryConnector
Factory Class: org.opencrawling.web.WebRepositoryConnectorFactory
The OpenCrawling Web Repository Connector (oc-web-repository-connector) provides an in-process, lightweight, enterprise-ready web and intranet crawler. It crawls websites from one or more seed URLs, discovers links, respects standard web politeness rules, enforces enterprise network security guards (SSRF protection), and emits normalized documents into the OpenCrawling ingestion pipeline (claim-check storage, Apache Tika text extraction, vector embedding, and search indexation).
OpenCrawling offers two complementary web crawling solutions depending on the crawl volume and infrastructure requirements:
| Feature | Web Repository Connector (oc-web-repository-connector) |
Apache StormCrawler (oc-stormcrawler-repository-connector) |
|---|---|---|
| Architecture | In-process embedded crawler running inside OpenCrawling | Distributed cluster running atop Apache Storm 3.1.0 |
| Infrastructure | Zero additional infrastructure required | Requires an Apache Storm Nimbus / Supervisor cluster |
| Ideal Use Cases | Product documentation sites, corporate intranets, knowledge bases, blogs, portals (up to tens of thousands of pages per crawl) | Large-scale, continuous, multi-tenant web crawling (millions of URLs) |
| Concurrency |
Java 25 Virtual Threads (Thread.ofVirtual()) |
Distributed Storm Worker tasks & Spouts / Bolts |
| Security Guards | Built-in strict SSRF Guard, DNS rebinding protection, and Zero-Trust ACL modes | Network-level VPC isolation |
The connector uses Java 25 Virtual Threads to coordinate crawling workers while enforcing host-level politeness and strict backpressure towards downstream consumers:
flowchart TD
subgraph Web_Repository_Connector["Web Repository Connector (oc-web-repository-connector)"]
Seeds["Seed URLs / Sitemaps"] --> HostReg["HostRegistry (Shared Politeness Gate)"]
HostReg --> Fetcher["HttpFetcher (Apache HttpClient 5)"]
Fetcher --> SSRF["SsrfGuard (DNS Resolution & IP Filter)"]
SSRF --> Net["Target Web Server"]
Net --> Response["HTTP Response (HTML / PDF / Office)"]
Response --> Parser["HtmlPageParser (Jsoup)"]
Parser --> Links["Discovered Links (Queue)"]
Links --> Policy["UrlPolicy & UrlNormalizer"]
Policy --> HostReg
end
subgraph OpenCrawling_Pipeline["OpenCrawling Ingestion Pipeline"]
Response --> Doc["RepositoryDocument (Reactive Flux)"]
Doc --> ClaimCheck["Claim-Check Store (Local / Ozone / S3)"]
ClaimCheck --> Tika["Apache Tika 4 (Isolated Fork Parser)"]
Tika --> Embedding["Embedding Service (Ollama / OpenAI)"]
Embedding --> VectorStore["Vector Store (PGVector / Milvus / Solr / etc.)"]
end
All crawl jobs in the OpenCrawling runtime share a singleton HostRegistry. It enforces a minimum delay (delayMs, default 1000ms) between consecutive HTTP requests directed to the same hostname, ensuring the crawler never overwhelms target servers even when multiple jobs run simultaneously. If a website's robots.txt specifies a higher Crawl-delay, that delay is respected up to maxCrawlDelayMs.
-
Double-Checked
robots.txtCaching: Concurrent virtual threads crawling pages from the same host serialize cleanly on the host politeness gate. The first thread caches parsed rules before releasing permits, preventing duplicaterobots.txtHTTP requests. -
Instant Suspended Host Drainage: When a host encounters consecutive network or HTTP 5xx errors exceeding
hostErrorThreshold(default20), the host is suspended for the crawl run. All remaining queued URLs for that host are drained immediately in$O(1)$ without waiting for politeness delays ($N \times \text{delayMs}$ ), allowing the job to finish without idle waiting. Any subsequent links discovered for the suspended host are dropped immediately upon enqueue.
Using crawler-commons 1.5, robots.txt rules are retrieved and cached per host:
- Evaluates
DisallowandAllowrules according to RFC 9309. - Automatically discovers sitemaps from
Sitemap:directives inrobots.txtor parses explicitsitemapUrls. - Honors page-level
<meta name="robots" content="noindex, nofollow">directives andX-Robots-TagHTTP response headers.
To prevent infinite loops, calendar traps, and session-ID cycles:
-
Depth Limiting:
maxDepthrestricts link distance from seeds (default3, 0 to 50). -
Spider Trap Guards: Automatically rejects URLs exceeding
maxUrlLength(default2048), query parameter counts exceedingmaxQueryParams(default32), or path structures exceedingmaxRepeatedSegments(default3). -
Linear-Time Regex Matching:
includePatternsandexcludePatternsuse Google's RE2/J regular expression engine to guarantee linear execution time and eliminate Regular Expression Denial of Service (ReDoS) vulnerabilities. -
URL Normalization: Strips URL fragments (
#...), normalizes default HTTP/HTTPS ports, lowercases hostnames, and strips tracking parameters (utm_*,fbclid,gclid,msclkid,jsessionid,phpsessid,sid, and user-definedstripQueryParams). -
Canonical URLs & Deduplication: Follows
<link rel="canonical">and computes cryptographic SHA-256 content hashes to discard redundant pages.
The connector incorporates an enterprise-grade Server-Side Request Forgery (SsrfGuard) defense mechanism operating directly at the HTTP client DNS resolution layer:
-
Blocked Ranges: By default, blocks all connections to loopback (
127.0.0.0/8,::1), RFC 1918 private subnets (10.0.0.0/8,172.16.0.0/12,192.168.0.0/16), link-local addresses (169.254.0.0/16,fe80::/10), cloud instance metadata endpoints (169.254.169.254), RFC 8215 local-use NAT64 translation prefix (64:ff9b:1::/48), and IPv6 unique-local networks (fc00::/7). -
IPv4-Mapped IPv6 Defense-in-Depth: Automatically unwraps IPv4-mapped IPv6 addresses (
::ffff:0:0/96, e.g.::ffff:127.0.0.1or::ffff:10.0.0.1) and RFC 6052 NAT64 (64:ff9b::/96) at the DNS level, evaluating the embedded IPv4 address directly against IPv4 CIDR rules to prevent intranet bypasses in custom resolver environments. - DNS Rebinding Protection: Re-validates every IP address resolved by DNS on initial requests and on every subsequent HTTP redirect hop.
-
Intranet Crawling Support: Internal intranets can be safely crawled by enabling
allowPrivateNetworks: truecombined with an explicitprivateAllowlistspecifying permitted hostnames (e.g.intranet.company.com,.corp.local) or CIDR ranges (10.20.0.0/16).
-
PUBLICMode: Applies thepublicread key (security_read_keys: [public]). WhenallowPrivateNetworksis enabled, settingaclMode: publicrequires explicit confirmation viaconfirmPublicExposure: trueto prevent accidental public indexing of internal systems. -
STATICMode: Assigns static security identities (aclIdentity) to all ingested pages, enabling role-based or group-based access control (e.g.,role:engineering,group:internal-docs).
The connector emits a reactive Flux<RepositoryDocument>. Crawling virtual threads use a thread-safe Semaphore demand gate (sink.onRequest(...)), ensuring the crawler pauses fetching whenever downstream components (Tika extraction, AI embedding, vector writing) are saturated.
Settings are resolved in the following priority order (first match wins):
- Job
crawlSpec(JSON payload inPOST /api/jobs) - Connector configuration (Admin UI connector form or
connectors.json) - Spring Boot property (
spring.opencrawling.connector.web.<kebab-case-name>) - Built-in default
| Setting | Spring Property | Default | Description |
|---|---|---|---|
seeds |
spring.opencrawling.connector.web.seeds |
(Job path) | Start URLs (space- or newline-separated in job path) |
maxDepth |
spring.opencrawling.connector.web.max-depth |
3 |
Maximum link hops from seed URLs (0 to 50) |
maxPages |
spring.opencrawling.connector.web.max-pages |
1000 |
Maximum page requests executed per crawl run |
delayMs |
spring.opencrawling.connector.web.delay-ms |
1000 |
Minimum wait time (ms) between requests to the same host |
maxCrawlDelayMs |
spring.opencrawling.connector.web.max-crawl-delay-ms |
30000 |
Maximum politeness delay to honor if robots.txt specifies a high Crawl-delay
|
scope |
spring.opencrawling.connector.web.scope |
DOMAIN (if stayOnDomain=true) |
Crawl boundary: HOST, DOMAIN, PREFIX (directory path), or ANY
|
stayOnDomain |
spring.opencrawling.connector.web.stay-on-domain |
true |
Restricts crawling to seeds' registrable domain (including subdomains) |
userAgent |
spring.opencrawling.connector.web.user-agent |
OpenCrawlingBot/1.0 (+https://opencrawling.github.io) |
HTTP User-Agent header (must contain a valid URL or email) |
robotsAgentName |
spring.opencrawling.connector.web.robots-agent-name |
derived from userAgent
|
Custom product token matched against User-agent: lines in robots.txt
|
respectRobots |
spring.opencrawling.connector.web.respect-robots |
true |
When false, bypasses robots.txt (only for owned sites; logs warnings) |
sitemapUrls |
spring.opencrawling.connector.web.sitemap-urls |
none | Explicit list of sitemap URLs to parse |
discoverSitemaps |
spring.opencrawling.connector.web.discover-sitemaps |
true |
Automatically parse Sitemap: lines declared in robots.txt
|
includePatterns |
spring.opencrawling.connector.web.include-patterns |
none | RE2/J regex list; URLs must match at least one to be indexed |
excludePatterns |
spring.opencrawling.connector.web.exclude-patterns |
none | RE2/J regex list; matching URLs are rejected (takes precedence over includes) |
stripQueryParams |
spring.opencrawling.connector.web.strip-query-params |
utm_*, fbclid, gclid, msclkid, *sessid*
|
Additional query parameters to remove during URL normalization |
maxContentBytes |
spring.opencrawling.connector.web.max-content-bytes |
52428800 (50 MB) |
Maximum allowed response body size; larger responses are skipped |
acceptMimeTypes |
spring.opencrawling.connector.web.accept-mime-types |
HTML, text, Markdown, PDF, Office, OpenDocument, RTF | Allowed MIME types for indexing |
maxThreads |
spring.opencrawling.connector.web.max-threads |
16 |
Number of concurrent virtual threads allocated per crawl |
connectTimeoutMs |
spring.opencrawling.connector.web.connect-timeout-ms |
10000 (10s) |
HTTP socket connection timeout |
readTimeoutMs |
spring.opencrawling.connector.web.read-timeout-ms |
30000 (30s) |
HTTP socket read timeout |
maxRedirects |
spring.opencrawling.connector.web.max-redirects |
5 |
Maximum redirect hops followed per URL |
maxRetries |
spring.opencrawling.connector.web.max-retries |
2 |
Retry attempts for transient 5xx, 429, or network errors |
allowPrivateNetworks |
spring.opencrawling.connector.web.allow-private-networks |
false |
When true, permits crawling private/intranet hosts |
privateAllowlist |
spring.opencrawling.connector.web.private-allowlist |
none | Permitted hostnames (intranet.corp.com, .corp.local) or CIDRs (10.0.0.0/8) |
aclMode |
spring.opencrawling.connector.web.acl-mode |
PUBLIC |
Document access control mode: PUBLIC or STATIC
|
aclIdentity |
spring.opencrawling.connector.web.acl-identity |
none | Identity key (user/group) attached when aclMode=STATIC
|
maxUrlLength |
spring.opencrawling.connector.web.max-url-length |
2048 |
Maximum URL length before triggering spider trap rejection |
maxQueryParams |
spring.opencrawling.connector.web.max-query-params |
32 |
Maximum query parameters before triggering spider trap rejection |
maxRepeatedSegments |
spring.opencrawling.connector.web.max-repeated-segments |
3 |
Maximum repeated directory path segments (e.g. /a/b/a/b/a/b) |
hostErrorThreshold |
spring.opencrawling.connector.web.host-error-threshold |
20 |
Consecutive host errors before aborting requests to that host |
Each crawled page emits an OpenCrawling RepositoryDocument with the following metadata:
| Metadata Field | Type | Description | Example |
|---|---|---|---|
name |
String | Filename extracted from path or host | features.html |
mimeType |
String | Response Content-Type (clean MIME) |
text/html, application/pdf
|
source |
String | Ingestion source identifier | web |
title |
String | HTML <title> or document title |
OpenCrawling Documentation |
description |
String |
<meta name="description"> content |
Enterprise Data Ingestion Framework |
language |
String |
<html lang="..."> or Content-Language |
en |
keywords |
String |
<meta name="keywords"> content |
search, rag, crawler |
web_host |
String | Fully qualified domain name | docs.opencrawling.org |
web_canonical_url |
String | Resolved canonical URL | https://docs.opencrawling.org/intro |
web_depth |
Integer | Link hop distance from seed | 1 |
web_referrer |
String | URL of the page that provided the link | https://docs.opencrawling.org/ |
web_seed |
String | Seed URL that initiated this discovery | https://docs.opencrawling.org/ |
web_anchor_text |
String | Anchor text on referring link | Getting Started Guide |
http_status |
Integer | HTTP response status code | 200 |
http_server |
String |
Server response header |
nginx/1.24 |
http_etag |
String | HTTP ETag header | W/"65432-1" |
http_last_modified |
String |
Last-Modified header |
Wed, 21 Oct 2026 07:28:00 GMT |
http_content_length |
Long | Content length in bytes | 45230 |
web_content_hash |
String | SHA-256 hash of response body | a3b7... |
web_fetched_at |
Long | Epoch milliseconds when fetched | 1762356000000 |
jobId |
String | OpenCrawling Job Identifier | job-web-docs-01 |
Note
The document ID is deterministically computed as web:<sha256(jobId:normalizedUrl)>. This ensures document IDs remain stable across crawl executions, enabling accurate deduplication and downstream updates.
Crawls an official documentation portal up to 4 hops deep, restricting links to /docs/ and discarding images and binaries:
{
"name": "OpenCrawling Documentation Crawl",
"repositoryConnector": "WebCrawler",
"outputConnector": "PGVector_Output",
"path": "https://docs.opencrawling.org/",
"crawlSpec": {
"maxDepth": 4,
"maxPages": 5000,
"delayMs": 1000,
"scope": "PREFIX",
"includePatterns": [
"^/docs/"
],
"excludePatterns": [
"\\.(png|jpe?g|gif|svg|webp|css|js|woff2?)$",
"/print/"
],
"discoverSitemaps": true
}
}Crawls a private enterprise portal behind RFC 1918 addressing, tagging all documents with an internal employee group security key:
{
"name": "Internal Engineering Portal",
"repositoryConnector": "WebCrawler",
"outputConnector": "PGVector_Output",
"path": "https://wiki.corp.internal/",
"crawlSpec": {
"maxDepth": 5,
"maxPages": 10000,
"allowPrivateNetworks": true,
"privateAllowlist": [
"wiki.corp.internal",
"10.50.0.0/16"
],
"aclMode": "STATIC",
"aclIdentity": "group:engineering-staff",
"maxThreads": 8,
"delayMs": 500
}
}The connector comes with comprehensive unit and integration test coverage:
mvn test -pl oc-web-repository-connectorKey test suites:
-
WebRepositoryConnectorTest: Spins up an embedded JDK HTTP server (127.0.0.1) verifying robots.txt parsing, sitemaps, depth boundaries, redirect loops, canonical URL handling, duplicate content skips,noindex/nofollow, and host politeness delay sharing across concurrent crawl jobs. -
SsrfGuardTest: Verifies comprehensive rejection of loopback, private IPv4/IPv6, cloud metadata (169.254.169.254), and DNS rebinding attacks while honoring allowlist rules. -
UrlNormalizerTest: Verifies tracking parameter removal, case normalization, port stripping, and path canonicalization. -
UrlPolicyTest: Verifies spider trap detection, query parameter caps, and RE2/J pattern evaluation.
- Home
- User & Installation Guide
- Quick Start
- VM & Bare-Metal Distribution Installation
- Local Development & Fast Docker Builds
- User Guide
- Architecture
- Zero-Trust Security & ACL Management
- Security and Identity Management
- Document Authorization & OpenFGA
- Enterprise Multi-Agent Integration
- Zero-Trust Data Provider Agent & Active Ingestion
- Document Lifecycle Actions & Tombstones
- Apache Tika 4 Text Extraction
- Configuration Guide
- Native Job Scheduler
- Platform API (
/api/v1) - Platform API: Traceability
- Connectors Reference
- Alfresco Content Services (ACS) Connector
- Alfresco Process Services (APS) Connector
- Flowable Repository Connector
- Camunda Repository Connector
- OASIS CMIS Repository Connector
- Doxis Repository Connector
- M-Files Repository Connector
- SharePoint Online Repository Connector
- Google Drive Repository Connector
- Atlassian Confluence Repository Connector
- IBM FileNet P8 Repository Connector
- OpenText Documentum Repository Connector
- Zammad Repository Connector
- Hyland OpenArch Integration (Repository, Output & MCP)
- Hyland Content Innovation Cloud (CIC) Connector
- Relational Database (JDBC) Connector
- Web Repository Connector
- Apache StormCrawler Repository Connector
- gRPC Push Repository Connector
- Elasticsearch Output Connector
- Apache Solr Output Connector
- Luxir Output Connector
- Apache SeaTunnel Output Connector
- Apache Ozone Output Connector (Migration Mode)
- AWS S3 Output Connector (Migration Mode)
- Alfresco Content Services Output Connector (Migration Mode)
- Apache Iceberg Output Connector
- Doxis Output Connector
- Connector Execution Modes
- Model Context Protocol
- Apache Ozone Claim Check Store
- Lakehouse Audit Reporting & Ledger
- AI-Powered Observability (AIOps)
- Auto-Narrativization Copilot
- Maven Connector Archetypes
- Java Client SDK
- Python Client SDK
- OpenCrawling CLI
- OpenAPI Specification & Swagger UI
- gRPC Internal Transport
- Migration Guide (ManifoldCF)
- Integration Test — Decoupled Pipeline
- M-Files Integration Test
- SharePoint Online Integration Test
- Google Drive Integration Test
- Confluence Integration Test
- IBM FileNet Integration Test
- OpenText Documentum Integration Test
- Zammad Integration Test
- Hyland OpenArch Integration Test
- Hyland CIC Integration Test
- Web Crawler Test
- Elasticsearch Integration Test
- Local Docker Build Test
- Root CLI Runner Scripts Test
- Secure MCP Server Integration Test
- Narrativization Integration Test
- Reporting Integration Test
- AWS S3 Migration Integration Test
- Alfresco Migration Integration Test
- OpenFGA Authorization Integration Test
- CLI Decoupled Integration Test
- Java Client SDK Decoupled Test
- Apache Iceberg Decoupled Test
- Doxis Output Integration Test
- Platform API v1 Test Suite