-
Notifications
You must be signed in to change notification settings - Fork 1
Home
Piergiorgio Lucidi edited this page Jul 21, 2026
·
4 revisions
OpenCrawling is a high-performance, enterprise-grade data ingestion, federation, and security mapping framework. It bridges legacy or enterprise content repositories to downstream AI, vector search, and Retrieval-Augmented Generation (RAG) infrastructure.
Use the sidebar or the quick links below to navigate the documentation:
- Home: Overview and core concepts.
- Architecture: Microservices layout, event streams, and the Claim Check Pattern.
- Configuration Guide: Setting up repository, transformation, and output connectors.
- Apache Ozone Claim Check Store: Scalable object storage pattern using Apache Ozone.
- Model Context Protocol: Running and query-authorizing the Secure MCP Server.
- AI-Powered Observability (AIOps): OpenTelemetry correlated spans and AI Root Cause Analysis (RCA).
Unlike standard scrapers, OpenCrawling is designed from the ground up for the security and volume demands of enterprise environments:
- 🔒 Security First (ACL Mapping): Automatically synchronizes document Access Control Lists (ACLs) containing Security SIDs. AI applications and vector queries respect native document rights, ensuring users only see search results they are authorized to access.
- ⚡ Asynchronous Scale (Decoupled Microservices): Ingestion tasks run as separate components communicating asynchronously via Apache Kafka. Heavy deep-learning embeddings or text extractions can be scaled horizontally without blocking crawlers.
- 🚀 Modern JVM Performance: Built with Java 25 and Spring Boot, utilizing Virtual Threads (Project Loom) and Structured Concurrency for massive, non-blocking network I/O throughput.