Skip to content

Apache Incubator Proposal

Jump to bottom
Piergiorgio Lucidi edited this page Oct 24, 2026 · 2 revisions

Apache Incubator Proposal: OpenCrawling

Abstract

OpenCrawling is an open-source, enterprise-grade, high-performance data crawling, content ingestion, and security-aware vector search platform. Built on modern Java 25 (leveraging Virtual Threads and Structured Concurrency), Spring Boot 4, and Spring AI, OpenCrawling serves as the reference implementation of the Open Ingestion Standard (OIS) and provides a secure Model Context Protocol (MCP) server interface. It orchestrates scalable data flows from heterogeneous enterprise repositories and distributed web crawlers (e.g., SharePoint, S3, CMIS/Alfresco, BPMN engines like Camunda and Flowable, Apache StormCrawler, gRPC streams) to downstream vector databases and lakehouses (e.g., Apache Solr Dense Vector Search, Apache SeaTunnel for 100+ target fan-out, Apache Iceberg, Apache Ozone, Milvus, Qdrant, OpenSearch, Vespa, pgvector) with source-level Access Control List (ACL) security enforcement and end-to-end Change Data Capture (CDC) delete lifecycle management.

Proposal

The OpenCrawling community proposes to incubate OpenCrawling as a new project within the Apache Software Foundation (ASF). OpenCrawling provides a decoupled, vendor-neutral enterprise data integration framework that bridges the gap between traditional enterprise content management (ECM) repositories, web-scale crawlers, and modern Large Language Model (LLM) / Retrieval-Augmented Generation (RAG) architectures.

The project encompasses:

  1. Core Ingestion Runtime (oc-core, oc-runtime): A distributed, asynchronous engine built with virtual threads, claim-check metadata patterns, and Apache Kafka event streams.
  2. Repository & Crawler Connectors: Standardized connectors for scanning enterprise repositories and distributed web-scale crawl topologies via Apache StormCrawler (oc-stormcrawler-bolt and oc-stormcrawler-repository-connector).
  3. Downstream Vector & Data Integration Sinks: Output connectors for indexing dense vector embeddings into search engines (including native Apache Solr 10/9 Dense Vector Search) and distributed data integration bridges like Apache SeaTunnel for multi-sink fan-out across 100+ destinations.
  4. Open Ingestion Standard (OIS): The formal JSON/YAML schema specifications defining unified document payloads, ACL security SIDs, document lifecycle states (UPSERT / DELETE tombstones), and crawler job configurations.
  5. Secure Model Context Protocol (MCP) Server: A Zero-Trust context retrieval server enforcing document-level permissions (Active Directory SIDs, LDAP groups, user principals) at query time.
  6. Observability & Developer Tooling: AI-Powered Observability (AIOps) over OpenTelemetry traces, Auto-Narrativization Copilot, Java Client SDK, and Maven Archetypes for custom connector development.

Background

In enterprise AI and RAG architectures, LLM agents require seamless access to unstructured content stored across legacy and cloud repositories. However, traditional ingestion pipelines often strip out or ignore source-level security metadata (ACLs), leading to context leakage where an AI model synthesizes responses using confidential documents that the requesting user does not have permissions to view.

Furthermore, legacy crawling tools (such as Apache ManifoldCF or Apache Nutch) were architected over a decade ago prior to the emergence of vector databases, LLMs, Model Context Protocol (MCP), and modern Java features like Virtual Threads (JEP 444) and Structured Concurrency.

OpenCrawling was created to address this modern ingestion crisis by providing a native Java 25/Spring AI implementation engineered specifically for LLM search scenarios, zero-trust context retrieval, and high-throughput asynchronous processing.

Rationale

The Apache Software Foundation is the natural home for OpenCrawling. ASF has long been the global center of innovation for enterprise search, big data integration, and distributed infrastructure, hosting cornerstone projects such as Apache Lucene, Apache Solr, Apache Tika, Apache Kafka, Apache Iceberg, Apache Ozone, Apache Storm, Apache StormCrawler, Apache SeaTunnel, Apache ManifoldCF, and Apache Nutch.

Bringing OpenCrawling to the ASF offers multiple mutual benefits:

  • Ecosystem Integration: OpenCrawling directly integrates with and builds upon existing Apache projects across every layer of the ingestion and search lifecycle:
    • Apache Tika: Native integration for enterprise document parsing, OCR, and metadata extraction (supporting Apache Tika 4.x).
    • Apache Kafka: Decoupled, asynchronous event-driven streaming architecture for claim-check payloads and embedded vector chunks.
    • Apache Ozone: Distributed, enterprise-scale object store utilized for high-throughput claim-check payload offloading (oc-ozone-output-connector and Ozone claim-check store).
    • Apache Storm & Apache StormCrawler: Native web-scale distributed crawling. OpenCrawling provides oc-stormcrawler-bolt for deployment into Storm worker topologies and oc-stormcrawler-repository-connector for dynamic crawl supervision via the Nimbus REST API, propagating HTTP 404/410 status streams as OIS DELETE tombstones to eliminate stale vector embeddings.
    • Apache SeaTunnel: High-throughput distributed data integration. With oc-seatunnel-output-connector, OpenCrawling streams embedded chunks and ACLs directly to SeaTunnel Zeta, mapping OIS DELETE tombstones to RowKind.DELETE for synchronized lakehouse and vector store fan-out across 100+ downstream sinks.
    • Apache Solr: Native Dense Vector Search (oc-solr-output-connector) utilizing Solr 10 & 9.x solr.DenseVectorField, HNSW Approximate Nearest Neighbor (ANN) search, Solr 10 scalar & binary quantization, SolrCloud/ZooKeeper clustering, automated REST schema provisioning, and Zero-Trust ACL query-time filtering.
    • Apache Iceberg: Open lakehouse table format connector for long-term document archiving, vector analytics, and compliance audit reporting.
    • Apache Maven: Connector archetype distribution for rapid community extension (org.opencrawling.archetypes).
  • Vendor-Neutral Governance: Neutral governance under the Apache Way is vital to establishing OpenCrawling and OIS as industry-wide, vendor-agnostic ingestion standards.
  • Community Sustainability: Operating as an Apache project will attract a broader community of enterprise adopters, cloud providers, AI framework developers, and search engine vendors.

Initial Goals

During incubation, the OpenCrawling project will focus on the following milestones:

  1. ASF Migration & Infrastructure:

    • Transfer codebases (opencrawling, open-ingestion-standard, oc-stormcrawler-bolt, opencrawling.github.io) to Apache infrastructure (github.com/apache/incubator-opencrawling).
    • Rebrand build artifacts to org.apache.opencrawling.
    • Setup ASF-compliant CI/CD pipelines using GitHub Actions.

  2. Community & Governance:

    • Adopt the Apache Way for all decisions, roadmap discussions, and release voting.
    • Expand the contributor base across independent developers, enterprise search users, and corporate contributors.
    • Establish joint collaboration channels with the Apache Solr, Apache Storm, Apache SeaTunnel, and Apache Tika communities.

  3. Ecosystem & Connector Expansion:

    • Release additional output connectors (Elasticsearch, RESTHeart).
    • Add native integration for fine-grained authorization frameworks (e.g., OpenFGA).
    • Enhance gRPC support for high-efficiency inter-microservice communication.
    • Standardize OIS specification drafts under ASF governance.

  4. Compliance & Licensing:

    • Complete IP clearance and execute software grant agreements.
    • Ensure all third-party dependencies strictly conform to Apache License Category A policies.

Current Status

Meritocracy

The OpenCrawling project was established with meritocratic principles from day one. Design decisions, architecture changes, issue tracking, and roadmap discussions take place openly on GitHub through RFCs, Pull Requests, and public wiki pages.

Community

The OpenCrawling community includes developers and architects from enterprise search, ECM, distributed computing, and AI backgrounds. Community channels include GitHub Discussions, Slack, and social media announcements. The project actively encourages external contributions via Maven archetypes and modular connector development.

Core Developers & Initial Contributors

The initial core developers and contributors are experienced software architects and open-source veterans with extensive experience in enterprise search, content management, distributed systems, and ASF governance:

  • Piergiorgio Lucidi (piergiorgio@apache.org) – Founder, Lead Architect. ASF Member, PMC Member and Committer on multiple Apache projects (including Apache ManifoldCF and Apache Chemistry), and PMC Member of Apache Community Development.
  • Michael Cizmar (michael@michaelcizmar.com) – Lead Architect & Developer. Specialist in enterprise search and cloud infrastructure.
  • Luis Cabaceira (luis.cabaceira@texter.ai) – Lead Architect & Developer. Specialist in document processing, Natural Language Processing, and AI integration.
  • Davide Polato (davide.polato@gmail.com) – Core Engineer & Contributor. Specialist in distributed streaming topologies, Apache Storm, and Apache StormCrawler integration.
  • Maurizio Turatti (maurizio.turatti@softinstigate.com) – Strategy & Enterprise Architecture Advisor. Co-founder of SoftInstigate, co-creator of RESTHeart, and distributed systems architect with extensive background at Sun Microsystems and Alfresco.
  • Simon Hearne (simon.hearne@gmail.com) – Solution Architect at Zilliz. Contributor and maintainer of the Milvus Output Connector.

Known Risks

Orphaned Products

The risk of OpenCrawling becoming orphaned is low. The project solves an active, urgent security and performance problem in enterprise AI adoption (RAG ACL context leakage). The core maintainers are committed to its long-term evolution and actively use it in production environments.

Inexperience with Open Source

The project leadership has deep experience with open-source communities. Piergiorgio Lucidi is an active ASF Member and PMC member with over a decade of experience guiding projects through the Apache Way.

Homogenous Developers

The initial committers come from diverse geographical locations (Italy, United States, Portugal) and distinct organizations/consultancies. Incubating at Apache will further diversify the developer base by encouraging contributions from enterprise organizations, search vendors, and vector database creators.

Reliance on Third-Party Products

OpenCrawling is designed to be vendor-neutral. Core dependencies are open-source libraries under permissive licenses (Apache 2.0, MIT, BSD):

  • Spring Boot & Spring AI (Apache 2.0)
  • Apache Tika (Apache 2.0)
  • Apache Kafka (Apache 2.0)
  • Apache SolrJ (Apache 2.0)
  • Apache Storm & StormCrawler (Apache 2.0)
  • Apache SeaTunnel (Apache 2.0)
  • Apache Ozone Client (Apache 2.0)
  • Apache Iceberg SDK (Apache 2.0)
  • PostgreSQL / pgvector (PostgreSQL License / MIT)
  • Docker & OpenTelemetry (Apache 2.0)

There are no GPL/AGPL dependencies in the runtime core.

Relationship with Sponsored Products / Brand

OpenCrawling is an independent project. The name "OpenCrawling" has been used for the open source codebase. The trademark will be transferred to the Apache Software Foundation upon incubation acceptance.

Documentation & Existing Artifacts

Initial Source & Intellectual Property Submission

Initial Source Code

The initial codebase to be granted to the ASF resides in the following GitHub repositories:

  • opencrawling/opencrawling (Core engine, microservices, UI, connectors including Apache Solr and Apache SeaTunnel, MCP server)
  • opencrawling/oc-stormcrawler-bolt (Apache Storm worker bolt streaming OIS payloads and delete tombstones)
  • opencrawling/open-ingestion-standard (JSON schemas, whitepaper, specifications)
  • opencrawling/opencrawling.github.io (Project web site and documentation source)

All source code is currently licensed under the Apache License, Version 2.0.

Software Grant / ICLA / CCLA

All core contributors will submit Individual Contributor License Agreements (ICLAs) and corporate software grants will be executed upon acceptance into the Incubator.

External Dependencies

All major external dependencies of OpenCrawling use Apache-compatible licenses (Category A):

Dependency License
Java Development Kit (JDK 25) GPLv2 + Classpath Exception
Spring Boot / Spring AI Apache License 2.0
Apache Tika (4.x) Apache License 2.0
Apache Kafka Clients Apache License 2.0
Apache SolrJ (Solr 10/9 Client) Apache License 2.0
Apache Storm & StormCrawler Apache License 2.0
Apache SeaTunnel API & Engine Apache License 2.0
Apache Ozone Client Apache License 2.0
Apache Iceberg SDK Apache License 2.0
Jackson / Slf4j / Logback Apache 2.0 / MIT / EPL 1.0
Milvus / Qdrant Java SDKs Apache License 2.0
OpenTelemetry Java SDK Apache License 2.0
React / Vite / Tailwind (Admin UI) MIT

Cryptography

OpenCrawling uses standard TLS/HTTPS protocols and hashing routines provided by the standard Java Virtual Machine (JDK) and Spring Security framework for secure transport. It does not include custom cryptographic algorithms or controlled export software.

Required Resources

Mailing Lists

  • dev@opencrawling.incubator.apache.org
  • commits@opencrawling.incubator.apache.org
  • private@opencrawling.incubator.apache.org (PPMC)

Git Repositories

  • https://github.com/apache/incubator-opencrawling
  • https://github.com/apache/incubator-opencrawling-site

Issue Tracking

  • GitHub Issues on apache/incubator-opencrawling (or ASF Jira project OPENCRAWLING)

CI/CD Infrastructure

  • GitHub Actions workflows for automated build, test, multi-arch Docker image generation, and Sonar/Scorecard quality checks.

Initial Committers & PPMC Members

  • Piergiorgio Lucidi (piergiorgio@apache.org) – Initial Committer & PPMC
  • Michael Cizmar (michael@michaelcizmar.com) – Initial Committer & PPMC
  • Luis Cabaceira (luis.cabaceira@texter.ai) – Initial Committer & PPMC
  • Davide Polato (davide.polato@gmail.com) – Initial Committer & PPMC
  • Maurizio Turatti (maurizio.turatti@softinstigate.com) – Initial Committer & PPMC
  • Simon Hearne (simon.hearne@gmail.com) – Initial Committer & PPMC

(Note: Additional mentors and committers will be welcomed during the discussion period on general@incubator.apache.org.)

Champions & Mentors

  • Champion: Piergiorgio Lucidi (piergiorgio@apache.org) – ASF Member
  • Mentors:
    • (TBD - Interested ASF Members/Incubator PMC members invited to step forward during proposal discussion)

Sponsoring Entity

The Apache Incubator PMC is requested to be the sponsoring entity for this project.

🗺️ Documentation

🧪 Testing

🤝 Community

Clone this wiki locally