Repository navigation
Enterprise Multi Agent Integration
OpenCrawling acts as a Zero-Trust, identity-aware data plane for AI agents and multi-agent swarms (#203, PR #260). Operating on behalf of authenticated enterprise users, an autonomous agent or orchestrator can:
-
Discover what a source repository holds before any content is ingested (
listRepositoryConnectors,exploreRepositoryTree,queryRepositoryDryRun). -
Decide & Ingest specific folder scopes or individual documents dynamically (
triggerDynamicJob,ingestDocument,removeDeletedDocument), with custom token chunking, idempotency, and optional four-eyes human approval. -
React to Business Processes when enterprise BPM workflows (Flowable, Alfresco Process Services) reach process milestones (
POST /api/events/bpm/document-transition). -
Enforce Pre-Embedding Governance (
pii-masking, signed HTTP webhooks, customPreEmbeddingGovernanceHookbeans) before text is chunked and vectorized. -
Inspect Migrated & Lakehouse Metadata in Apache Ozone, AWS S3 (
<key>.ois.json), or Apache Iceberg (IcebergSidecarStore,#302) via companion OIS sidecar representations without downloading raw binary payloads (listSidecarStores,readOisSidecars). -
Search, Time-Travel & Retrieve indexed knowledge across all 11 target vector, search & lakehouse backends (
secureVectorSearch,getDocumentContent,listAccessibleSources, plus Apache IceberglistIcebergSnapshots,getIcebergDocumentAsOf,queryIcebergEntities,getIcebergIngestionStats).
Every operation and response is strictly filtered by what the delegating user is authorized to read.
flowchart TD
User(["👤 Enterprise User<br/>(alice@corp, groups: finance)"])
IdP["🔐 Identity Provider (IdP)<br/>OAuth 2.0 Token Exchange (RFC 8693)<br/>sub: alice | act.sub: research-orchestrator"]
Swarm["🤖 Multi-Agent Swarm<br/>(Orchestrator & Concurrent Sub-Agents)"]
User -->|"1. Delegates task"| Swarm
Swarm -->|"2. Exchanges token (act.sub)"| IdP
IdP -->|"3. Signed RS256 JWT"| Swarm
subgraph OC["OpenCrawling Identity-Aware Data Plane"]
Auth["🛡️ CallerResolver & JWT Resource Server<br/>(Ignores forged tool args; enforces OAuth2 scopes)"]
subgraph Capabilities["Agentic Capabilities (MCP & REST)"]
Disc["🔭 Repository Discovery<br/>(SourceAclVisibility)"]
Dyn["🚚 Dynamic Jobs & Single-Doc<br/>(Approval Gate & Custom Chunking)"]
Bpm["🔔 BPM Workflow Events<br/>(HMAC / Basic / oc:events:write)"]
Side["🗂️ OIS Sidecar Reader<br/>(AES-GCM Sealed Cursors)"]
Search["🔎 Vector Search<br/>(10 Search Engines)"]
end
Gov["🛡️ Pre-Embedding Governance Gate<br/>(PII Masking & Signed HTTP Webhooks)"]
Gate["🔒 Unified DocumentAccessGate<br/>(metadata | shadow | intersection | openfga)"]
Audit["📊 Lakehouse Audit Ledger<br/>(triggered_by, on_behalf_of, X-Correlation-Id)"]
end
Swarm -->|"MCP (/mcp) or REST (/api/**)"| Auth
Auth --> Disc & Dyn & Side & Search
Bpm --> Dyn
Dyn -->|"Crawl & Extract"| Gov
Gov -->|"Approved / Sanitized"| Search
Side --> Gate
Search --> Gate
Dyn & Gov --> Audit
Search (secureVectorSearch, getDocumentContent, listAccessibleSources, plus openarch_search and openarch_fetch_document) and the observability tools (fetchJobTraces, getErrorLogs, queryThroughputMetrics) are always on. Every other capability is disabled by default and can be enabled independently:
| Capability | MCP Tools | REST Endpoints | Required Scope (Default) | Enabled By |
|---|---|---|---|---|
| Search indexed content |
secureVectorSearch, getDocumentContent, listAccessibleSources
|
— | (authenticated caller) | Always enabled |
| Look before ingesting (Discovery) |
listRepositoryConnectors, exploreRepositoryTree, queryRepositoryDryRun
|
GET /api/discovery/connectorsGET /api/discovery/{connector}/treePOST /api/discovery/{connector}/dry-run
|
(authenticated caller) | opencrawling.discovery.enabled=true |
| Ingest a folder (Dynamic Jobs) |
triggerDynamicJob, getDynamicJobStatus, listDynamicJobs, stopDynamicJob
|
POST /api/jobs/dynamicGET /api/jobs/dynamicGET /api/jobs/dynamic/{jobId}POST /api/jobs/dynamic/{jobId}/stopPOST /api/jobs/dynamic/{jobId}/approvePOST /api/jobs/dynamic/{jobId}/reject
|
oc:jobs:write to start a job; status, list and stop accept oc:jobs:write or oc:jobs:run; approve and reject also require membership of admin-groups
|
opencrawling.jobs.dynamic.enabled=true |
| Ingest or remove one document |
ingestDocument, removeDeletedDocument
|
POST /api/jobs/dynamic/documents |
oc:jobs:run |
opencrawling.jobs.dynamic.enabled=true |
| React to a business process | — | POST /api/events/bpm/document-transition |
oc:events:write (or HMAC / Basic)
|
opencrawling.events.bpm.enabled=true |
| Read migrated metadata (Sidecars) |
listSidecarStores, readOisSidecars
|
GET /api/sidecarsGET /api/sidecars/{connector}
|
oc:sidecars:read |
opencrawling.sidecars.enabled=true |
Agents must authenticate with a cryptographic bearer token verified by the OpenCrawling runtime:
opencrawling:
security:
identity:
mode: jwt
protect-api: true # Enforce JWT bearer verification on REST endpoints as well
spring:
security:
oauth2:
resourceserver:
jwt:
issuer-uri: https://idp.example.com/realms/corp
audiences: opencrawling-
The Token Decides, Never the Tool Arguments: With
opencrawling.security.identity.mode=jwt, theuserPrincipalanduserRolesarguments on every MCP tool and REST endpoint are ignored. An agent that attempts to claim another user's identity or role in its tool arguments still receives only the view authorized for the token's subject. -
Strict Principal & Group Separation: The caller's user principal and group memberships are extracted from separate JWT claims (
principal-claimsvs.group-claims) and are never conflated, preserving prefix-aware matching (user:vs.group:, #224). See Model Context Protocol. -
Least-Privilege Scope Gating: OAuth2 scopes are read from the token's
scopeorscpclaim. Each capability enforces its own dedicated scope: a token carryingoc:sidecars:readcan inspect OIS sidecars but cannot trigger folder crawls (oc:jobs:write) or single-document ingestions (oc:jobs:run). -
OAuth 2.0 Token Exchange Delegation (RFC 8693):
- When an agent acts on behalf of a human user, it should present a delegated token obtained via OAuth 2.0 Token Exchange (RFC 8693).
- If the token contains an
act(actor) claim, OpenCrawling recordsact.subas the acting agent (triggered_by) and the delegating user ason_behalf_of. That is the mapped principal (the first ofjwt.principal-claimspresent,preferred_usernamebeforesubby default), not necessarily the rawsub. - Without an
actclaim, the token's principal is recorded alone.
- Multi-Agent Swarm Safety: A swarm consists of multiple concurrent sub-agents working in parallel, potentially on behalf of different users. Because identity and permissions are resolved per request directly from that request's verified bearer token, concurrent agent sessions never leak state or cross-contaminate results.
A typical autonomous agent workflow progresses through five zero-trust stages:
-
Discover (
listRepositoryConnectors→exploreRepositoryTree):- The agent enumerates discoverable connectors and walks folder hierarchies breadth-first.
- Visibility is decided per node against the permissions the source repository reports live (
SourceAclVisibility). - Folders the user cannot read are never opened or traversed, and no binary content is downloaded.
-
Preview (
queryRepositoryDryRun):- Previews how many documents in a folder or read-only query (
CMIS SQL,Alfresco AFTS) the user is permitted to read, returning representative metadata samples.
- Previews how many documents in a folder or read-only query (
-
Ingest (
triggerDynamicJob/ingestDocument):- Starts a crawl on an allow-listed connector (
opencrawling.jobs.dynamic.connectors). -
Per-Job Chunking: The agent can tailor
chunkSize,minChunkSizeChars,minChunkLengthToEmbed, andkeepSeparatorwithin administrator-configured bounds (min-chunk-size..max-chunk-size). -
Idempotency: Supplying an
idempotencyKey(Idempotency-Keyheader) guarantees that retries return the existing job rather than spawning duplicates. -
Four-Eyes Human Approval: When
require-approval: trueis enabled, jobs enterPENDING_APPROVALand wait for a member ofadmin-groups(other than the requester) to approve via REST (POST /api/jobs/dynamic/{jobId}/approve). No MCP tool can approve a job.
- Starts a crawl on an allow-listed connector (
-
Follow (
getDynamicJobStatus/listDynamicJobs/stopDynamicJob):- Users and their delegated agents see and control only their own jobs. Members of
admin-groupssee and can stop every job.stopDynamicJobhalts execution cleanly before the next document.
- Users and their delegated agents see and control only their own jobs. Members of
-
Search & Retrieve (
secureVectorSearch/getDocumentContent):- Returns only chunks authorized for the user via
DocumentAccessGate. Inopenfgamode, group membership changes take effect immediately on search without re-crawling.
- Returns only chunks authorized for the user via
Every ingested document preserves its original source repository permissions, regardless of which user or agent triggered the crawl—ensuring the vector index never grants anyone broader access than the source repository.
{"method": "tools/call", "params": {"name": "exploreRepositoryTree",
"arguments": {"connector": "Alfresco_Main", "start": "/Finance", "depth": 2}}}
{"method": "tools/call", "params": {"name": "triggerDynamicJob",
"arguments": {"connector": "Alfresco_Main", "start": "/Finance/2026", "idempotencyKey": "fin-2026-q3", "chunkSize": 400}}}See Model Context Protocol — Repository Discovery and Dynamic Jobs.
Enterprise BPM engines (Flowable and Alfresco Process Services) can push real-time document lifecycle transitions (POST /api/events/bpm/document-transition) when a workflow reaches a business milestone (e.g., "contract signed" or "record archived").
-
Administrator-Defined Routing: Matching
routesentries map(engine, processDefinitionKey, transition)to a registered connector and lifecycle action (UPSERTorDELETE). -
Multi-Mechanism Authentication: Senders authenticate via HMAC-SHA256 request signing (
X-OpenCrawling-Signature), HTTP Basic static tokens, or a JWT bearer token carryingoc:events:write. -
Per-Sender Deduplication: Event IDs are deduplicated within
dedup-retention(default24h).
See Configuration Guide — BPM Document Events and Document Lifecycle Actions & Tombstones.
When opencrawling.governance.enabled=true, every extracted document passes through an ordered chain of governance hooks inside IngestionConsumer before it is split into chunks and embedded:
-
Built-in PII Masking (
pii-masking): Detects and redacts emails, Luhn-validated payment cards, and check-digit-verified IBANs. -
External HTTP Governance Service (
http): Calls an external compliance webhook signed with HMAC-SHA256 (X-OpenCrawling-Signature) and bounded retries. -
Custom Hooks (
PreEmbeddingGovernanceHookSPI): Pluggable Spring beans returningAPPROVE,SANITIZE, orREJECT.
-
Permission Immutability: Governance hooks can never modify or widen document permissions or identity fields (
security,acl,sys_acl,security_*,acl_*,documentId,uri,lastModified). -
Transient Governance Metadata (
drop-metadata-after-gate,#414, PR#417): Metadata keys configured viaopencrawling.governance.drop-metadata-after-gate(e.g.zammad.participantsprovided for anonymization hooks) remain readable inside governance hooks and are stripped byIngestionConsumerright after the gate (or before chunking when the gate is off) so they are never persisted downstream. Protected keys cannot be dropped; outputs written directly outside the Kafka pipeline still receive the keys. -
Fail-Closed Execution: If any hook errors or times out (
UNAVAILABLE), nothing is embedded and no tombstone is emitted. The claim check is left in storage, but nothing re-processes it automatically and the Kafka message is not retried; the next crawl of the document tries again. -
Rejections Purge Stale Chunks & OpenFGA Tuples: On every
REJECTunderon-rejected-update: purge(default), aDELETEmessage for the document is published on the chunks topic to remove previously indexed chunks, and anIngestionMessage.governanceRejectionnotice is published on the documents topic soAuthorizationSyncConsumerpurges its OpenFGA relationship tuples (#268, PR #282). Seton-rejected-update: keepto leave earlier chunks in place. The OpenFGA tuples then keep the permissions of the newest crawled version, which is the rejected one. -
Direct Outputs & Decoupled Pipeline Enforcement (
#351, PR#353): With governance on, jobs that would send documents straight to an output connector—and so bypassIngestionConsumer—are refused before crawling starts (409 Conflict, releasing any per-job connector built for the start). RAG jobs on the runtime's active vector store write through the Kafka pipeline only, so they are governed and run; a job with its own Qdrant or Vespa output targeting another destination, SeaTunnel reading akafka-topicother thanopencrawling-embedded, or a direct-only RAG output (Hyland CIC, OpenArch) sends directly and is refused. Dynamic jobs, BPM events, OIS ingestion, and gRPC push publish to Kafka and pass the gate.
See Configuration Guide — Pre-Embedding Governance.
Documents migrated to Apache Ozone or AWS S3 in Migration Mode store a companion OIS metadata sidecar (<key>.ois.json) alongside each binary object. With opencrawling.sidecars.enabled=true, listSidecarStores and readOisSidecars (GET /api/sidecars/**) let agents inspect migrated document metadata, SHA-256 checksums, source lineage, and page aggregates by MIME type and source without downloading binaries:
-
Unified Gate Filtering: Each sidecar is evaluated in batches through
DocumentAccessGateon its document ID andsecurityconfiguration across all 4 modes (metadata,shadow,intersection,openfga). - Zero Permission & Count Leakage: Returned sidecar views strip all permission fields and never disclose how many sidecars were hidden.
-
AES-GCM Sealed Paging Cursors:
nextCursoris encrypted with AES-GCM and bound to the caller's tenant, principal, connector, and prefix so keys of hidden sidecars never leak.
See Model Context Protocol — OIS Sidecars and Configuration Guide — OIS Sidecars.
-
One Unified Decision Model:
-
Vector Search & OIS Sidecars pass through
DocumentAccessGate(metadata,shadow,intersection, oropenfgamode). -
Repository Discovery & Pre-Crawl Checks evaluate the permissions reported live by the source repository using
SourceAclVisibility(which appliesEffectiveAclmetadata rules), regardless of authorization mode, because OpenFGA only tracks already-indexed documents. Consequently, group memberships held exclusively in OpenFGA are not applied during discovery: a caller may see less in discovery than in search, never more.
-
Vector Search & OIS Sidecars pass through
- Source Permission Preservation: Nothing an agent does can widen a document's ACLs.
-
Strict Fail-Closed Semantics: Unverifiable callers receive
401 Unauthorized. If authorization or governance cannot reach a decision, nothing is returned or embedded. REST discovery, sidecars and dynamic jobs answer503 Service Unavailable; MCP search returns the error to the client; governance runs in the Kafka consumer, so it has no HTTP status and simply does not embed the document. -
Hard Resource & Rate Bounds:
- Discovery walks, sidecar scans, and governance webhooks enforce strict depth, node count, scan count, byte size, and timeout caps.
- Dynamic jobs enforce per-user running limits (
max-running-per-user), per-user pending approval limits (max-pending-per-user), and global concurrency caps (429 Too Many Requests).
-
Zero Information Leakage (Anti-Probing):
- Unreadable folders, out-of-scope nodes, and non-existent paths return identical responses.
- Hidden item counts are withheld by default (
reveal-hidden-counts: false). - Upstream repository and object storage exceptions are logged server-side and never echoed to callers (
502 Bad Gateway).
-
End-to-End Lakehouse & Trace Auditing:
- Every job and document execution records
triggered_by(the agentact.subor user),trigger_source(mcp,dynamic-api,bpm, and for other runsapi,ois,grpc-push,startup),on_behalf_of(the delegating user), andX-Correlation-Idacross both the job summary and every row of the columnar Apache Parquet / Avro Lakehouse Audit Ledger (see Lakehouse Audit Reporting & Ledger). - Governance decisions and hook audit attributes are recorded per document in the in-memory trace store (
TelemetryTraceStore), keyed by document id, which the observability tools read. They are not exported as OpenTelemetry spans.
- Every job and document execution records
opencrawling:
security:
identity:
mode: jwt
protect-api: true
authorization:
mode: openfga # metadata (default) | shadow | intersection | openfga
discovery:
enabled: true
jobs:
dynamic:
enabled: true
connectors: Alfresco_Main # Registered repository connectors agents may crawl
admin-groups: oc-admins
require-approval: false
events:
bpm:
enabled: false
governance:
enabled: true
hooks: pii-masking,http
sidecars:
enabled: true
connectors: Lake_S3 # Registered Ozone/S3 output connectorsEvery property and environment variable is documented in the Configuration Guide.
OpenCrawling continuously verifies multi-agent security invariants in the JVM test suite:
Generates a deterministic 400-document corpus (fixed seed 203L) covering case variants (Alice, USER:Alice), user: and group: prefixes, bare identities (#224), explicit denies, write grants, public grants, inheritance flags, empty ACLs, and missing security objects (null). The corpus is evaluated down both pipelines for 8 distinct caller profiles:
-
As Discovery Nodes: Evaluated directly through
SourceAclVisibility. -
Through the Full Ingestion & Search Pipeline: Processed through the real
IngestionConsumer, enriched viaVectorStoreWriterConsumer.ensureSecurityMetadata, serialized through a JSON round-trip (as stored in vector databases), and filtered byDocumentAccessGateinmetadatamode. -
Verified Invariants:
- Exact Set Agreement: Discovery and search return the exact same set of documents for every caller across all documents.
-
Flattened-Only Store Compatibility: Identical agreement holds for vector stores that persist only flattened permission fields (
security_allowed_read/security_denied_read) without the nestedsecurityobject. -
Pre-Filter Completeness (
security_read_keys): Proves that the vector store's native SQL/filter pre-filter (ReadKeys.forCaller) never hides a document thatDocumentAccessGateallows. -
Empty ACL & Missing Security Fail-Closed (
SecurityConfig.none(), #261, PR #262): Documents with an empty permission list or a missingsecurityobject (security == nullor omitted in JSON) default toSecurityConfig.none()(never public) acrossIngestionMessage,OzoneOutputConnector, andS3OutputConnector. With no rules,EffectiveAclevaluates the legacyaclstring alone—so a blankaclgrants no one across discovery, vector search, OpenFGA (AuthorizationSyncConsumer), and Ozone/S3.ois.jsonsidecars (acl = "public"remains public, and explicitSecurityConfig.createPublic()placeholders remain unchanged).
Executes against a live Streamable HTTP MCP server (/mcp) protected by the Spring Security OAuth2 Resource Server using RS256-signed, delegated tokens (act.sub: "research-orchestrator"):
-
Agent Tool Surface Discovery: Verifies that the tools an authenticated MCP client discovers include these 14 agent tools (
secureVectorSearch,getDocumentContent,listAccessibleSources,listRepositoryConnectors,exploreRepositoryTree,queryRepositoryDryRun,triggerDynamicJob,getDynamicJobStatus,listDynamicJobs,stopDynamicJob,ingestDocument,removeDeletedDocument,listSidecarStores,readOisSidecars). With every feature on, a pgvector deployment exposes 19 tools in total. -
Concurrent 4-Agent Swarm with Forged Arguments: Four sub-agents run concurrently across 12 rounds for two users (
aliceinfinanceandbobinhr). In every call (exploreRepositoryTree,readOisSidecars,secureVectorSearch), each agent passes the other user'suserPrincipalanduserRolesin the tool arguments. The test proves that every agent receives strictly its own token user's view and that every backend search query executes under the token's verified identity. -
Delegated Job Ownership & Isolation: Verifies that a job triggered by a delegated token records
research-orchestratorastriggeredByandaliceasonBehalfOf/owner, and that the same orchestrator acting on behalf ofbobcannot inspect, list, or re-triggeralice's folder job. -
Strict Per-Tool Scope Enforcement: Confirms that a token scoped to
oc:sidecars:readcan read OIS sidecars but fails when callingtriggerDynamicJoboringestDocument, while a token scoped tooc:jobs:writecan explore and crawl but is denied access toreadOisSidecars. - Untrusted Issuer Key Rejection: Confirms that a valid JWT signed by an unknown RSA key is rejected during MCP session initialization.
- Home
- User & Installation Guide
- Quick Start
- VM & Bare-Metal Distribution Installation
- Local Development & Fast Docker Builds
- User Guide
- Architecture
- Zero-Trust Security & ACL Management
- Security and Identity Management
- Document Authorization & OpenFGA
- Enterprise Multi-Agent Integration
- Zero-Trust Data Provider Agent & Active Ingestion
- Document Lifecycle Actions & Tombstones
- Apache Tika 4 Text Extraction
- Configuration Guide
- Native Job Scheduler
- Platform API (
/api/v1) - Platform API: Traceability
- Connectors Reference
- Alfresco Content Services (ACS) Connector
- Alfresco Process Services (APS) Connector
- Flowable Repository Connector
- Camunda Repository Connector
- OASIS CMIS Repository Connector
- Doxis Repository Connector
- M-Files Repository Connector
- SharePoint Online Repository Connector
- Google Drive Repository Connector
- Atlassian Confluence Repository Connector
- IBM FileNet P8 Repository Connector
- OpenText Documentum Repository Connector
- Zammad Repository Connector
- Hyland OpenArch Integration (Repository, Output & MCP)
- Hyland Content Innovation Cloud (CIC) Connector
- Relational Database (JDBC) Connector
- Web Repository Connector
- Apache StormCrawler Repository Connector
- gRPC Push Repository Connector
- Elasticsearch Output Connector
- Apache Solr Output Connector
- Luxir Output Connector
- Apache SeaTunnel Output Connector
- Apache Ozone Output Connector (Migration Mode)
- AWS S3 Output Connector (Migration Mode)
- Alfresco Content Services Output Connector (Migration Mode)
- Apache Iceberg Output Connector
- Doxis Output Connector
- Connector Execution Modes
- Model Context Protocol
- Apache Ozone Claim Check Store
- Lakehouse Audit Reporting & Ledger
- AI-Powered Observability (AIOps)
- Auto-Narrativization Copilot
- Maven Connector Archetypes
- Java Client SDK
- Python Client SDK
- OpenCrawling CLI
- OpenAPI Specification & Swagger UI
- gRPC Internal Transport
- Migration Guide (ManifoldCF)
- Integration Test — Decoupled Pipeline
- M-Files Integration Test
- SharePoint Online Integration Test
- Google Drive Integration Test
- Confluence Integration Test
- IBM FileNet Integration Test
- OpenText Documentum Integration Test
- Zammad Integration Test
- Hyland OpenArch Integration Test
- Hyland CIC Integration Test
- Web Crawler Test
- Elasticsearch Integration Test
- Local Docker Build Test
- Root CLI Runner Scripts Test
- Secure MCP Server Integration Test
- Narrativization Integration Test
- Reporting Integration Test
- AWS S3 Migration Integration Test
- Alfresco Migration Integration Test
- OpenFGA Authorization Integration Test
- CLI Decoupled Integration Test
- Java Client SDK Decoupled Test
- Apache Iceberg Decoupled Test
- Doxis Output Integration Test
- Platform API v1 Test Suite