Skip to content

Enterprise Multi Agent Integration

Piergiorgio Lucidi edited this page Nov 20, 2026 · 9 revisions

🤖 Enterprise Multi-Agent Integration

OpenCrawling acts as a Zero-Trust, identity-aware data plane for AI agents and multi-agent swarms (#203, PR #260). Operating on behalf of authenticated enterprise users, an autonomous agent or orchestrator can:

  1. Discover what a source repository holds before any content is ingested (listRepositoryConnectors, exploreRepositoryTree, queryRepositoryDryRun).
  2. Decide & Ingest specific folder scopes or individual documents dynamically (triggerDynamicJob, ingestDocument, removeDeletedDocument), with custom token chunking, idempotency, and optional four-eyes human approval.
  3. React to Business Processes when enterprise BPM workflows (Flowable, Alfresco Process Services) reach process milestones (POST /api/events/bpm/document-transition).
  4. Enforce Pre-Embedding Governance (pii-masking, signed HTTP webhooks, custom PreEmbeddingGovernanceHook beans) before text is chunked and vectorized.
  5. Inspect Migrated & Lakehouse Metadata in Apache Ozone, AWS S3 (<key>.ois.json), or Apache Iceberg (IcebergSidecarStore, #302) via companion OIS sidecar representations without downloading raw binary payloads (listSidecarStores, readOisSidecars).
  6. Search, Time-Travel & Retrieve indexed knowledge across all 11 target vector, search & lakehouse backends (secureVectorSearch, getDocumentContent, listAccessibleSources, plus Apache Iceberg listIcebergSnapshots, getIcebergDocumentAsOf, queryIcebergEntities, getIcebergIngestionStats).

Every operation and response is strictly filtered by what the delegating user is authorized to read.


🏛️ Architectural Overview

flowchart TD
    User(["👤 Enterprise User<br/>(alice@corp, groups: finance)"])
    IdP["🔐 Identity Provider (IdP)<br/>OAuth 2.0 Token Exchange (RFC 8693)<br/>sub: alice | act.sub: research-orchestrator"]
    Swarm["🤖 Multi-Agent Swarm<br/>(Orchestrator & Concurrent Sub-Agents)"]

    User -->|"1. Delegates task"| Swarm
    Swarm -->|"2. Exchanges token (act.sub)"| IdP
    IdP -->|"3. Signed RS256 JWT"| Swarm

    subgraph OC["OpenCrawling Identity-Aware Data Plane"]
        Auth["🛡️ CallerResolver & JWT Resource Server<br/>(Ignores forged tool args; enforces OAuth2 scopes)"]
        
        subgraph Capabilities["Agentic Capabilities (MCP & REST)"]
            Disc["🔭 Repository Discovery<br/>(SourceAclVisibility)"]
            Dyn["🚚 Dynamic Jobs & Single-Doc<br/>(Approval Gate & Custom Chunking)"]
            Bpm["🔔 BPM Workflow Events<br/>(HMAC / Basic / oc:events:write)"]
            Side["🗂️ OIS Sidecar Reader<br/>(AES-GCM Sealed Cursors)"]
            Search["🔎 Vector Search<br/>(10 Search Engines)"]
        end

        Gov["🛡️ Pre-Embedding Governance Gate<br/>(PII Masking & Signed HTTP Webhooks)"]
        Gate["🔒 Unified DocumentAccessGate<br/>(metadata | shadow | intersection | openfga)"]
        Audit["📊 Lakehouse Audit Ledger<br/>(triggered_by, on_behalf_of, X-Correlation-Id)"]
    end

    Swarm -->|"MCP (/mcp) or REST (/api/**)"| Auth
    Auth --> Disc & Dyn & Side & Search
    Bpm --> Dyn
    Dyn -->|"Crawl & Extract"| Gov
    Gov -->|"Approved / Sanitized"| Search
    Side --> Gate
    Search --> Gate
    Dyn & Gov --> Audit
Loading

🛠️ What an Agent Can Do

Search (secureVectorSearch, getDocumentContent, listAccessibleSources, plus openarch_search and openarch_fetch_document) and the observability tools (fetchJobTraces, getErrorLogs, queryThroughputMetrics) are always on. Every other capability is disabled by default and can be enabled independently:

Capability MCP Tools REST Endpoints Required Scope (Default) Enabled By
Search indexed content secureVectorSearch, getDocumentContent, listAccessibleSources — (authenticated caller) Always enabled
Look before ingesting (Discovery) listRepositoryConnectors, exploreRepositoryTree, queryRepositoryDryRun GET /api/discovery/connectors
GET /api/discovery/{connector}/tree
POST /api/discovery/{connector}/dry-run
(authenticated caller) opencrawling.discovery.enabled=true
Ingest a folder (Dynamic Jobs) triggerDynamicJob, getDynamicJobStatus, listDynamicJobs, stopDynamicJob POST /api/jobs/dynamic
GET /api/jobs/dynamic
GET /api/jobs/dynamic/{jobId}
POST /api/jobs/dynamic/{jobId}/stop
POST /api/jobs/dynamic/{jobId}/approve
POST /api/jobs/dynamic/{jobId}/reject
oc:jobs:write to start a job; status, list and stop accept oc:jobs:write or oc:jobs:run; approve and reject also require membership of admin-groups opencrawling.jobs.dynamic.enabled=true
Ingest or remove one document ingestDocument, removeDeletedDocument POST /api/jobs/dynamic/documents oc:jobs:run opencrawling.jobs.dynamic.enabled=true
React to a business process — POST /api/events/bpm/document-transition oc:events:write (or HMAC / Basic) opencrawling.events.bpm.enabled=true
Read migrated metadata (Sidecars) listSidecarStores, readOisSidecars GET /api/sidecars
GET /api/sidecars/{connector}
oc:sidecars:read opencrawling.sidecars.enabled=true

🔐 Identity, Delegation & Swarm Isolation

Agents must authenticate with a cryptographic bearer token verified by the OpenCrawling runtime:

opencrawling:
  security:
    identity:
      mode: jwt
      protect-api: true          # Enforce JWT bearer verification on REST endpoints as well
spring:
  security:
    oauth2:
      resourceserver:
        jwt:
          issuer-uri: https://idp.example.com/realms/corp
          audiences: opencrawling
  • The Token Decides, Never the Tool Arguments: With opencrawling.security.identity.mode=jwt, the userPrincipal and userRoles arguments on every MCP tool and REST endpoint are ignored. An agent that attempts to claim another user's identity or role in its tool arguments still receives only the view authorized for the token's subject.
  • Strict Principal & Group Separation: The caller's user principal and group memberships are extracted from separate JWT claims (principal-claims vs. group-claims) and are never conflated, preserving prefix-aware matching (user: vs. group:, #224). See Model Context Protocol.
  • Least-Privilege Scope Gating: OAuth2 scopes are read from the token's scope or scp claim. Each capability enforces its own dedicated scope: a token carrying oc:sidecars:read can inspect OIS sidecars but cannot trigger folder crawls (oc:jobs:write) or single-document ingestions (oc:jobs:run).
  • OAuth 2.0 Token Exchange Delegation (RFC 8693):
    • When an agent acts on behalf of a human user, it should present a delegated token obtained via OAuth 2.0 Token Exchange (RFC 8693).
    • If the token contains an act (actor) claim, OpenCrawling records act.sub as the acting agent (triggered_by) and the delegating user as on_behalf_of. That is the mapped principal (the first of jwt.principal-claims present, preferred_username before sub by default), not necessarily the raw sub.
    • Without an act claim, the token's principal is recorded alone.
  • Multi-Agent Swarm Safety: A swarm consists of multiple concurrent sub-agents working in parallel, potentially on behalf of different users. Because identity and permissions are resolved per request directly from that request's verified bearer token, concurrent agent sessions never leak state or cross-contaminate results.

🔄 The Agentic Ingestion Flow

A typical autonomous agent workflow progresses through five zero-trust stages:

  1. Discover (listRepositoryConnectors → exploreRepositoryTree):
    • The agent enumerates discoverable connectors and walks folder hierarchies breadth-first.
    • Visibility is decided per node against the permissions the source repository reports live (SourceAclVisibility).
    • Folders the user cannot read are never opened or traversed, and no binary content is downloaded.
  2. Preview (queryRepositoryDryRun):
    • Previews how many documents in a folder or read-only query (CMIS SQL, Alfresco AFTS) the user is permitted to read, returning representative metadata samples.
  3. Ingest (triggerDynamicJob / ingestDocument):
    • Starts a crawl on an allow-listed connector (opencrawling.jobs.dynamic.connectors).
    • Per-Job Chunking: The agent can tailor chunkSize, minChunkSizeChars, minChunkLengthToEmbed, and keepSeparator within administrator-configured bounds (min-chunk-size..max-chunk-size).
    • Idempotency: Supplying an idempotencyKey (Idempotency-Key header) guarantees that retries return the existing job rather than spawning duplicates.
    • Four-Eyes Human Approval: When require-approval: true is enabled, jobs enter PENDING_APPROVAL and wait for a member of admin-groups (other than the requester) to approve via REST (POST /api/jobs/dynamic/{jobId}/approve). No MCP tool can approve a job.
  4. Follow (getDynamicJobStatus / listDynamicJobs / stopDynamicJob):
    • Users and their delegated agents see and control only their own jobs. Members of admin-groups see and can stop every job. stopDynamicJob halts execution cleanly before the next document.
  5. Search & Retrieve (secureVectorSearch / getDocumentContent):
    • Returns only chunks authorized for the user via DocumentAccessGate. In openfga mode, group membership changes take effect immediately on search without re-crawling.

Every ingested document preserves its original source repository permissions, regardless of which user or agent triggered the crawl—ensuring the vector index never grants anyone broader access than the source repository.

{"method": "tools/call", "params": {"name": "exploreRepositoryTree",
  "arguments": {"connector": "Alfresco_Main", "start": "/Finance", "depth": 2}}}

{"method": "tools/call", "params": {"name": "triggerDynamicJob",
  "arguments": {"connector": "Alfresco_Main", "start": "/Finance/2026", "idempotencyKey": "fin-2026-q3", "chunkSize": 400}}}

See Model Context Protocol — Repository Discovery and Dynamic Jobs.


🔔 Event-Driven Ingestion from Business Processes

Enterprise BPM engines (Flowable and Alfresco Process Services) can push real-time document lifecycle transitions (POST /api/events/bpm/document-transition) when a workflow reaches a business milestone (e.g., "contract signed" or "record archived").

  • Administrator-Defined Routing: Matching routes entries map (engine, processDefinitionKey, transition) to a registered connector and lifecycle action (UPSERT or DELETE).
  • Multi-Mechanism Authentication: Senders authenticate via HMAC-SHA256 request signing (X-OpenCrawling-Signature), HTTP Basic static tokens, or a JWT bearer token carrying oc:events:write.
  • Per-Sender Deduplication: Event IDs are deduplicated within dedup-retention (default 24h).

See Configuration Guide — BPM Document Events and Document Lifecycle Actions & Tombstones.


🛡️ Governance Before Embedding

When opencrawling.governance.enabled=true, every extracted document passes through an ordered chain of governance hooks inside IngestionConsumer before it is split into chunks and embedded:

  • Built-in PII Masking (pii-masking): Detects and redacts emails, Luhn-validated payment cards, and check-digit-verified IBANs.
  • External HTTP Governance Service (http): Calls an external compliance webhook signed with HMAC-SHA256 (X-OpenCrawling-Signature) and bounded retries.
  • Custom Hooks (PreEmbeddingGovernanceHook SPI): Pluggable Spring beans returning APPROVE, SANITIZE, or REJECT.

Key Governance Invariants

  • Permission Immutability: Governance hooks can never modify or widen document permissions or identity fields (security, acl, sys_acl, security_*, acl_*, documentId, uri, lastModified).
  • Transient Governance Metadata (drop-metadata-after-gate, #414, PR #417): Metadata keys configured via opencrawling.governance.drop-metadata-after-gate (e.g. zammad.participants provided for anonymization hooks) remain readable inside governance hooks and are stripped by IngestionConsumer right after the gate (or before chunking when the gate is off) so they are never persisted downstream. Protected keys cannot be dropped; outputs written directly outside the Kafka pipeline still receive the keys.
  • Fail-Closed Execution: If any hook errors or times out (UNAVAILABLE), nothing is embedded and no tombstone is emitted. The claim check is left in storage, but nothing re-processes it automatically and the Kafka message is not retried; the next crawl of the document tries again.
  • Rejections Purge Stale Chunks & OpenFGA Tuples: On every REJECT under on-rejected-update: purge (default), a DELETE message for the document is published on the chunks topic to remove previously indexed chunks, and an IngestionMessage.governanceRejection notice is published on the documents topic so AuthorizationSyncConsumer purges its OpenFGA relationship tuples (#268, PR #282). Set on-rejected-update: keep to leave earlier chunks in place. The OpenFGA tuples then keep the permissions of the newest crawled version, which is the rejected one.
  • Direct Outputs & Decoupled Pipeline Enforcement (#351, PR #353): With governance on, jobs that would send documents straight to an output connector—and so bypass IngestionConsumer—are refused before crawling starts (409 Conflict, releasing any per-job connector built for the start). RAG jobs on the runtime's active vector store write through the Kafka pipeline only, so they are governed and run; a job with its own Qdrant or Vespa output targeting another destination, SeaTunnel reading a kafka-topic other than opencrawling-embedded, or a direct-only RAG output (Hyland CIC, OpenArch) sends directly and is refused. Dynamic jobs, BPM events, OIS ingestion, and gRPC push publish to Kafka and pass the gate.

See Configuration Guide — Pre-Embedding Governance.


🗂️ Reading Migrated Content's Metadata (OIS Sidecars)

Documents migrated to Apache Ozone or AWS S3 in Migration Mode store a companion OIS metadata sidecar (<key>.ois.json) alongside each binary object. With opencrawling.sidecars.enabled=true, listSidecarStores and readOisSidecars (GET /api/sidecars/**) let agents inspect migrated document metadata, SHA-256 checksums, source lineage, and page aggregates by MIME type and source without downloading binaries:

  • Unified Gate Filtering: Each sidecar is evaluated in batches through DocumentAccessGate on its document ID and security configuration across all 4 modes (metadata, shadow, intersection, openfga).
  • Zero Permission & Count Leakage: Returned sidecar views strip all permission fields and never disclose how many sidecars were hidden.
  • AES-GCM Sealed Paging Cursors: nextCursor is encrypted with AES-GCM and bound to the caller's tenant, principal, connector, and prefix so keys of hidden sidecars never leak.

See Model Context Protocol — OIS Sidecars and Configuration Guide — OIS Sidecars.


🔒 Runtime Security Guarantees & Invariants

  1. One Unified Decision Model:
    • Vector Search & OIS Sidecars pass through DocumentAccessGate (metadata, shadow, intersection, or openfga mode).
    • Repository Discovery & Pre-Crawl Checks evaluate the permissions reported live by the source repository using SourceAclVisibility (which applies EffectiveAcl metadata rules), regardless of authorization mode, because OpenFGA only tracks already-indexed documents. Consequently, group memberships held exclusively in OpenFGA are not applied during discovery: a caller may see less in discovery than in search, never more.
  2. Source Permission Preservation: Nothing an agent does can widen a document's ACLs.
  3. Strict Fail-Closed Semantics: Unverifiable callers receive 401 Unauthorized. If authorization or governance cannot reach a decision, nothing is returned or embedded. REST discovery, sidecars and dynamic jobs answer 503 Service Unavailable; MCP search returns the error to the client; governance runs in the Kafka consumer, so it has no HTTP status and simply does not embed the document.
  4. Hard Resource & Rate Bounds:
    • Discovery walks, sidecar scans, and governance webhooks enforce strict depth, node count, scan count, byte size, and timeout caps.
    • Dynamic jobs enforce per-user running limits (max-running-per-user), per-user pending approval limits (max-pending-per-user), and global concurrency caps (429 Too Many Requests).
  5. Zero Information Leakage (Anti-Probing):
    • Unreadable folders, out-of-scope nodes, and non-existent paths return identical responses.
    • Hidden item counts are withheld by default (reveal-hidden-counts: false).
    • Upstream repository and object storage exceptions are logged server-side and never echoed to callers (502 Bad Gateway).
  6. End-to-End Lakehouse & Trace Auditing:
    • Every job and document execution records triggered_by (the agent act.sub or user), trigger_source (mcp, dynamic-api, bpm, and for other runs api, ois, grpc-push, startup), on_behalf_of (the delegating user), and X-Correlation-Id across both the job summary and every row of the columnar Apache Parquet / Avro Lakehouse Audit Ledger (see Lakehouse Audit Reporting & Ledger).
    • Governance decisions and hook audit attributes are recorded per document in the in-memory trace store (TelemetryTraceStore), keyed by document id, which the observability tools read. They are not exported as OpenTelemetry spans.

⚙️ Unified Configuration Checklist

opencrawling:
  security:
    identity:
      mode: jwt
      protect-api: true
    authorization:
      mode: openfga                 # metadata (default) | shadow | intersection | openfga
  discovery:
    enabled: true
  jobs:
    dynamic:
      enabled: true
      connectors: Alfresco_Main      # Registered repository connectors agents may crawl
      admin-groups: oc-admins
      require-approval: false
  events:
    bpm:
      enabled: false
  governance:
    enabled: true
    hooks: pii-masking,http
  sidecars:
    enabled: true
    connectors: Lake_S3              # Registered Ozone/S3 output connectors

Every property and environment variable is documented in the Configuration Guide.


🧪 Verification & In-JVM Test Suites (#203, PR #260; #261, PR #262)

OpenCrawling continuously verifies multi-agent security invariants in the JVM test suite:

1. Discovery vs. Search Agreement (DiscoveryVersusSearchTest & IngestionMessageTest)

Generates a deterministic 400-document corpus (fixed seed 203L) covering case variants (Alice, USER:Alice), user: and group: prefixes, bare identities (#224), explicit denies, write grants, public grants, inheritance flags, empty ACLs, and missing security objects (null). The corpus is evaluated down both pipelines for 8 distinct caller profiles:

  • As Discovery Nodes: Evaluated directly through SourceAclVisibility.
  • Through the Full Ingestion & Search Pipeline: Processed through the real IngestionConsumer, enriched via VectorStoreWriterConsumer.ensureSecurityMetadata, serialized through a JSON round-trip (as stored in vector databases), and filtered by DocumentAccessGate in metadata mode.
  • Verified Invariants:
    • Exact Set Agreement: Discovery and search return the exact same set of documents for every caller across all documents.
    • Flattened-Only Store Compatibility: Identical agreement holds for vector stores that persist only flattened permission fields (security_allowed_read / security_denied_read) without the nested security object.
    • Pre-Filter Completeness (security_read_keys): Proves that the vector store's native SQL/filter pre-filter (ReadKeys.forCaller) never hides a document that DocumentAccessGate allows.
    • Empty ACL & Missing Security Fail-Closed (SecurityConfig.none(), #261, PR #262): Documents with an empty permission list or a missing security object (security == null or omitted in JSON) default to SecurityConfig.none() (never public) across IngestionMessage, OzoneOutputConnector, and S3OutputConnector. With no rules, EffectiveAcl evaluates the legacy acl string alone—so a blank acl grants no one across discovery, vector search, OpenFGA (AuthorizationSyncConsumer), and Ozone/S3 .ois.json sidecars (acl = "public" remains public, and explicit SecurityConfig.createPublic() placeholders remain unchanged).

2. Multi-Agent Swarm Protocol Verification (AgentSwarmHttpTest)

Executes against a live Streamable HTTP MCP server (/mcp) protected by the Spring Security OAuth2 Resource Server using RS256-signed, delegated tokens (act.sub: "research-orchestrator"):

  • Agent Tool Surface Discovery: Verifies that the tools an authenticated MCP client discovers include these 14 agent tools (secureVectorSearch, getDocumentContent, listAccessibleSources, listRepositoryConnectors, exploreRepositoryTree, queryRepositoryDryRun, triggerDynamicJob, getDynamicJobStatus, listDynamicJobs, stopDynamicJob, ingestDocument, removeDeletedDocument, listSidecarStores, readOisSidecars). With every feature on, a pgvector deployment exposes 19 tools in total.
  • Concurrent 4-Agent Swarm with Forged Arguments: Four sub-agents run concurrently across 12 rounds for two users (alice in finance and bob in hr). In every call (exploreRepositoryTree, readOisSidecars, secureVectorSearch), each agent passes the other user's userPrincipal and userRoles in the tool arguments. The test proves that every agent receives strictly its own token user's view and that every backend search query executes under the token's verified identity.
  • Delegated Job Ownership & Isolation: Verifies that a job triggered by a delegated token records research-orchestrator as triggeredBy and alice as onBehalfOf / owner, and that the same orchestrator acting on behalf of bob cannot inspect, list, or re-trigger alice's folder job.
  • Strict Per-Tool Scope Enforcement: Confirms that a token scoped to oc:sidecars:read can read OIS sidecars but fails when calling triggerDynamicJob or ingestDocument, while a token scoped to oc:jobs:write can explore and crawl but is denied access to readOisSidecars.
  • Untrusted Issuer Key Rejection: Confirms that a valid JWT signed by an unknown RSA key is rejected during MCP session initialization.

🗺️ Documentation


🎓 Tutorials


🧪 Testing


🤝 Community

Clone this wiki locally