Skip to content

Zero Trust Data Provider Agent

Piergiorgio Lucidi edited this page Nov 20, 2026 · 9 revisions

🛡️ Zero-Trust Data Provider Agent & Active Ingestion

OpenCrawling started as an ingestion and vector-search engine. With issue #203 it also became a data provider that AI agents can drive safely. An agent can now:

  • look inside an enterprise repository before ingesting anything;
  • decide what to ingest and start the crawl itself;
  • have business processes push single documents;
  • send content through governance before it is embedded;
  • read migrated metadata, and search.

Every one of those steps is decided for the user the agent works for, against the permissions the source repository enforces.

This page explains what was built, how it was built, and why each design decision was taken. The reference pages linked at the end hold every setting and endpoint.

Note

Status.

  • Merged: delivered in 19 merged pull requests (#228 to #262), all reviewed and merged by the maintainer.
  • Fixed: the two defects found by the live end-to-end run are fixed on main: #265 (Alfresco DENIED entries) and #266 (search without dimensions). Other follow-ups are listed under Known limitations.
  • Off by default: every agent capability is off until an administrator enables it.

OpenCrawling as a Zero-Trust Data Provider Agent


Contents

  1. The problem in one minute
  2. Architecture
  3. The active ingestion cycle
  4. How we built it
  5. Architecture decisions
  6. Security model and threat coverage
  7. Governance before embedding
  8. Event-driven ingestion from business processes
  9. Reading migrated metadata
  10. Audit and provenance
  11. Deploying it
  12. How it is verified
  13. From proposal to delivery
  14. Known limitations and follow-ups
  15. Troubleshooting
  16. Reference

⏱️ The problem in one minute

Issue #203 named three problems that stop enterprises from connecting agents to their repositories:

Problem What it looks like What OpenCrawling does now
Context leaks Agent tools search without knowing who asked, so HR, legal or finance content reaches the wrong person. Every call carries a verified identity. Every answer is filtered by that user's source permissions, and an explicit deny always wins.
Batch latency A contract signed at 10:00 is invisible to agents until the nightly crawl. A BPM engine posts an event and that one document is re-ingested within seconds.
Passive ingestion A human must configure every crawl in advance, so an agent can only use what someone guessed it would need. The agent discovers what exists, previews it, and starts a bounded crawl of exactly the folder or document it needs. All of this happens within what an administrator allowed.

🏛️ Architecture

The diagram at the top of the page shows the four layers. Read it left to right:

  1. Callers: a user, the agents acting for them, BPM engines, and human reviewers.
  2. Identity and policy: one OAuth2 resource server.
    • It turns a bearer token into a caller: the principal, the groups, and the acting agent from act.sub.
    • It ignores any identity claimed in tool arguments.
    • It checks the scope each capability needs.
  3. Agent capabilities: six independent features.
    • Search is the only one on by default.
    • Each feature is exposed as MCP tools and as REST endpoints.
  4. Data plane: the decoupled Kafka pipeline that already existed.
    • Content goes to the claim-check store, then the documents topic, extraction, the governance gate, chunking, embedding, and the vector writer.
    • The authorization gate and OpenFGA sit next to the pipeline.

Two properties make the design hold together:

  • Every path an agent can trigger feeds the same pipeline. It goes through the documents topic, so governance and the OpenFGA permission sync see every document without extra code.
  • Every answer passes one of two deciders, both using the same rules. They are described in Security model.

🔄 The active ingestion cycle

The active ingestion cycle

The issue described the "apex pattern" as discover, decide, define, execute. In the delivered system that becomes six steps, each with its own control:

Step MCP tools REST What protects it
1. Discover listRepositoryConnectors, exploreRepositoryTree GET /api/discovery/connectors, GET /api/discovery/{connector}/tree Decided on the ACL the repository reports now. Hidden folders are never opened.
2. Preview queryRepositoryDryRun POST /api/discovery/{connector}/dry-run Honest counts: EXACT, LOWER_BOUND or ESTIMATE. A query returns no count unless reveal-hidden-counts is on.
3. Ingest triggerDynamicJob, ingestDocument, removeDeletedDocument POST /api/jobs/dynamic, POST /api/jobs/dynamic/documents Allow-listed connectors, start folder in scope and readable, idempotency keys, per-user limits.
4. Approve (optional) none POST /api/jobs/dynamic/{jobId}/approve, …/reject Another admin, never the requester. Re-checked under the requester's identity.
5. Govern none (automatic) none Ordered hooks before chunking. Permissions are immutable. Fails closed.
6. Use secureVectorSearch, getDocumentContent, listAccessibleSources none The store pre-filter narrows candidates, then the gate re-checks every hit.

Agents follow their jobs with getDynamicJobStatus, listDynamicJobs and stopDynamicJob (GET /api/jobs/dynamic, GET /api/jobs/dynamic/{jobId}, POST /api/jobs/dynamic/{jobId}/stop).

A worked session

Here an agent works for alice over MCP with a delegated token. The identity comes from the token, so no userPrincipal is sent. The values below are illustrative, but the field names are the real ones.

1. Discover. The agent explores the /Finance folder.

// tools/call exploreRepositoryTree
{"connector": "Alfresco_Main", "start": "/Finance", "depth": 2}
// result: only what alice may read; folders she cannot read are not opened
{"connector": "Alfresco_Main",
 "nodes": [{"id": "…", "sourceId": "…", "parentId": "…", "name": "2026", "path": "/Finance/2026",
            "type": "FOLDER", "depth": 1, "mimeType": null, "sizeBytes": null, "lastModified": "…"}],
 "hiddenCount": null, "truncated": false, "truncationReason": null}

2. Preview. It asks how many documents a crawl of /Finance/2026 would take.

// tools/call queryRepositoryDryRun
{"connector": "Alfresco_Main", "scope": "/Finance/2026", "sampleSize": 3}
// result: a folder dry run checks every document, so the count is alice's own
{"connector": "Alfresco_Main", "matchCount": 42, "countAccuracy": "EXACT", "samples": […],
 "hiddenCount": null, "truncated": false, "truncationReason": null, "note": null}

3. Ingest. It starts the crawl, with its own chunk size and an idempotency key.

// tools/call triggerDynamicJob
{"connector": "Alfresco_Main", "start": "/Finance/2026", "idempotencyKey": "fin-2026-q3", "chunkSize": 400}
// result
{"jobId": "dyn-3f9c2a1b7e40", "executionId": "exec-5d1e9a02",
 "statusUrl": "/api/jobs/dynamic/dyn-3f9c2a1b7e40", "state": "RUNNING", "replayed": false}

4. Follow. It checks the job until it ends.

// tools/call getDynamicJobStatus {"jobId": "dyn-3f9c2a1b7e40"}
{"jobId": "dyn-3f9c2a1b7e40", "kind": "FOLDER", "state": "COMPLETED", "documents": 42,
 "owner": "alice", "triggerSource": "mcp", "approvedBy": null, "chunking": {"chunkSize": 400, …}, …}

5. Use. secureVectorSearch now returns chunks of those documents, but only to users their source ACL allows. That includes alice, and anyone else the source allows; the agent's request did not change who can read them.


🏗️ How we built it

Delivery roadmap

Approach

Three working rules shaped the delivery:

  1. Build the trust layer before giving agents any power.
    • Before #203, MCP tools trusted whatever identity the client sent, and stopping a job only changed a status string.
    • Letting agents start crawls on top of that would have been unsafe.
    • So the first phases made the caller verifiable and job control real. Discovery, ingestion, governance and sidecars all build on those two foundations.
  2. Small, reviewable pull requests in dependency order. Each PR:
    • delivered one capability;
    • carried its tests and docs;
    • passed the full CI before merge, including the 63-suite integration matrix.
  3. Prove that the tests would catch a regression.
    • Coverage shows that code ran. It does not show that a test would notice if the code were wrong.
    • So every feature PR was mutation-tested: we broke the new rules on purpose, one by one, and confirmed that a test failed each time.

Phases

Phase Why at this point Pull requests What it established
0. Prerequisites Agents would inherit any flaw in how identities match #228, #230 user: grants match only principals and group: grants only groups. The pgvector pre-filter applies the same rule as the gate.
A. Foundations No agent feature is safe without a verified caller and real job control #231 (#187), #232, #233, #235, #238 Bearer-token identity (jwt mode), connectors built by a factory (an unknown class no longer silently crawls the file system), real stop, run caps, run provenance, per-caller quotas
B. Discovery An agent must see before it acts, without reading content #242, #243, #244 A discovery SPI with CMIS and Alfresco implementations. REST and MCP tools filtered by the live source ACL.
C. Agent ingestion Act on what discovery found, within limits #250, #251, #253, #256, #257 Dynamic folder jobs, single-document ingest and removal, BPM events, optional human approval, per-job chunking
D. Governance Agents widen what gets ingested, so content must be checked before it is embedded #258 Pre-embedding gate with PII masking, a signed HTTP hook and an SPI
E. Sidecars Migrated content must be usable without downloading binaries #259 An OIS sidecar reader behind the same gate as search
F. Verification Prove the invariants end to end, not just per feature #260, #262 Discovery-versus-search agreement, a multi-agent swarm over real MCP, and "no security object" never meaning public

Engineering evidence

PR Capability Deliberate breaks (mutants) caught
#231 Verified caller (#187) 4 / 4
#232 Connector factory 4 / 4
#233 Run registry, real stop 5 / 5
#235 Run provenance 6 / 6
#238 Per-caller quotas 6 / 6
#242 Discovery SPI + CMIS 9 / 9
#243 Discovery REST + MCP 10 / 10
#244 Alfresco discovery 13 / 13
#250 Dynamic jobs 18 / 18
#251 Single document 23 / 23
#253 BPM events 19 / 19
#256 Human approval 13 / 13
#257 Per-job chunking 27 / 27
#258 Governance gate 47 / 47
#259 OIS sidecars 44 / 44
#260 Agreement and swarm tests 7 / 7
Total 255 / 255

In some cases a mutant survived because the check it broke could never be reached. We then removed the redundant check rather than keeping untested code. Each PR description records its mutants and its test counts.


🧭 Architecture decisions

Each decision below gives:

  • the problem it solves;
  • the principle it applies;
  • the alternatives we rejected, and why;
  • the cost we accepted;
  • the evidence that enforces it.

The decisions were agreed before or during implementation, and the issue and PR threads record them.

# Decision Principle
01 Identity comes from a verified token, never from tool arguments Complete mediation; never trust the client
02 Delegation is explicit: user as subject, agent as actor Accountability (RFC 8693)
03 One scope per capability Least privilege
04 Agents name registered connectors, never endpoints or credentials SSRF and confused-deputy prevention
05 Discovery decides on the live source ACL in every mode Fail-safe under-showing
06 Indexed content has one enforcement point Complete mediation
07 Documents keep source permissions, whoever triggers the crawl Permissions follow the data
08 Agent jobs run only on the decoupled pipeline One governed path
09 Fail closed; no permissions means nobody Fail-safe defaults
10 Hidden looks exactly like missing No existence oracle
11 Everything is bounded and says when it was truncated Availability; honest results
12 Human approval: optional, four-eyes, REST-only, re-checked Separation of privilege
13 Retries are safe by construction Idempotency
14 Removal only when the source says the document is gone Integrity
15 BPM: routes authorize; three ways to authenticate Meet real engines where they are
16 Governance between extraction and chunking, impossible to bypass Complete mediation; fail closed
17 Per-job chunking exposes only what the splitter really does No false controls
18 Sidecars: same gate, sealed cursor, read-only stores Defence in depth
19 Provenance records only verified identities Trustworthy audit
20 Additive and off by default Safe adoption

ADR-01: Identity comes from a verified token, never from tool arguments

  • Context:
    • The MCP tools took userPrincipal and userRoles as arguments.
    • An agent, or a prompt injected into it, could claim to be anyone, and the server had no way to tell.
  • Decision:
    • opencrawling.security.identity.mode=jwt makes the runtime an OAuth2 resource server. Spring Security validates the signature, expiry and issuer, and the audience when spring.security.oauth2.resourceserver.jwt.audiences is set. Set it: without it, the runtime warns that it accepts any token from the issuer.
    • CallerResolver builds the caller from the token: the principal from principal-claims, the groups from group-claims, and the tenant.
    • The identity arguments are then ignored.
    • Every new tool must get its caller from CallerResolver.
  • Principle: complete mediation. The client's claim about itself is never the authority. This also matches the MCP authorization model, in which the MCP server is an OAuth 2.1 resource server.
  • Alternatives rejected:
    • Keep asserted arguments: they can be spoofed by construction.
    • A static API key per agent: it proves which agent is calling, but not which user it acts for. Every agent would then need the union of its users' rights.
  • Cost:
    • It needs an identity provider. The default stays none, so existing deployments are unaffected (ADR-20).
    • With protect-api enabled, set opencrawling.security.identity.admin-ui.client-id so the admin UI signs users in via OpenID Connect PKCE (see Security and Identity Management).
  • Evidence:
    • In AgentSwarmHttpTest, every call forges the other user's identity in its arguments, and every answer is still the token user's view.
    • A token signed by an unknown key opens no MCP session.

ADR-02: Delegation is explicit: the user is the subject, the agent is the actor

  • Context:
    • An agent acts for a person.
    • Treating the agent as its own principal would make it a confused deputy: it would hold broad rights and decide alone whose data to show.
  • Decision:
    • Agents present a token obtained by OAuth 2.0 Token Exchange (RFC 8693).
    • The token's subject is the user, and the act.sub claim names the agent.
    • Access is decided for the user. The agent is recorded as triggered_by and the user as on_behalf_of.
  • Principle: accountability without privilege escalation. The agent never has more access than the user it serves.
  • Alternatives rejected:
    • Pass the user's own token through: there would be no record of which agent acted.
    • A service account for the agent: it over-privileges the agent and turns it into a confused deputy.
  • Cost: it depends on the identity provider supporting token exchange and putting the agent in act.
  • Evidence: a delegated job records research-orchestrator as triggeredBy and alice as owner. The same orchestrator acting for bob gets the same answer as for a job that does not exist: a tool error over MCP, 404 over REST. It also cannot see alice's job in its list, or start a crawl on her folder.

Delegation and the agent swarm

ADR-03: One scope per capability

  • Context: a workflow service that refreshes documents should not also be able to crawl whole folders, and a research agent that reads sidecars should not be able to start jobs.

  • Decision: scopes are read from the scope or scp claim:

    Scope What it allows
    oc:jobs:write validate and start folder jobs
    oc:jobs:run single documents
    oc:sidecars:read sidecars
    oc:events:write BPM events sent with a bearer token
    • Following and stopping your own jobs accepts either job scope.
    • Discovery and search need no scope, because the data ACL already limits them to what the user may read.
  • Principle: least privilege.

  • Alternatives rejected: one "agent" scope. A leaked token would then grant everything.

  • Cost: administrators configure scopes in the identity provider.

  • Evidence: AgentSwarmHttpTest.eachToolAsksForItsOwnScope (oc:sidecars:read, oc:jobs:write), DynamicJobServiceTest.eachOperationNeedsItsOwnScopeAndEitherScopeFollowsJobs (oc:jobs:run, either scope follows jobs) and BpmEventServiceTest.aTokenCallerNeedsTheEventScope (oc:events:write).

ADR-04: Agents name registered connectors; they never supply endpoints or credentials

  • Context: the issue's sample payload let an agent pass a repository endpoint URL and connector configuration.
  • Decision:
    • An agent names a connector registered in connectors.json and listed in opencrawling.jobs.dynamic.connectors, plus a start folder or a document id. That is all it can choose.
    • A connector that is not listed answers exactly like one that does not exist.
    • The connector's own credentials, root, exclusions and filters always apply.
  • Principle: prevent SSRF (OWASP Top 10, A10) and credential exfiltration. An agent that could choose the host would make OpenCrawling probe internal networks, or send its credentials to a server the attacker controls.
  • Alternatives rejected: validate agent-supplied URLs against an allow-list. It is fragile (DNS rebinding, redirects) and still lets agents mix credentials and hosts.
  • Cost: new sources need an administrator. We consider that a feature.
  • Evidence: DynamicJobServiceTest covers unlisted, unregistered and non-discoverable connectors; CmisDiscoveryTest covers connectors that crawl a query or a change log, which have no crawl start.

ADR-05: Discovery decides on the live source ACL in every authorization mode

  • Context:
    • OpenFGA only holds tuples for documents that have already been indexed.
    • In openfga mode, deciding discovery through the configured provider would hide every folder, and every document not yet ingested. Discovery would then be useless exactly when it is needed.
  • Decision: discovery, dry runs, the start folder of a dynamic job and single-document ingests are decided by SourceAclVisibility. It applies the metadata rules to the ACL the repository reports now, whatever opencrawling.security.authorization.mode is. An item with an unknown ACL is hidden.
  • Principle: fail-safe defaults. Memberships held only in OpenFGA are not applied here, so discovery can show less than search, never more.
  • Alternatives rejected:
    • The configured provider: it hides everything that is not indexed in OpenFGA modes.
    • The indexed ACL: there isn't one before ingestion, and it would be stale.
  • Cost: a user whose access comes only from an OpenFGA membership may not see a folder in discovery that search would show them.
  • Evidence: DiscoveryVersusSearchTest runs a 400-document corpus with mixed permissions down both paths for 8 callers, with search in metadata mode, and both return the same set. That proves both paths apply the same rules. Documents with no security are checked separately and are hidden both ways. The OpenFGA side of this decision is a design argument: OpenFGA has no tuples for content that is not indexed.

Where access is decided

ADR-06: Indexed content has one enforcement point

  • Context: with ten search backends and a sidecar reader, separate ACL checks per feature would drift apart over time.
  • Decision:
    • Every MCP search backend and the sidecar reader pass their candidates to one DocumentAccessGate, which asks the configured provider: metadata, shadow, intersection or openfga.
    • The store's pre-filter (security_read_keys) only narrows the candidates.
    • The gate decides every hit. In openfga mode, where tuples can allow what the stored metadata does not, the pre-filter is skipped, so it can never hide an allowed hit.
  • Principle: complete mediation, with defence in depth.
  • Alternatives rejected: trust the store's filter. Stores differ in filter semantics, and some have none.
  • Cost: a batch check per page. In openfga mode the gate also over-fetches candidates, bounded by overfetch-factor (10) and candidate-budget (500).
  • Evidence: the pre-filter safety check in DiscoveryVersusSearchTest proves the pre-filter never hides a document the gate allows (in metadata mode).

ADR-07: Documents keep their source permissions, whoever triggers the crawl

  • Context: an agent may start a crawl of a folder that contains documents its user cannot read.
  • Decision:
    • The crawl takes everything the connector's own rules take, and every document is indexed with its own source permissions.
    • The requester's access decides only whether the crawl may start: the start folder must be readable by them.
  • Principle: permissions follow the data. The index is shared by all users, so it must hold each document's real ACL. A copy filtered for one requester would be wrong for everyone else.
  • Alternatives rejected: index only what the requester can read. That produces an index whose content depends on who asked first.
  • Cost: an agent can cause documents it cannot read to be ingested. They stay invisible to it, and the optional approval gate can review such requests.
  • Evidence: the live run in How it is verified: a crawl started for alice made each user see exactly their own documents.

ADR-08: Agent jobs run only on the decoupled pipeline

  • Context: in the monolithic path, an output connector embeds content itself. That would bypass both the governance gate and the OpenFGA sync.
  • Decision:
    • Dynamic jobs run with no output connector, always in RAG mode.
    • Every document goes to the documents topic, and from there through ingestion, governance and the permission sync.
  • Principle: one governed path. The OpenFGA sync then needs no extra code for new ingestion routes.
  • Alternatives rejected: let agents pick the pipeline mode or the output. That gives a second, ungoverned route.
  • Cost: agents cannot run migration jobs or choose a destination. Those remain administrator decisions.
  • Evidence: the dynamic-job tests assert that no output connector is used.

ADR-09: Fail closed, and no permissions means nobody

  • Context: with permissive defaults, every misconfiguration becomes a data leak.
  • Decision:
    • An unverifiable caller gets 401.
    • An authorization provider that cannot decide returns nothing: REST answers 503, and MCP calls fail.
    • A governance hook that cannot decide means nothing is embedded.
    • A misconfigured gate stops the runtime at startup.
    • A document with no security object is SecurityConfig.none() (#261, #262). Only its legacy acl string can grant access, and with a blank acl nobody can read it.
  • Principle: fail-safe defaults (Saltzer and Schroeder).
  • Alternatives rejected: defaulting to public for compatibility. That was the behaviour #262 removed.
  • Cost: misconfigured sources produce invisible documents instead of leaked ones. That is diagnosable, and the opposite would not be.
  • Evidence: IngestionMessageTest, DiscoveryVersusSearchTest, and the startup-failure tests for the gate and the HTTP hook.

ADR-10: Hidden looks exactly like missing

  • Context: an error that says "forbidden" confirms that something exists.
  • Decision:
    • A folder or document the caller cannot read, one outside the connector's root, and one that does not exist all get the same answer.
    • Hidden counts are off by default (reveal-hidden-counts: false).
    • Repository and storage errors are logged and never passed to the caller (502 with a fixed message).
    • Sidecar pages never say how many sidecars were hidden.
  • Principle: no existence oracle. RFC 9110 explicitly lets a server answer 404 to hide a resource it will not show.
  • Cost: harder debugging for callers. The server log keeps the detail.
  • Evidence: "same answer" assertions in the discovery, dynamic-job, single-document and sidecar tests.

ADR-11: Everything is bounded, and says when it was truncated

  • Context: an agent can ask for the whole repository in one call.

  • Decision: every walk, scan and call has a per-request limit and a hard ceiling:

    Area Limits
    Discovery max-depth 3, max-nodes 200, max-scan 10,000, timeout 30 s, each with a hard ceiling
    Sidecars 50 per page (at most 200), 5,000 keys scanned, 8 MB per answer
    Governance HTTP calls 30 s per attempt, 3 attempts
    Jobs max-running-per-user (2), max-pending-per-user (10), plus the runtime's run caps (429)
    • Results say when they stopped early (truncated, truncationReason), and counts say how exact they are (EXACT, LOWER_BOUND, ESTIMATE).
  • Principle: availability, and never presenting partial data as complete.

  • Cost: agents must page or narrow their requests on large repositories.

  • Evidence: the bound tests in each feature. A discovery timeout returns 200 with truncationReason: TIMEOUT, not an error.

ADR-12: Human approval is optional, four-eyes, REST-only and re-checked

  • Context: some organisations want a person to approve what agents ingest. Others want full automation.
  • Decision:
    • With require-approval: true, every folder job or document ingest first passes every normal check, then waits in PENDING_APPROVAL. This applies whether an agent or a person asked for it.
    • A member of admin-groups who is not the requester approves or rejects it over REST.
    • On approval the request is checked again under the requester's identity, and then starts under the same job id.
    • Unanswered requests expire after 24 hours.
    • There is no MCP tool for approval.
  • Principle: separation of privilege. An agent holding an approval tool could approve its own work. Re-checking at approval time closes the time-of-check/time-of-use gap while the job waited.
  • Alternatives rejected: approval always on: it blocks automated deployments. Approval as an MCP tool: it defeats the purpose.
  • Cost: pending state is held in memory, like job status (see Known limitations).
  • Evidence: 13 of 13 approval mutants caught. The tests cover self-approval, approval by a non-admin, and access that changed while the job waited.

ADR-13: Retries are safe by construction

  • Context: agent loops retry tool calls, so the same crawl could start twice.
  • Decision:
    • An idempotencyKey (REST Idempotency-Key header) belongs to the user. It is fingerprinted with the whole request, chunking included.
    • The same key with the same request returns the first job ("replayed": true).
    • The same key with a different request is refused (409).
    • A refused start does not use up the key.
  • Principle: idempotent operations, following the IETF Idempotency-Key HTTP header draft.
  • Evidence: the idempotency tests in DynamicJobServiceTest.

ADR-14: A document is removed only when its source says it is gone

  • Context: a remove request is a way to hide a live document from every user's search.
  • Decision:
    • removeDeletedDocument and BPM DELETE routes publish a tombstone only when the repository reports the document gone.
    • Asking to remove a live document you can read gets 409.
    • Every other request gets the same 202, so the answer reveals nothing.
  • Principle: integrity. Agents and workflows can keep the index in step with the source, but cannot make it diverge from it.
  • Evidence: the single-document tests for live, gone, hidden and out-of-scope documents.

ADR-15: BPM events: routes authorize, and three ways to authenticate

  • Context:
    • A BPM engine's service account rarely appears in document ACLs.
    • Neither Flowable nor Alfresco Process Services signs requests or fetches OAuth2 tokens out of the box. Flowable's HTTP task sets headers from expressions, so HMAC needs a small signing bean. The APS REST call supports Basic auth.
  • Decision:
    • Routes authorize: an administrator's route (engine, process, transition → connector, action) authorizes the event. The connector still limits the document to its scope.
    • Three ways to authenticate:
      • HMAC-SHA256 over timestamp + "." + body, with a 5-minute window, compared in constant time, in the same timestamped scheme Stripe-style webhooks use;
      • a per-source static token over Basic auth;
      • a bearer token with oc:events:write.
    • Bounded: secrets must be at least 32 characters, events are deduplicated per sender for 24 hours, and bodies are limited to 64 KiB.
  • Principle: meet real engines where they are, without weakening the model. Basic is the weakest of the three, so use HMAC or a bearer token where you can, and always TLS.
  • Cost: HMAC from Flowable needs a small signing bean.
  • Evidence: 19 of 19 BPM mutants caught. The tests cover bad signatures, stale timestamps, duplicate events and missing scopes.

Event-driven ingestion

ADR-16: Governance sits between extraction and chunking, and cannot be bypassed

  • Context: regulated content must be checked, and possibly redacted, before any vector is computed from it.
  • Decision:
    • In IngestionConsumer, after Tika extraction and before chunking, an ordered chain of hooks (pii-masking, http, or any PreEmbeddingGovernanceHook bean) answers APPROVE, SANITIZE or REJECT.
    • Protected keys (acl, security, security_*, acl_*, sys_acl, documentId, uri, lastModified) are restored after every hook.
    • A hook that fails means nothing is embedded.
    • With governance on, runs that would embed through an output connector are refused (409). That means every non-migration run started through the job API.
    • A rejection removes earlier chunks and OpenFGA relationship tuples unless on-rejected-update: keep (#268, PR #282).
  • Principle: complete mediation and fail-safe defaults. Purging on reject means the system stops serving text that governance no longer approves.
  • Alternatives rejected:
    • Govern after embedding: the vectors already encode the content.
    • A hook may edit permissions: it would turn governance into an access-widening channel.
  • Cost:
    • While governance is on, the job API refuses every non-migration run (409). It always hands documents to an output connector, the runtime's own one when the job names none. Crawls then run through the decoupled crawler containers or dynamic jobs.
    • Outcomes are kept as a per-document Governance span in the runtime's trace store, not in the execution ledger: the consumer may run in its own container, with no ledger for the crawl.
  • Evidence: 47 of 47 governance mutants caught.

ADR-17: Per-job chunking exposes only what the splitter really does

  • Context: the issue asked for chunkSize and chunkOverlap. Spring AI's TokenTextSplitter has no overlap. Its "max chunks" option silently puts the rest of a document into one oversized chunk.
  • Decision:
    • A job may set chunkSize, minChunkSizeChars, minChunkLengthToEmbed and keepSeparator.
    • chunkSize must stay within min-chunk-size and max-chunk-size (100 and 2,048 tokens by default, never above 8,192).
    • There is no overlap or chunk-count setting.
  • Principle: no false controls. A setting that is accepted but ignored is worse than no setting.
  • Evidence: 27 of 27 chunking mutants caught.

ADR-18: Sidecars use the search gate, a sealed cursor and read-only stores

  • Context: migrated content's metadata is valuable to agents, and its object keys can name confidential things.
  • Decision:
    • Each sidecar is decided by the same DocumentAccessGate as search, using its document id and its security. Answers carry no permissions.
    • The paging cursor is sealed with AES-GCM and bound to the caller, the store and the prefix. Its TTL is 1 hour.
    • Stores are opened read-only, and only registered Ozone or S3 output connectors listed in opencrawling.sidecars.connectors can be read.
  • Principle: defence in depth.
    • The cursor holds the last key looked at, which may belong to a hidden sidecar, so plain encoding would leak it.
    • Authenticated encryption (NIST SP 800-38D) also stops a cursor from being forged or reused by someone else.
  • Evidence: 44 of 44 sidecar mutants caught.

ADR-19: Provenance records only verified identities

  • Context: an audit trail that accepts asserted identities can be poisoned.
  • Decision:
    • Every execution records how it started: trigger_source is api, dynamic-api, mcp, bpm, ois, grpc-push or startup.
    • It also records the correlation id.
    • Identity fields (triggered_by, on_behalf_of, caller_groups, caller_tenant) are filled only for verified callers; anonymous runs carry none.
    • The identity is read on the request thread, because the run's own thread has no security context.
  • Principle: a trustworthy audit trail.
  • Evidence: 6 of 6 provenance mutants caught. Delegated runs record both the agent and the user.

ADR-20: Additive and off by default

  • Decision:
    • Identity mode none, the metadata authorization mode, and every agent capability off: existing deployments behave exactly as before.
    • New ledger columns are optional, and old readers keep working.
  • Principle: safe adoption. Teams enable one capability at a time.

🔐 Security model and threat coverage

Threat Control Where
A prompt-injected agent claims another user's identity Identity from the token; arguments ignored CallerResolver (ADR-01)
A token issued for another application Issuer validation, plus audience validation once audiences is set Spring Security resource server
An agent aims the crawler at an internal host Registered, allow-listed connectors only DynamicJobService (ADR-04)
An agent maps folders it cannot read Hidden folders never opened; source-ACL decisions SourceAclVisibility (ADR-05)
Probing for hidden documents through errors or counts Uniform answers; no hidden counts; no upstream errors ADR-10
An agent hides a live document from everyone Removal only when the source reports it gone ADR-14
Flooding with jobs Per-user, per-caller and global caps (429) JobRunRegistry, ADR-11
Duplicate crawls from retries Idempotency keys per user ADR-13
An agent approves its own job Approval is REST-only and four-eyes ADR-12
A replayed or forged BPM event HMAC with a timestamp window; dedup per sender ADR-15
Ingestion that skips governance 409 for every job-API run that would use an output connector; dynamic jobs and the decoupled crawler never use one ADR-08, ADR-16
A governance hook widens permissions Protected keys restored after every hook ADR-16
A paging cursor reveals hidden keys AES-GCM sealed and caller-bound cursor ADR-18
A poisoned audit trail Only verified identities recorded ADR-19
A document with no ACL becomes public SecurityConfig.none(): nobody can read it ADR-09

🧪 Governance before embedding

Governance before embedding

opencrawling:
  governance:
    enabled: true
    hooks: pii-masking,http          # in order; a bean's name() also works
    on-rejected-update: purge        # or keep
    drop-metadata-after-gate: zammad.participants  # optional: visible to hooks, stripped before chunking/storage
    pii:
      types: EMAIL,PAYMENT_CARD,IBAN
    http:
      url: https://governance.example.com/evaluate
      secret: ${GOVERNANCE_SECRET}   # at least 32 characters
  • Transient governance metadata (drop-metadata-after-gate): Keys listed here (for example zammad.participants provided so an anonymization hook can redact names) remain visible to governance hooks and are removed right after the gate—or before chunking when the gate is off—so they are never stored downstream. Protected keys (acl, security, ois:*, etc.) cannot be dropped.

The pii-masking hook:

  • It masks email addresses, payment cards (Luhn-checked) and IBANs (ISO 13616 mod-97 checked) as [REDACTED:TYPE].
  • It never rejects a document.

The http hook contract. The hook sends POST with this JSON body:

{"documentId": "…", "uri": "…", "text": "…",
 "metadata": {"key": ["value"]},
 "security": {"inheritanceEnabled": false, "permissions": [{"identity": "…", "identityType": "…", "displayName": "…", "access": "read"}]}}

Headers:

  • X-OpenCrawling-Timestamp: epoch seconds;
  • X-OpenCrawling-Signature: sha256= followed by the lowercase hex of HMAC-SHA256(secret, timestamp + "." + body).

Reply: 200 with {"decision": "APPROVE" | "SANITIZE" | "REJECT", "text": "…", "metadata": {…}, "reason": "…", "audit": {…}}. A SANITIZE reply must include text.

Errors:

  • 429, 5xx and network errors are retried: 3 attempts, with backoff starting at 500 ms and capped at 5 s.
  • Any other status, an invalid reply, or a reply over 32 MiB makes the document unavailable, so nothing is embedded.

Full settings: Configuration Guide, Pre-Embedding Governance.


🔔 Event-driven ingestion from business processes

opencrawling:
  events:
    bpm:
      enabled: true
      sources:
        - name: flowable-prod
          hmac-secret: ${BPM_FLOWABLE_SECRET}   # at least 32 characters
        - name: aps-prod
          token: ${BPM_APS_TOKEN}               # Basic auth: aps-prod:<token>
      routes:
        - name: contract-signed
          engine: flowable                      # blank = any engine
          process: contractApproval
          transition: signed
          connector: Alfresco_Main
          action: UPSERT
        - process: recordsDisposal
          transition: destroyed
          connector: Alfresco_Main
          action: DELETE

The first matching route wins. An event that no route matches gets 200 with status: IGNORED. Signing an event from a shell, for example to test a route:

ts=$(date +%s)
body='{"eventId":"evt-7f3a","engine":"flowable","processDefinitionKey":"contractApproval","transition":"signed","documentId":"<repository id>"}'
sig=$(printf '%s.%s' "$ts" "$body" | openssl dgst -sha256 -hmac "$BPM_FLOWABLE_SECRET" | sed 's/^.*= //')
curl -sS https://opencrawling.example.com/api/events/bpm/document-transition \
  -H 'Content-Type: application/json' \
  -H 'X-OpenCrawling-Source: flowable-prod' \
  -H "X-OpenCrawling-Timestamp: $ts" \
  -H "X-OpenCrawling-Signature: sha256=$sig" \
  --data-binary "$body"

The answer: {status, eventId, route, connector, document, jobId, executionId, note, duplicate}.

A repeated eventId: it returns the first answer, with duplicate: true.

Full settings: Configuration Guide, BPM Document Events and Document Lifecycle Actions & Tombstones.


🗂️ Reading migrated metadata

Content migrated to Apache Ozone or AWS S3 has an OIS sidecar (<key>.ois.json, written with OisSidecar conforming to schemas/document.schema.json) next to each binary. SidecarService reads both current OIS-conformant sidecars (content with sha256 and sizeBytes) and earlier contentRef (checksumSha256, contentLength) sidecars, flattening typed/nested metadata blocks (such as ois:provenance and ois:governance) into <key>.<field> string lists in SidecarView.

What the reader returns: readOisSidecars (REST GET /api/sidecars/{connector}) returns the sidecars the caller may read, in key order. Each one gives:

  • the id, URI and source (including source.instance stripped of credentials, query, and fragment);
  • the file name, MIME type, size and SHA-256 checksum;
  • the metadata (including pipeline-stamped ois:provenance, and ois:governance when governed);
  • page totals by MIME type and by source type.

What it never does: it never downloads binaries and never returns permissions.

What it skips: keys under .tombstones/ and .opencrawling/, anything that is not an UPSERT, invalid sidecars, and oversized sidecars.

See Model Context Protocol, OIS sidecars.


📊 Audit and provenance

Every execution, started by an agent or not, records its provenance. It goes to the run summary (summary_<executionId>.json, as a nested provenance object) and to every row of the Parquet/Avro ledger (plus document-level ois:source.*, ois:provenance.*, and ois:governance.* metadata stamped by JobOrchestrator and IngestionConsumer via OisMetadata and protected from source spoofing or hook modification by GovernanceGate, recording onBehalfOf / actorAgent only when callerVerified is true):

Ledger column Meaning
trigger_source api, dynamic-api, mcp, bpm, ois, grpc-push or startup
triggered_by the acting agent (act.sub) for delegated tokens, otherwise the verified user; bpm:<source> for a BPM source authenticated by HMAC or static token
on_behalf_of the user, when an agent acted for them
caller_groups, caller_tenant the verified caller's groups and tenant
caller_verified whether the identity came from a verified token or an authenticated BPM source
correlation_id from X-Correlation-Id, the MCP correlationId, or the BPM eventId; dropped if it fails ^[A-Za-z0-9._:/@-]{1,128}$

See Lakehouse Audit Reporting & Ledger.


⚙️ Deploying it

Profiles

Profile Enable For
Secure search security.identity.mode: jwt Agents that only read the index
Discovery + discovery.enabled Agents that plan before asking for content
Agentic ingestion + jobs.dynamic.enabled, connectors, admin-groups Agents that start crawls
Governed + governance.enabled, hooks Regulated content
Event-driven + events.bpm.enabled, sources, routes Process-triggered freshness
opencrawling:
  security:
    identity:
      mode: jwt
      protect-api: true                 # REST needs a token too; set admin-ui.client-id when using the admin UI
      admin-ui:
        client-id: opencrawling-admin-ui # optional: OIDC PKCE sign-in for oc-admin-ui (see Security-and-Identity-Management)
    authorization:
      mode: metadata                    # or shadow, intersection, openfga
  discovery:
    enabled: true
  jobs:
    max-concurrent-runs: 20             # 0 = no cap
    max-concurrent-runs-per-caller: 4   # per acting agent
    dynamic:
      enabled: true
      connectors: Alfresco_Main         # registered connectors agents may crawl
      admin-groups: oc-admins
      require-approval: true
  governance:
    enabled: true
    hooks: pii-masking
  sidecars:
    enabled: true
    connectors: Lake_S3
spring:
  security:
    oauth2:
      resourceserver:
        jwt:
          issuer-uri: https://idp.example.com/realms/corp
          audiences: opencrawling

Recommended rollout

This is our recommendation, not a requirement:

  1. Turn on jwt mode first. Check that search results are unchanged for real users. If your provider sends full group paths (/finance), turn that off or write grants with the full path.
  2. Run shadow mode before openfga. The metadata rules keep deciding while disagreements are counted (oc.authz.shadow.mismatch).
  3. Enable discovery, then dynamic jobs with require-approval: true and a short connector allow-list. Relax approval once you trust your scopes and limits.
  4. Add governance before you widen the allow-list to sensitive repositories.
  5. Use user: / group: prefixes in grants where a user and a group can share a name (see Known limitations).

✅ How it is verified

In the test suite

  • Discovery and search agree (DiscoveryVersusSearchTest).
    • A deterministic 400-document corpus covers case variants, user: and group: prefixes, bare identities, denies, public grants, inheritance flags, empty ACLs and missing security objects.
    • It goes through discovery as nodes, and through the real ingestion consumer, the writer's permission fields and a JSON round trip for search.
    • For 8 callers, both paths return the same documents.
    • The store pre-filter never hides a document the gate allows.
  • Agents stay apart (AgentSwarmHttpTest).
    • The test runs real MCP over Streamable HTTP with RS256-signed delegated tokens.
    • 4 sub-agents work for 2 users, 12 rounds each, in parallel, and every call forges the other user's identity.
    • Each tool asks for its own scope.
    • A token signed by another key opens no session.
  • Each feature carries unit and context tests, plus the mutation results listed in How we built it.

Live, end to end

Live end-to-end verification

The stack:

  • two machines: one ran the runtime and its services, the other Kafka, pgvector, Ollama, Apache Ozone and OpenFGA;
  • a real Alfresco Content Services 26.1 repository;
  • jwt identity and openfga authorization;
  • governance through the HTTP hook.

The run:

  • An agent with a delegated token discovered a test folder and crawled it for alice.
  • Governance approved four documents, masked an email address and an IBAN in a fifth, and rejected a draft.
  • Each user then searched.
  • In a separate check, a membership written directly into OpenFGA gave bob the finance documents at once, and removing it took them away, with no re-crawl.

What it found:

  • Alfresco DENIED entries were dropped. The Alfresco crawler ignored DENIED, so carol could find a document Alfresco denies her. Fixed by #265.
  • Search without dimensions found nothing, because it read the wrong pgvector table. Fixed by #266.

Both fixes are merged. The run was repeated with them applied: carol lost the document in search and in getDocumentContent, and OpenFGA recorded the deny.


🧾 From proposal to delivery

The issue proposed What was delivered Why it differs
explore_repository_tree, query_repository_dryrun with userPrincipal arguments exploreRepositoryTree, queryRepositoryDryRun, listRepositoryConnectors camelCase like the existing tools; identity from the token (ADR-01)
Dry runs for CMIS and OnBase CMIS and Alfresco (Hyland CIC later added discovery too) There is no OnBase connector yet (#129)
read_ois_sidecars with a bucket argument readOisSidecars, listSidecarStores on registered stores Agents never name storage locations (ADR-04)
POST /api/jobs/dynamic with a full connector configuration and endpoint A registered connector plus a start folder, idempotency key and chunking SSRF and credential exfiltration (ADR-04)
trigger_dynamic_job MCP tool triggerDynamicJob plus status, list and stop tools Agents must follow and stop what they start
POST /api/jobs/single-document POST /api/jobs/dynamic/documents, ingestDocument, removeDeletedDocument Removal guarded by the source (ADR-14)
pipelineMode, output collection, ephemeral Always RAG into the deployment's pipeline; status kept 24 h One governed path (ADR-08)
chunkSize + chunkOverlap chunkSize and three real splitter settings The splitter has no overlap (ADR-17)
PreEmbeddingGovernanceHook.evaluate(document, security) evaluate(GovernanceSubject) with read-only security Hooks see extracted text; permissions immutable (ADR-16)
AxMirror interceptor A generic signed HTTP hook No public AxMirror API contract; any governance service can implement the hook
BPM webhook for Flowable and APS Generic event, admin routes, three authentication methods ADR-15
"Sub-second" freshness Seconds We do not promise what a crawl, extraction and embedding cannot guarantee
Swarm and 4-phase integration tests AgentSwarmHttpTest, DiscoveryVersusSearchTest, and a live two-machine run
Keycloak token exchange Any OpenID Connect provider; claims configurable Not tied to one vendor

🚧 Known limitations and follow-ups

We list these openly, with their status and their impact.

Limitation Impact Status
The Alfresco crawler ignores DENIED entries, lets an EVERYONE grant drop every other rule (denies included), and reads missing permissions as public A user denied in Alfresco but allowed through a group can find the document Fixed in #265 (re-crawl Alfresco sources to correct existing entries)
Search, document content and source listing read the plain vector_store table unless dimensions is passed Agents calling without dimensions find nothing in pgvector Fixed in #266
identityType is not used: a bare identity matches a user or a group of the same name, in the metadata rules and in OpenFGA A user grant alice also matches a role named alice Fixed in PR #286 (EffectiveAcl resolves identityType to user: / group: prefixes; re-crawl existing sources)
The OpenFGA sync keeps tuples for documents governance rejected No exposure (nothing is embedded), but stale tuples Fixed in #268, PR #282
A governance hook that is unavailable is not retried automatically The document is not embedded until the next crawl By design (fail closed); a retry channel is future work
Dynamic job status and pending approvals live in memory Lost on restart; the ledger keeps the durable record Known
Search decisions use the token's principal and groups, not its tenant Documents have no tenant dimension; the tenant scopes job ownership and quotas Known
The admin UI cannot send bearer tokens Keep protect-api off where the UI is used Fixed in PR #304 (#187): configure admin-ui.client-id for OIDC PKCE sign-in (see Security and Identity Management)
RFC 9728 protected-resource metadata is not served MCP clients must be configured with the identity provider Future work
The observability MCP tools are always on. queryThroughputMetrics reports made-up headline figures, and fetchJobTraces invents traces for jobs it has no spans for Agents should not treat those figures as facts Pre-existing
Turning gRPC push off stops startup; the Tika separate-process mode fails outside the Docker image Found during the live run; unrelated to agents Fixed in #269, PR #285 and #270, PR #284
The admin UI's global chunk size and overlap settings are saved but never used Only per-job chunking (ADR-17) changes how documents are split Fixed in #271, PR #283

🩺 Troubleshooting

Symptom Likely cause
401 on /mcp or a tool No bearer token, an expired token, or a token from an unknown key in jwt mode
403 The token does not carry the scope… Missing oc:jobs:write, oc:jobs:run, oc:sidecars:read or oc:events:write
404 for a connector that exists It is not listed in jobs.dynamic.connectors or sidecars.connectors (answers like a missing one)
400 for a folder the agent expects Outside the connector's root, excluded, unreadable by the user, or missing (same answer)
409 when starting an admin job Governance is on. The job API always uses an output connector, so every non-migration run is refused. Crawl with the decoupled crawler containers or dynamic jobs, or turn governance off.
409 on removal The document still exists in the repository
429 max-running-per-user, max-pending-per-user or the runtime run caps
Search returns nothing in pgvector Check that the embedding service is running and that dimensions, if passed, matches the embedding model
A user sees less in discovery than in search Expected when their access comes only from an OpenFGA membership (ADR-05)
Keycloak groups do not match grants The "Full group path" mapper sends /finance; turn it off or grant /finance
BPM 401 …outside the accepted window The sender's clock is more than max-clock-skew (5 min) off

📚 Reference

Standards and sources:

🗺️ Documentation


🎓 Tutorials


🧪 Testing


🤝 Community

Clone this wiki locally