Repository navigation
Zero Trust Data Provider Agent
OpenCrawling started as an ingestion and vector-search engine. With issue #203 it also became a data provider that AI agents can drive safely. An agent can now:
- look inside an enterprise repository before ingesting anything;
- decide what to ingest and start the crawl itself;
- have business processes push single documents;
- send content through governance before it is embedded;
- read migrated metadata, and search.
Every one of those steps is decided for the user the agent works for, against the permissions the source repository enforces.
This page explains what was built, how it was built, and why each design decision was taken. The reference pages linked at the end hold every setting and endpoint.
Note
Status.
- Merged: delivered in 19 merged pull requests (#228 to #262), all reviewed and merged by the maintainer.
-
Fixed: the two defects found by the live end-to-end run are fixed on
main: #265 (AlfrescoDENIEDentries) and #266 (search withoutdimensions). Other follow-ups are listed under Known limitations. - Off by default: every agent capability is off until an administrator enables it.
- The problem in one minute
- Architecture
- The active ingestion cycle
- How we built it
- Architecture decisions
- Security model and threat coverage
- Governance before embedding
- Event-driven ingestion from business processes
- Reading migrated metadata
- Audit and provenance
- Deploying it
- How it is verified
- From proposal to delivery
- Known limitations and follow-ups
- Troubleshooting
- Reference
Issue #203 named three problems that stop enterprises from connecting agents to their repositories:
| Problem | What it looks like | What OpenCrawling does now |
|---|---|---|
| Context leaks | Agent tools search without knowing who asked, so HR, legal or finance content reaches the wrong person. | Every call carries a verified identity. Every answer is filtered by that user's source permissions, and an explicit deny always wins. |
| Batch latency | A contract signed at 10:00 is invisible to agents until the nightly crawl. | A BPM engine posts an event and that one document is re-ingested within seconds. |
| Passive ingestion | A human must configure every crawl in advance, so an agent can only use what someone guessed it would need. | The agent discovers what exists, previews it, and starts a bounded crawl of exactly the folder or document it needs. All of this happens within what an administrator allowed. |
The diagram at the top of the page shows the four layers. Read it left to right:
- Callers: a user, the agents acting for them, BPM engines, and human reviewers.
-
Identity and policy: one OAuth2 resource server.
- It turns a bearer token into a caller: the principal, the groups, and the acting agent from
act.sub. - It ignores any identity claimed in tool arguments.
- It checks the scope each capability needs.
- It turns a bearer token into a caller: the principal, the groups, and the acting agent from
-
Agent capabilities: six independent features.
- Search is the only one on by default.
- Each feature is exposed as MCP tools and as REST endpoints.
-
Data plane: the decoupled Kafka pipeline that already existed.
- Content goes to the claim-check store, then the documents topic, extraction, the governance gate, chunking, embedding, and the vector writer.
- The authorization gate and OpenFGA sit next to the pipeline.
Two properties make the design hold together:
- Every path an agent can trigger feeds the same pipeline. It goes through the documents topic, so governance and the OpenFGA permission sync see every document without extra code.
- Every answer passes one of two deciders, both using the same rules. They are described in Security model.
The issue described the "apex pattern" as discover, decide, define, execute. In the delivered system that becomes six steps, each with its own control:
| Step | MCP tools | REST | What protects it |
|---|---|---|---|
| 1. Discover |
listRepositoryConnectors, exploreRepositoryTree
|
GET /api/discovery/connectors, GET /api/discovery/{connector}/tree
|
Decided on the ACL the repository reports now. Hidden folders are never opened. |
| 2. Preview | queryRepositoryDryRun |
POST /api/discovery/{connector}/dry-run |
Honest counts: EXACT, LOWER_BOUND or ESTIMATE. A query returns no count unless reveal-hidden-counts is on. |
| 3. Ingest |
triggerDynamicJob, ingestDocument, removeDeletedDocument
|
POST /api/jobs/dynamic, POST /api/jobs/dynamic/documents
|
Allow-listed connectors, start folder in scope and readable, idempotency keys, per-user limits. |
| 4. Approve (optional) | none |
POST /api/jobs/dynamic/{jobId}/approve, …/reject
|
Another admin, never the requester. Re-checked under the requester's identity. |
| 5. Govern | none (automatic) | none | Ordered hooks before chunking. Permissions are immutable. Fails closed. |
| 6. Use |
secureVectorSearch, getDocumentContent, listAccessibleSources
|
none | The store pre-filter narrows candidates, then the gate re-checks every hit. |
Agents follow their jobs with getDynamicJobStatus, listDynamicJobs and stopDynamicJob (GET /api/jobs/dynamic, GET /api/jobs/dynamic/{jobId}, POST /api/jobs/dynamic/{jobId}/stop).
Here an agent works for alice over MCP with a delegated token. The identity comes from the token, so no userPrincipal is sent. The values below are illustrative, but the field names are the real ones.
1. Discover. The agent explores the /Finance folder.
2. Preview. It asks how many documents a crawl of /Finance/2026 would take.
// tools/call queryRepositoryDryRun
{"connector": "Alfresco_Main", "scope": "/Finance/2026", "sampleSize": 3}
// result: a folder dry run checks every document, so the count is alice's own
{"connector": "Alfresco_Main", "matchCount": 42, "countAccuracy": "EXACT", "samples": […],
"hiddenCount": null, "truncated": false, "truncationReason": null, "note": null}3. Ingest. It starts the crawl, with its own chunk size and an idempotency key.
// tools/call triggerDynamicJob
{"connector": "Alfresco_Main", "start": "/Finance/2026", "idempotencyKey": "fin-2026-q3", "chunkSize": 400}
// result
{"jobId": "dyn-3f9c2a1b7e40", "executionId": "exec-5d1e9a02",
"statusUrl": "/api/jobs/dynamic/dyn-3f9c2a1b7e40", "state": "RUNNING", "replayed": false}4. Follow. It checks the job until it ends.
// tools/call getDynamicJobStatus {"jobId": "dyn-3f9c2a1b7e40"}
{"jobId": "dyn-3f9c2a1b7e40", "kind": "FOLDER", "state": "COMPLETED", "documents": 42,
"owner": "alice", "triggerSource": "mcp", "approvedBy": null, "chunking": {"chunkSize": 400, …}, …}5. Use. secureVectorSearch now returns chunks of those documents, but only to users their source ACL allows. That includes alice, and anyone else the source allows; the agent's request did not change who can read them.
Three working rules shaped the delivery:
-
Build the trust layer before giving agents any power.
- Before #203, MCP tools trusted whatever identity the client sent, and stopping a job only changed a status string.
- Letting agents start crawls on top of that would have been unsafe.
- So the first phases made the caller verifiable and job control real. Discovery, ingestion, governance and sidecars all build on those two foundations.
-
Small, reviewable pull requests in dependency order. Each PR:
- delivered one capability;
- carried its tests and docs;
- passed the full CI before merge, including the 63-suite integration matrix.
-
Prove that the tests would catch a regression.
- Coverage shows that code ran. It does not show that a test would notice if the code were wrong.
- So every feature PR was mutation-tested: we broke the new rules on purpose, one by one, and confirmed that a test failed each time.
| Phase | Why at this point | Pull requests | What it established |
|---|---|---|---|
| 0. Prerequisites | Agents would inherit any flaw in how identities match | #228, #230 |
user: grants match only principals and group: grants only groups. The pgvector pre-filter applies the same rule as the gate. |
| A. Foundations | No agent feature is safe without a verified caller and real job control | #231 (#187), #232, #233, #235, #238 | Bearer-token identity (jwt mode), connectors built by a factory (an unknown class no longer silently crawls the file system), real stop, run caps, run provenance, per-caller quotas |
| B. Discovery | An agent must see before it acts, without reading content | #242, #243, #244 | A discovery SPI with CMIS and Alfresco implementations. REST and MCP tools filtered by the live source ACL. |
| C. Agent ingestion | Act on what discovery found, within limits | #250, #251, #253, #256, #257 | Dynamic folder jobs, single-document ingest and removal, BPM events, optional human approval, per-job chunking |
| D. Governance | Agents widen what gets ingested, so content must be checked before it is embedded | #258 | Pre-embedding gate with PII masking, a signed HTTP hook and an SPI |
| E. Sidecars | Migrated content must be usable without downloading binaries | #259 | An OIS sidecar reader behind the same gate as search |
| F. Verification | Prove the invariants end to end, not just per feature | #260, #262 | Discovery-versus-search agreement, a multi-agent swarm over real MCP, and "no security object" never meaning public |
| PR | Capability | Deliberate breaks (mutants) caught |
|---|---|---|
| #231 | Verified caller (#187) | 4 / 4 |
| #232 | Connector factory | 4 / 4 |
| #233 | Run registry, real stop | 5 / 5 |
| #235 | Run provenance | 6 / 6 |
| #238 | Per-caller quotas | 6 / 6 |
| #242 | Discovery SPI + CMIS | 9 / 9 |
| #243 | Discovery REST + MCP | 10 / 10 |
| #244 | Alfresco discovery | 13 / 13 |
| #250 | Dynamic jobs | 18 / 18 |
| #251 | Single document | 23 / 23 |
| #253 | BPM events | 19 / 19 |
| #256 | Human approval | 13 / 13 |
| #257 | Per-job chunking | 27 / 27 |
| #258 | Governance gate | 47 / 47 |
| #259 | OIS sidecars | 44 / 44 |
| #260 | Agreement and swarm tests | 7 / 7 |
| Total | 255 / 255 |
In some cases a mutant survived because the check it broke could never be reached. We then removed the redundant check rather than keeping untested code. Each PR description records its mutants and its test counts.
Each decision below gives:
- the problem it solves;
- the principle it applies;
- the alternatives we rejected, and why;
- the cost we accepted;
- the evidence that enforces it.
The decisions were agreed before or during implementation, and the issue and PR threads record them.
| # | Decision | Principle |
|---|---|---|
| 01 | Identity comes from a verified token, never from tool arguments | Complete mediation; never trust the client |
| 02 | Delegation is explicit: user as subject, agent as actor | Accountability (RFC 8693) |
| 03 | One scope per capability | Least privilege |
| 04 | Agents name registered connectors, never endpoints or credentials | SSRF and confused-deputy prevention |
| 05 | Discovery decides on the live source ACL in every mode | Fail-safe under-showing |
| 06 | Indexed content has one enforcement point | Complete mediation |
| 07 | Documents keep source permissions, whoever triggers the crawl | Permissions follow the data |
| 08 | Agent jobs run only on the decoupled pipeline | One governed path |
| 09 | Fail closed; no permissions means nobody | Fail-safe defaults |
| 10 | Hidden looks exactly like missing | No existence oracle |
| 11 | Everything is bounded and says when it was truncated | Availability; honest results |
| 12 | Human approval: optional, four-eyes, REST-only, re-checked | Separation of privilege |
| 13 | Retries are safe by construction | Idempotency |
| 14 | Removal only when the source says the document is gone | Integrity |
| 15 | BPM: routes authorize; three ways to authenticate | Meet real engines where they are |
| 16 | Governance between extraction and chunking, impossible to bypass | Complete mediation; fail closed |
| 17 | Per-job chunking exposes only what the splitter really does | No false controls |
| 18 | Sidecars: same gate, sealed cursor, read-only stores | Defence in depth |
| 19 | Provenance records only verified identities | Trustworthy audit |
| 20 | Additive and off by default | Safe adoption |
-
Context:
- The MCP tools took
userPrincipalanduserRolesas arguments. - An agent, or a prompt injected into it, could claim to be anyone, and the server had no way to tell.
- The MCP tools took
-
Decision:
-
opencrawling.security.identity.mode=jwtmakes the runtime an OAuth2 resource server. Spring Security validates the signature, expiry and issuer, and the audience whenspring.security.oauth2.resourceserver.jwt.audiencesis set. Set it: without it, the runtime warns that it accepts any token from the issuer. -
CallerResolverbuilds the caller from the token: the principal fromprincipal-claims, the groups fromgroup-claims, and the tenant. - The identity arguments are then ignored.
- Every new tool must get its caller from
CallerResolver.
-
- Principle: complete mediation. The client's claim about itself is never the authority. This also matches the MCP authorization model, in which the MCP server is an OAuth 2.1 resource server.
-
Alternatives rejected:
- Keep asserted arguments: they can be spoofed by construction.
- A static API key per agent: it proves which agent is calling, but not which user it acts for. Every agent would then need the union of its users' rights.
-
Cost:
- It needs an identity provider. The default stays
none, so existing deployments are unaffected (ADR-20). - With
protect-apienabled, setopencrawling.security.identity.admin-ui.client-idso the admin UI signs users in via OpenID Connect PKCE (see Security and Identity Management).
- It needs an identity provider. The default stays
-
Evidence:
- In
AgentSwarmHttpTest, every call forges the other user's identity in its arguments, and every answer is still the token user's view. - A token signed by an unknown key opens no MCP session.
- In
-
Context:
- An agent acts for a person.
- Treating the agent as its own principal would make it a confused deputy: it would hold broad rights and decide alone whose data to show.
-
Decision:
- Agents present a token obtained by OAuth 2.0 Token Exchange (RFC 8693).
- The token's subject is the user, and the
act.subclaim names the agent. - Access is decided for the user. The agent is recorded as
triggered_byand the user ason_behalf_of.
- Principle: accountability without privilege escalation. The agent never has more access than the user it serves.
-
Alternatives rejected:
- Pass the user's own token through: there would be no record of which agent acted.
- A service account for the agent: it over-privileges the agent and turns it into a confused deputy.
-
Cost: it depends on the identity provider supporting token exchange and putting the agent in
act. -
Evidence: a delegated job records
research-orchestratorastriggeredByandaliceas owner. The same orchestrator acting forbobgets the same answer as for a job that does not exist: a tool error over MCP,404over REST. It also cannot see alice's job in its list, or start a crawl on her folder.
-
Context: a workflow service that refreshes documents should not also be able to crawl whole folders, and a research agent that reads sidecars should not be able to start jobs.
-
Decision: scopes are read from the
scopeorscpclaim:Scope What it allows oc:jobs:writevalidate and start folder jobs oc:jobs:runsingle documents oc:sidecars:readsidecars oc:events:writeBPM events sent with a bearer token - Following and stopping your own jobs accepts either job scope.
- Discovery and search need no scope, because the data ACL already limits them to what the user may read.
-
Principle: least privilege.
-
Alternatives rejected: one "agent" scope. A leaked token would then grant everything.
-
Cost: administrators configure scopes in the identity provider.
-
Evidence:
AgentSwarmHttpTest.eachToolAsksForItsOwnScope(oc:sidecars:read,oc:jobs:write),DynamicJobServiceTest.eachOperationNeedsItsOwnScopeAndEitherScopeFollowsJobs(oc:jobs:run, either scope follows jobs) andBpmEventServiceTest.aTokenCallerNeedsTheEventScope(oc:events:write).
- Context: the issue's sample payload let an agent pass a repository endpoint URL and connector configuration.
-
Decision:
- An agent names a connector registered in
connectors.jsonand listed inopencrawling.jobs.dynamic.connectors, plus a start folder or a document id. That is all it can choose. - A connector that is not listed answers exactly like one that does not exist.
- The connector's own credentials, root, exclusions and filters always apply.
- An agent names a connector registered in
- Principle: prevent SSRF (OWASP Top 10, A10) and credential exfiltration. An agent that could choose the host would make OpenCrawling probe internal networks, or send its credentials to a server the attacker controls.
- Alternatives rejected: validate agent-supplied URLs against an allow-list. It is fragile (DNS rebinding, redirects) and still lets agents mix credentials and hosts.
- Cost: new sources need an administrator. We consider that a feature.
-
Evidence:
DynamicJobServiceTestcovers unlisted, unregistered and non-discoverable connectors;CmisDiscoveryTestcovers connectors that crawl a query or a change log, which have no crawl start.
-
Context:
- OpenFGA only holds tuples for documents that have already been indexed.
- In
openfgamode, deciding discovery through the configured provider would hide every folder, and every document not yet ingested. Discovery would then be useless exactly when it is needed.
-
Decision: discovery, dry runs, the start folder of a dynamic job and single-document ingests are decided by
SourceAclVisibility. It applies the metadata rules to the ACL the repository reports now, whateveropencrawling.security.authorization.modeis. An item with an unknown ACL is hidden. - Principle: fail-safe defaults. Memberships held only in OpenFGA are not applied here, so discovery can show less than search, never more.
-
Alternatives rejected:
- The configured provider: it hides everything that is not indexed in OpenFGA modes.
- The indexed ACL: there isn't one before ingestion, and it would be stale.
- Cost: a user whose access comes only from an OpenFGA membership may not see a folder in discovery that search would show them.
-
Evidence:
DiscoveryVersusSearchTestruns a 400-document corpus with mixed permissions down both paths for 8 callers, with search inmetadatamode, and both return the same set. That proves both paths apply the same rules. Documents with no security are checked separately and are hidden both ways. The OpenFGA side of this decision is a design argument: OpenFGA has no tuples for content that is not indexed.
- Context: with ten search backends and a sidecar reader, separate ACL checks per feature would drift apart over time.
-
Decision:
- Every MCP search backend and the sidecar reader pass their candidates to one
DocumentAccessGate, which asks the configured provider:metadata,shadow,intersectionoropenfga. - The store's pre-filter (
security_read_keys) only narrows the candidates. - The gate decides every hit. In
openfgamode, where tuples can allow what the stored metadata does not, the pre-filter is skipped, so it can never hide an allowed hit.
- Every MCP search backend and the sidecar reader pass their candidates to one
- Principle: complete mediation, with defence in depth.
- Alternatives rejected: trust the store's filter. Stores differ in filter semantics, and some have none.
-
Cost: a batch check per page. In
openfgamode the gate also over-fetches candidates, bounded byoverfetch-factor(10) andcandidate-budget(500). -
Evidence: the pre-filter safety check in
DiscoveryVersusSearchTestproves the pre-filter never hides a document the gate allows (inmetadatamode).
- Context: an agent may start a crawl of a folder that contains documents its user cannot read.
-
Decision:
- The crawl takes everything the connector's own rules take, and every document is indexed with its own source permissions.
- The requester's access decides only whether the crawl may start: the start folder must be readable by them.
- Principle: permissions follow the data. The index is shared by all users, so it must hold each document's real ACL. A copy filtered for one requester would be wrong for everyone else.
- Alternatives rejected: index only what the requester can read. That produces an index whose content depends on who asked first.
- Cost: an agent can cause documents it cannot read to be ingested. They stay invisible to it, and the optional approval gate can review such requests.
-
Evidence: the live run in How it is verified: a crawl started for
alicemade each user see exactly their own documents.
- Context: in the monolithic path, an output connector embeds content itself. That would bypass both the governance gate and the OpenFGA sync.
-
Decision:
- Dynamic jobs run with no output connector, always in RAG mode.
- Every document goes to the documents topic, and from there through ingestion, governance and the permission sync.
- Principle: one governed path. The OpenFGA sync then needs no extra code for new ingestion routes.
- Alternatives rejected: let agents pick the pipeline mode or the output. That gives a second, ungoverned route.
- Cost: agents cannot run migration jobs or choose a destination. Those remain administrator decisions.
- Evidence: the dynamic-job tests assert that no output connector is used.
- Context: with permissive defaults, every misconfiguration becomes a data leak.
-
Decision:
- An unverifiable caller gets
401. - An authorization provider that cannot decide returns nothing: REST answers
503, and MCP calls fail. - A governance hook that cannot decide means nothing is embedded.
- A misconfigured gate stops the runtime at startup.
- A document with no security object is
SecurityConfig.none()(#261, #262). Only its legacyaclstring can grant access, and with a blankaclnobody can read it.
- An unverifiable caller gets
- Principle: fail-safe defaults (Saltzer and Schroeder).
- Alternatives rejected: defaulting to public for compatibility. That was the behaviour #262 removed.
- Cost: misconfigured sources produce invisible documents instead of leaked ones. That is diagnosable, and the opposite would not be.
-
Evidence:
IngestionMessageTest,DiscoveryVersusSearchTest, and the startup-failure tests for the gate and the HTTP hook.
- Context: an error that says "forbidden" confirms that something exists.
-
Decision:
- A folder or document the caller cannot read, one outside the connector's root, and one that does not exist all get the same answer.
- Hidden counts are off by default (
reveal-hidden-counts: false). - Repository and storage errors are logged and never passed to the caller (
502with a fixed message). - Sidecar pages never say how many sidecars were hidden.
-
Principle: no existence oracle. RFC 9110 explicitly lets a server answer
404to hide a resource it will not show. - Cost: harder debugging for callers. The server log keeps the detail.
- Evidence: "same answer" assertions in the discovery, dynamic-job, single-document and sidecar tests.
-
Context: an agent can ask for the whole repository in one call.
-
Decision: every walk, scan and call has a per-request limit and a hard ceiling:
Area Limits Discovery max-depth3,max-nodes200,max-scan10,000,timeout30 s, each with a hard ceilingSidecars 50 per page (at most 200), 5,000 keys scanned, 8 MB per answer Governance HTTP calls 30 s per attempt, 3 attempts Jobs max-running-per-user(2),max-pending-per-user(10), plus the runtime's run caps (429)- Results say when they stopped early (
truncated,truncationReason), and counts say how exact they are (EXACT,LOWER_BOUND,ESTIMATE).
- Results say when they stopped early (
-
Principle: availability, and never presenting partial data as complete.
-
Cost: agents must page or narrow their requests on large repositories.
-
Evidence: the bound tests in each feature. A discovery timeout returns
200withtruncationReason: TIMEOUT, not an error.
- Context: some organisations want a person to approve what agents ingest. Others want full automation.
-
Decision:
- With
require-approval: true, every folder job or document ingest first passes every normal check, then waits inPENDING_APPROVAL. This applies whether an agent or a person asked for it. - A member of
admin-groupswho is not the requester approves or rejects it over REST. - On approval the request is checked again under the requester's identity, and then starts under the same job id.
- Unanswered requests expire after 24 hours.
- There is no MCP tool for approval.
- With
- Principle: separation of privilege. An agent holding an approval tool could approve its own work. Re-checking at approval time closes the time-of-check/time-of-use gap while the job waited.
- Alternatives rejected: approval always on: it blocks automated deployments. Approval as an MCP tool: it defeats the purpose.
- Cost: pending state is held in memory, like job status (see Known limitations).
- Evidence: 13 of 13 approval mutants caught. The tests cover self-approval, approval by a non-admin, and access that changed while the job waited.
- Context: agent loops retry tool calls, so the same crawl could start twice.
-
Decision:
- An
idempotencyKey(RESTIdempotency-Keyheader) belongs to the user. It is fingerprinted with the whole request, chunking included. - The same key with the same request returns the first job (
"replayed": true). - The same key with a different request is refused (
409). - A refused start does not use up the key.
- An
- Principle: idempotent operations, following the IETF Idempotency-Key HTTP header draft.
-
Evidence: the idempotency tests in
DynamicJobServiceTest.
- Context: a remove request is a way to hide a live document from every user's search.
-
Decision:
-
removeDeletedDocumentand BPMDELETEroutes publish a tombstone only when the repository reports the document gone. - Asking to remove a live document you can read gets
409. - Every other request gets the same
202, so the answer reveals nothing.
-
- Principle: integrity. Agents and workflows can keep the index in step with the source, but cannot make it diverge from it.
- Evidence: the single-document tests for live, gone, hidden and out-of-scope documents.
-
Context:
- A BPM engine's service account rarely appears in document ACLs.
- Neither Flowable nor Alfresco Process Services signs requests or fetches OAuth2 tokens out of the box. Flowable's HTTP task sets headers from expressions, so HMAC needs a small signing bean. The APS REST call supports Basic auth.
-
Decision:
- Routes authorize: an administrator's route (engine, process, transition → connector, action) authorizes the event. The connector still limits the document to its scope.
-
Three ways to authenticate:
- HMAC-SHA256 over
timestamp + "." + body, with a 5-minute window, compared in constant time, in the same timestamped scheme Stripe-style webhooks use; - a per-source static token over Basic auth;
- a bearer token with
oc:events:write.
- HMAC-SHA256 over
- Bounded: secrets must be at least 32 characters, events are deduplicated per sender for 24 hours, and bodies are limited to 64 KiB.
- Principle: meet real engines where they are, without weakening the model. Basic is the weakest of the three, so use HMAC or a bearer token where you can, and always TLS.
- Cost: HMAC from Flowable needs a small signing bean.
- Evidence: 19 of 19 BPM mutants caught. The tests cover bad signatures, stale timestamps, duplicate events and missing scopes.
- Context: regulated content must be checked, and possibly redacted, before any vector is computed from it.
-
Decision:
- In
IngestionConsumer, after Tika extraction and before chunking, an ordered chain of hooks (pii-masking,http, or anyPreEmbeddingGovernanceHookbean) answers APPROVE, SANITIZE or REJECT. - Protected keys (
acl,security,security_*,acl_*,sys_acl,documentId,uri,lastModified) are restored after every hook. - A hook that fails means nothing is embedded.
- With governance on, runs that would embed through an output connector are refused (
409). That means every non-migration run started through the job API. - A rejection removes earlier chunks and OpenFGA relationship tuples unless
on-rejected-update: keep(#268, PR #282).
- In
- Principle: complete mediation and fail-safe defaults. Purging on reject means the system stops serving text that governance no longer approves.
-
Alternatives rejected:
- Govern after embedding: the vectors already encode the content.
- A hook may edit permissions: it would turn governance into an access-widening channel.
-
Cost:
- While governance is on, the job API refuses every non-migration run (
409). It always hands documents to an output connector, the runtime's own one when the job names none. Crawls then run through the decoupled crawler containers or dynamic jobs. - Outcomes are kept as a per-document
Governancespan in the runtime's trace store, not in the execution ledger: the consumer may run in its own container, with no ledger for the crawl.
- While governance is on, the job API refuses every non-migration run (
- Evidence: 47 of 47 governance mutants caught.
-
Context: the issue asked for
chunkSizeandchunkOverlap. Spring AI'sTokenTextSplitterhas no overlap. Its "max chunks" option silently puts the rest of a document into one oversized chunk. -
Decision:
- A job may set
chunkSize,minChunkSizeChars,minChunkLengthToEmbedandkeepSeparator. -
chunkSizemust stay withinmin-chunk-sizeandmax-chunk-size(100 and 2,048 tokens by default, never above 8,192). - There is no overlap or chunk-count setting.
- A job may set
- Principle: no false controls. A setting that is accepted but ignored is worse than no setting.
- Evidence: 27 of 27 chunking mutants caught.
- Context: migrated content's metadata is valuable to agents, and its object keys can name confidential things.
-
Decision:
- Each sidecar is decided by the same
DocumentAccessGateas search, using its document id and itssecurity. Answers carry no permissions. - The paging cursor is sealed with AES-GCM and bound to the caller, the store and the prefix. Its TTL is 1 hour.
- Stores are opened read-only, and only registered Ozone or S3 output connectors listed in
opencrawling.sidecars.connectorscan be read.
- Each sidecar is decided by the same
-
Principle: defence in depth.
- The cursor holds the last key looked at, which may belong to a hidden sidecar, so plain encoding would leak it.
- Authenticated encryption (NIST SP 800-38D) also stops a cursor from being forged or reused by someone else.
- Evidence: 44 of 44 sidecar mutants caught.
- Context: an audit trail that accepts asserted identities can be poisoned.
-
Decision:
- Every execution records how it started:
trigger_sourceisapi,dynamic-api,mcp,bpm,ois,grpc-pushorstartup. - It also records the correlation id.
- Identity fields (
triggered_by,on_behalf_of,caller_groups,caller_tenant) are filled only for verified callers; anonymous runs carry none. - The identity is read on the request thread, because the run's own thread has no security context.
- Every execution records how it started:
- Principle: a trustworthy audit trail.
- Evidence: 6 of 6 provenance mutants caught. Delegated runs record both the agent and the user.
-
Decision:
- Identity mode
none, themetadataauthorization mode, and every agent capability off: existing deployments behave exactly as before. - New ledger columns are optional, and old readers keep working.
- Identity mode
- Principle: safe adoption. Teams enable one capability at a time.
| Threat | Control | Where |
|---|---|---|
| A prompt-injected agent claims another user's identity | Identity from the token; arguments ignored |
CallerResolver (ADR-01) |
| A token issued for another application | Issuer validation, plus audience validation once audiences is set |
Spring Security resource server |
| An agent aims the crawler at an internal host | Registered, allow-listed connectors only |
DynamicJobService (ADR-04) |
| An agent maps folders it cannot read | Hidden folders never opened; source-ACL decisions |
SourceAclVisibility (ADR-05) |
| Probing for hidden documents through errors or counts | Uniform answers; no hidden counts; no upstream errors | ADR-10 |
| An agent hides a live document from everyone | Removal only when the source reports it gone | ADR-14 |
| Flooding with jobs | Per-user, per-caller and global caps (429) |
JobRunRegistry, ADR-11 |
| Duplicate crawls from retries | Idempotency keys per user | ADR-13 |
| An agent approves its own job | Approval is REST-only and four-eyes | ADR-12 |
| A replayed or forged BPM event | HMAC with a timestamp window; dedup per sender | ADR-15 |
| Ingestion that skips governance |
409 for every job-API run that would use an output connector; dynamic jobs and the decoupled crawler never use one |
ADR-08, ADR-16 |
| A governance hook widens permissions | Protected keys restored after every hook | ADR-16 |
| A paging cursor reveals hidden keys | AES-GCM sealed and caller-bound cursor | ADR-18 |
| A poisoned audit trail | Only verified identities recorded | ADR-19 |
| A document with no ACL becomes public |
SecurityConfig.none(): nobody can read it |
ADR-09 |
opencrawling:
governance:
enabled: true
hooks: pii-masking,http # in order; a bean's name() also works
on-rejected-update: purge # or keep
drop-metadata-after-gate: zammad.participants # optional: visible to hooks, stripped before chunking/storage
pii:
types: EMAIL,PAYMENT_CARD,IBAN
http:
url: https://governance.example.com/evaluate
secret: ${GOVERNANCE_SECRET} # at least 32 characters-
Transient governance metadata (
drop-metadata-after-gate): Keys listed here (for examplezammad.participantsprovided so an anonymization hook can redact names) remain visible to governance hooks and are removed right after the gate—or before chunking when the gate is off—so they are never stored downstream. Protected keys (acl,security,ois:*, etc.) cannot be dropped.
The pii-masking hook:
- It masks email addresses, payment cards (Luhn-checked) and IBANs (ISO 13616 mod-97 checked) as
[REDACTED:TYPE]. - It never rejects a document.
The http hook contract. The hook sends POST with this JSON body:
{"documentId": "…", "uri": "…", "text": "…",
"metadata": {"key": ["value"]},
"security": {"inheritanceEnabled": false, "permissions": [{"identity": "…", "identityType": "…", "displayName": "…", "access": "read"}]}}Headers:
-
X-OpenCrawling-Timestamp: epoch seconds; -
X-OpenCrawling-Signature:sha256=followed by the lowercase hex of HMAC-SHA256(secret,timestamp + "." + body).
Reply: 200 with {"decision": "APPROVE" | "SANITIZE" | "REJECT", "text": "…", "metadata": {…}, "reason": "…", "audit": {…}}. A SANITIZE reply must include text.
Errors:
-
429,5xxand network errors are retried: 3 attempts, with backoff starting at 500 ms and capped at 5 s. - Any other status, an invalid reply, or a reply over 32 MiB makes the document unavailable, so nothing is embedded.
Full settings: Configuration Guide, Pre-Embedding Governance.
opencrawling:
events:
bpm:
enabled: true
sources:
- name: flowable-prod
hmac-secret: ${BPM_FLOWABLE_SECRET} # at least 32 characters
- name: aps-prod
token: ${BPM_APS_TOKEN} # Basic auth: aps-prod:<token>
routes:
- name: contract-signed
engine: flowable # blank = any engine
process: contractApproval
transition: signed
connector: Alfresco_Main
action: UPSERT
- process: recordsDisposal
transition: destroyed
connector: Alfresco_Main
action: DELETEThe first matching route wins. An event that no route matches gets 200 with status: IGNORED. Signing an event from a shell, for example to test a route:
ts=$(date +%s)
body='{"eventId":"evt-7f3a","engine":"flowable","processDefinitionKey":"contractApproval","transition":"signed","documentId":"<repository id>"}'
sig=$(printf '%s.%s' "$ts" "$body" | openssl dgst -sha256 -hmac "$BPM_FLOWABLE_SECRET" | sed 's/^.*= //')
curl -sS https://opencrawling.example.com/api/events/bpm/document-transition \
-H 'Content-Type: application/json' \
-H 'X-OpenCrawling-Source: flowable-prod' \
-H "X-OpenCrawling-Timestamp: $ts" \
-H "X-OpenCrawling-Signature: sha256=$sig" \
--data-binary "$body"The answer: {status, eventId, route, connector, document, jobId, executionId, note, duplicate}.
A repeated eventId: it returns the first answer, with duplicate: true.
Full settings: Configuration Guide, BPM Document Events and Document Lifecycle Actions & Tombstones.
Content migrated to Apache Ozone or AWS S3 has an OIS sidecar (<key>.ois.json, written with OisSidecar conforming to schemas/document.schema.json) next to each binary. SidecarService reads both current OIS-conformant sidecars (content with sha256 and sizeBytes) and earlier contentRef (checksumSha256, contentLength) sidecars, flattening typed/nested metadata blocks (such as ois:provenance and ois:governance) into <key>.<field> string lists in SidecarView.
What the reader returns: readOisSidecars (REST GET /api/sidecars/{connector}) returns the sidecars the caller may read, in key order. Each one gives:
- the id, URI and source (including
source.instancestripped of credentials, query, and fragment); - the file name, MIME type, size and SHA-256 checksum;
- the metadata (including pipeline-stamped
ois:provenance, andois:governancewhen governed); - page totals by MIME type and by source type.
What it never does: it never downloads binaries and never returns permissions.
What it skips: keys under .tombstones/ and .opencrawling/, anything that is not an UPSERT, invalid sidecars, and oversized sidecars.
See Model Context Protocol, OIS sidecars.
Every execution, started by an agent or not, records its provenance. It goes to the run summary (summary_<executionId>.json, as a nested provenance object) and to every row of the Parquet/Avro ledger (plus document-level ois:source.*, ois:provenance.*, and ois:governance.* metadata stamped by JobOrchestrator and IngestionConsumer via OisMetadata and protected from source spoofing or hook modification by GovernanceGate, recording onBehalfOf / actorAgent only when callerVerified is true):
| Ledger column | Meaning |
|---|---|
trigger_source |
api, dynamic-api, mcp, bpm, ois, grpc-push or startup
|
triggered_by |
the acting agent (act.sub) for delegated tokens, otherwise the verified user; bpm:<source> for a BPM source authenticated by HMAC or static token |
on_behalf_of |
the user, when an agent acted for them |
caller_groups, caller_tenant
|
the verified caller's groups and tenant |
caller_verified |
whether the identity came from a verified token or an authenticated BPM source |
correlation_id |
from X-Correlation-Id, the MCP correlationId, or the BPM eventId; dropped if it fails ^[A-Za-z0-9._:/@-]{1,128}$
|
See Lakehouse Audit Reporting & Ledger.
| Profile | Enable | For |
|---|---|---|
| Secure search | security.identity.mode: jwt |
Agents that only read the index |
| Discovery | + discovery.enabled |
Agents that plan before asking for content |
| Agentic ingestion |
+ jobs.dynamic.enabled, connectors, admin-groups
|
Agents that start crawls |
| Governed |
+ governance.enabled, hooks
|
Regulated content |
| Event-driven |
+ events.bpm.enabled, sources, routes
|
Process-triggered freshness |
opencrawling:
security:
identity:
mode: jwt
protect-api: true # REST needs a token too; set admin-ui.client-id when using the admin UI
admin-ui:
client-id: opencrawling-admin-ui # optional: OIDC PKCE sign-in for oc-admin-ui (see Security-and-Identity-Management)
authorization:
mode: metadata # or shadow, intersection, openfga
discovery:
enabled: true
jobs:
max-concurrent-runs: 20 # 0 = no cap
max-concurrent-runs-per-caller: 4 # per acting agent
dynamic:
enabled: true
connectors: Alfresco_Main # registered connectors agents may crawl
admin-groups: oc-admins
require-approval: true
governance:
enabled: true
hooks: pii-masking
sidecars:
enabled: true
connectors: Lake_S3
spring:
security:
oauth2:
resourceserver:
jwt:
issuer-uri: https://idp.example.com/realms/corp
audiences: opencrawlingThis is our recommendation, not a requirement:
-
Turn on
jwtmode first. Check that search results are unchanged for real users. If your provider sends full group paths (/finance), turn that off or write grants with the full path. -
Run
shadowmode beforeopenfga. The metadata rules keep deciding while disagreements are counted (oc.authz.shadow.mismatch). -
Enable discovery, then dynamic jobs with
require-approval: trueand a short connector allow-list. Relax approval once you trust your scopes and limits. - Add governance before you widen the allow-list to sensitive repositories.
-
Use
user:/group:prefixes in grants where a user and a group can share a name (see Known limitations).
-
Discovery and search agree (
DiscoveryVersusSearchTest).- A deterministic 400-document corpus covers case variants,
user:andgroup:prefixes, bare identities, denies, public grants, inheritance flags, empty ACLs and missing security objects. - It goes through discovery as nodes, and through the real ingestion consumer, the writer's permission fields and a JSON round trip for search.
- For 8 callers, both paths return the same documents.
- The store pre-filter never hides a document the gate allows.
- A deterministic 400-document corpus covers case variants,
-
Agents stay apart (
AgentSwarmHttpTest).- The test runs real MCP over Streamable HTTP with RS256-signed delegated tokens.
- 4 sub-agents work for 2 users, 12 rounds each, in parallel, and every call forges the other user's identity.
- Each tool asks for its own scope.
- A token signed by another key opens no session.
- Each feature carries unit and context tests, plus the mutation results listed in How we built it.
The stack:
- two machines: one ran the runtime and its services, the other Kafka, pgvector, Ollama, Apache Ozone and OpenFGA;
- a real Alfresco Content Services 26.1 repository;
-
jwtidentity andopenfgaauthorization; - governance through the HTTP hook.
The run:
- An agent with a delegated token discovered a test folder and crawled it for
alice. - Governance approved four documents, masked an email address and an IBAN in a fifth, and rejected a draft.
- Each user then searched.
- In a separate check, a membership written directly into OpenFGA gave
bobthe finance documents at once, and removing it took them away, with no re-crawl.
What it found:
-
Alfresco
DENIEDentries were dropped. The Alfresco crawler ignoredDENIED, socarolcould find a document Alfresco denies her. Fixed by #265. -
Search without
dimensionsfound nothing, because it read the wrong pgvector table. Fixed by #266.
Both fixes are merged. The run was repeated with them applied: carol lost the document in search and in getDocumentContent, and OpenFGA recorded the deny.
| The issue proposed | What was delivered | Why it differs |
|---|---|---|
explore_repository_tree, query_repository_dryrun with userPrincipal arguments |
exploreRepositoryTree, queryRepositoryDryRun, listRepositoryConnectors
|
camelCase like the existing tools; identity from the token (ADR-01) |
| Dry runs for CMIS and OnBase | CMIS and Alfresco (Hyland CIC later added discovery too) | There is no OnBase connector yet (#129) |
read_ois_sidecars with a bucket argument |
readOisSidecars, listSidecarStores on registered stores |
Agents never name storage locations (ADR-04) |
POST /api/jobs/dynamic with a full connector configuration and endpoint |
A registered connector plus a start folder, idempotency key and chunking | SSRF and credential exfiltration (ADR-04) |
trigger_dynamic_job MCP tool |
triggerDynamicJob plus status, list and stop tools |
Agents must follow and stop what they start |
POST /api/jobs/single-document |
POST /api/jobs/dynamic/documents, ingestDocument, removeDeletedDocument
|
Removal guarded by the source (ADR-14) |
pipelineMode, output collection, ephemeral
|
Always RAG into the deployment's pipeline; status kept 24 h | One governed path (ADR-08) |
chunkSize + chunkOverlap
|
chunkSize and three real splitter settings |
The splitter has no overlap (ADR-17) |
PreEmbeddingGovernanceHook.evaluate(document, security) |
evaluate(GovernanceSubject) with read-only security |
Hooks see extracted text; permissions immutable (ADR-16) |
| AxMirror interceptor | A generic signed HTTP hook | No public AxMirror API contract; any governance service can implement the hook |
| BPM webhook for Flowable and APS | Generic event, admin routes, three authentication methods | ADR-15 |
| "Sub-second" freshness | Seconds | We do not promise what a crawl, extraction and embedding cannot guarantee |
| Swarm and 4-phase integration tests |
AgentSwarmHttpTest, DiscoveryVersusSearchTest, and a live two-machine run |
|
| Keycloak token exchange | Any OpenID Connect provider; claims configurable | Not tied to one vendor |
We list these openly, with their status and their impact.
| Limitation | Impact | Status |
|---|---|---|
The Alfresco crawler ignores DENIED entries, lets an EVERYONE grant drop every other rule (denies included), and reads missing permissions as public |
A user denied in Alfresco but allowed through a group can find the document | Fixed in #265 (re-crawl Alfresco sources to correct existing entries) |
Search, document content and source listing read the plain vector_store table unless dimensions is passed |
Agents calling without dimensions find nothing in pgvector |
Fixed in #266 |
identityType is not used: a bare identity matches a user or a group of the same name, in the metadata rules and in OpenFGA |
A user grant alice also matches a role named alice
|
Fixed in PR #286 (EffectiveAcl resolves identityType to user: / group: prefixes; re-crawl existing sources) |
| The OpenFGA sync keeps tuples for documents governance rejected | No exposure (nothing is embedded), but stale tuples | Fixed in #268, PR #282 |
| A governance hook that is unavailable is not retried automatically | The document is not embedded until the next crawl | By design (fail closed); a retry channel is future work |
| Dynamic job status and pending approvals live in memory | Lost on restart; the ledger keeps the durable record | Known |
| Search decisions use the token's principal and groups, not its tenant | Documents have no tenant dimension; the tenant scopes job ownership and quotas | Known |
| The admin UI cannot send bearer tokens | Keep protect-api off where the UI is used |
Fixed in PR #304 (#187): configure admin-ui.client-id for OIDC PKCE sign-in (see Security and Identity Management) |
| RFC 9728 protected-resource metadata is not served | MCP clients must be configured with the identity provider | Future work |
The observability MCP tools are always on. queryThroughputMetrics reports made-up headline figures, and fetchJobTraces invents traces for jobs it has no spans for |
Agents should not treat those figures as facts | Pre-existing |
| Turning gRPC push off stops startup; the Tika separate-process mode fails outside the Docker image | Found during the live run; unrelated to agents | Fixed in #269, PR #285 and #270, PR #284 |
| The admin UI's global chunk size and overlap settings are saved but never used | Only per-job chunking (ADR-17) changes how documents are split |
Fixed in #271, PR #283 |
| Symptom | Likely cause |
|---|---|
401 on /mcp or a tool |
No bearer token, an expired token, or a token from an unknown key in jwt mode |
403 The token does not carry the scope… |
Missing oc:jobs:write, oc:jobs:run, oc:sidecars:read or oc:events:write
|
404 for a connector that exists |
It is not listed in jobs.dynamic.connectors or sidecars.connectors (answers like a missing one) |
400 for a folder the agent expects |
Outside the connector's root, excluded, unreadable by the user, or missing (same answer) |
409 when starting an admin job |
Governance is on. The job API always uses an output connector, so every non-migration run is refused. Crawl with the decoupled crawler containers or dynamic jobs, or turn governance off. |
409 on removal |
The document still exists in the repository |
429 |
max-running-per-user, max-pending-per-user or the runtime run caps |
| Search returns nothing in pgvector | Check that the embedding service is running and that dimensions, if passed, matches the embedding model |
| A user sees less in discovery than in search | Expected when their access comes only from an OpenFGA membership (ADR-05) |
| Keycloak groups do not match grants | The "Full group path" mapper sends /finance; turn it off or grant /finance
|
BPM 401 …outside the accepted window
|
The sender's clock is more than max-clock-skew (5 min) off |
- Enterprise Multi-Agent Integration: the integration guide for agent builders
- Model Context Protocol: every tool, parameter and REST endpoint
- Configuration Guide: every setting
- Document Authorization & OpenFGA: the authorization modes and the permission sync
- Zero-Trust Security & ACL Management: the ACL model
- Document Lifecycle Actions & Tombstones: removals and BPM transitions
- Lakehouse Audit Reporting & Ledger: provenance columns
- Issue #203; foundations #187, #224, #261
Standards and sources:
- RFC 8693 (OAuth 2.0 Token Exchange)
- RFC 9068 (JWT access tokens)
- RFC 9110 §15.5.4 (hiding a resource with 404)
- OWASP A10:2021 SSRF
- IETF Idempotency-Key header draft
- NIST SP 800-38D (GCM)
- Saltzer and Schroeder, The Protection of Information in Computer Systems (1975)
- Model Context Protocol authorization
- Home
- User & Installation Guide
- Quick Start
- VM & Bare-Metal Distribution Installation
- Local Development & Fast Docker Builds
- User Guide
- Architecture
- Zero-Trust Security & ACL Management
- Security and Identity Management
- Document Authorization & OpenFGA
- Enterprise Multi-Agent Integration
- Zero-Trust Data Provider Agent & Active Ingestion
- Document Lifecycle Actions & Tombstones
- Apache Tika 4 Text Extraction
- Configuration Guide
- Native Job Scheduler
- Platform API (
/api/v1) - Platform API: Traceability
- Connectors Reference
- Alfresco Content Services (ACS) Connector
- Alfresco Process Services (APS) Connector
- Flowable Repository Connector
- Camunda Repository Connector
- OASIS CMIS Repository Connector
- Doxis Repository Connector
- M-Files Repository Connector
- SharePoint Online Repository Connector
- Google Drive Repository Connector
- Atlassian Confluence Repository Connector
- IBM FileNet P8 Repository Connector
- OpenText Documentum Repository Connector
- Zammad Repository Connector
- Hyland OpenArch Integration (Repository, Output & MCP)
- Hyland Content Innovation Cloud (CIC) Connector
- Relational Database (JDBC) Connector
- Web Repository Connector
- Apache StormCrawler Repository Connector
- gRPC Push Repository Connector
- Elasticsearch Output Connector
- Apache Solr Output Connector
- Luxir Output Connector
- Apache SeaTunnel Output Connector
- Apache Ozone Output Connector (Migration Mode)
- AWS S3 Output Connector (Migration Mode)
- Alfresco Content Services Output Connector (Migration Mode)
- Apache Iceberg Output Connector
- Doxis Output Connector
- Connector Execution Modes
- Model Context Protocol
- Apache Ozone Claim Check Store
- Lakehouse Audit Reporting & Ledger
- AI-Powered Observability (AIOps)
- Auto-Narrativization Copilot
- Maven Connector Archetypes
- Java Client SDK
- Python Client SDK
- OpenCrawling CLI
- OpenAPI Specification & Swagger UI
- gRPC Internal Transport
- Migration Guide (ManifoldCF)
- Integration Test — Decoupled Pipeline
- M-Files Integration Test
- SharePoint Online Integration Test
- Google Drive Integration Test
- Confluence Integration Test
- IBM FileNet Integration Test
- OpenText Documentum Integration Test
- Zammad Integration Test
- Hyland OpenArch Integration Test
- Hyland CIC Integration Test
- Web Crawler Test
- Elasticsearch Integration Test
- Local Docker Build Test
- Root CLI Runner Scripts Test
- Secure MCP Server Integration Test
- Narrativization Integration Test
- Reporting Integration Test
- AWS S3 Migration Integration Test
- Alfresco Migration Integration Test
- OpenFGA Authorization Integration Test
- CLI Decoupled Integration Test
- Java Client SDK Decoupled Test
- Apache Iceberg Decoupled Test
- Doxis Output Integration Test
- Platform API v1 Test Suite