Zero Trust Security and ACL Management

Zero-Trust Security & Enterprise ACL Management

Module Reference: oc-core, oc-alfresco-repository-connector, oc-cmis-repository-connector, oc-doxis-repository-connector, oc-mfiles-repository-connector, oc-sharepoint-repository-connector, oc-openarch-repository-connector, oc-openarch-output-connector, oc-flowable-repository-connector, oc-camunda-repository-connector, oc-aps-repository-connector, oc-mcp-server

📌 1. Executive Summary

In enterprise Retrieval-Augmented Generation (RAG) and semantic search architectures, indexing content without strictly preserving Access Control Lists (ACLs) introduces severe security vulnerabilities: context poisoning, confidential data leakage, and prompt authorization bypass. Traditional scrapers and vector indexing tools strip document access lists, treating all ingested chunks as globally readable by any querying LLM or user.

OpenCrawling introduces a unified, end-to-end Zero-Trust Security & ACL Management Architecture governed by the Open Ingestion Standard (OIS). This architecture bridges legacy Enterprise Content Management (ECM) systems (Alfresco Content Services, OASIS CMIS) and Business Process Management (BPMN) engines (Flowable, Camunda Platform 7, Alfresco Process Services) with modern vector databases and the Secure Model Context Protocol (MCP) Server.

flowchart TD
    subgraph Sources["Enterprise Repositories (ECM & BPMN)"]
        ACS["Alfresco Content Services<br/>(Core & Search REST API)"]
        CMIS["OASIS CMIS 1.0 / 1.1<br/>(Alfresco, Documentum, FileNet)"]
        Flowable["Flowable BPMN Engine<br/>(Historic & Runtime REST)"]
        Camunda["Camunda Platform 7<br/>(Engine REST API)"]
        APS["Alfresco Process Services<br/>(Enterprise REST API)"]
    end

    subgraph SecurityMappers["OpenCrawling Zero-Trust Security Mappers (Java 25)"]
        M_ACS["AlfrescoSecurityMapper<br/>(ACEs, GROUP_EVERYONE, Aspects)"]
        M_CMIS["CmisSecurityMapper<br/>(ACEs, anyone, Direct/Inherited)"]
        M_FLW["FlowableSecurityMapper<br/>(Identity Links, Candidate Groups)"]
        M_CAM["CamundaSecurityMapper<br/>(Identity Link Log, Scope IDs)"]
        M_APS["ApsSecurityMapper<br/>(Task Assignees, Candidate Groups)"]
    end

    ACS --> M_ACS
    CMIS --> M_CMIS
    Flowable --> M_FLW
    Camunda --> M_CAM
    APS --> M_APS

    subgraph KafkaStream["Apache Kafka Ingestion Fabric"]
        Envelope["OIS RepositoryDocument<br/>{ metadata + security.permissions + collocated_keys }"]
        M_ACS & M_CMIS & M_FLW & M_CAM & M_APS --> Envelope
        Envelope --> Kafka["Kafka Topic: opencrawling-raw-documents"]
        Kafka --> Ingestion["oc-ingestion-consumer (Chunking & Claim Check)"]
        Ingestion --> Embedding["oc-embedding-consumer (Vector Embeddings)"]
    end

    subgraph DualPath["Dual Zero-Trust Storage Strategies"]
        direction TB
        subgraph Collocated["1. Collocated Database Pattern (Zero-Lag Pushdown)"]
            PGV[("PostgreSQL + pgvector<br/>(Shared DB instance with BPMN/ECM)")]
            SQL["Query Pushdown via SQL EXISTS<br/>JOIN with ACT_RU_IDENTITYLINK / ACT_HI_IDENTITYLINK"]
            PGV --- SQL
        end

        subgraph Distributed["2. Distributed Pre-Filtering Pattern (OIS Security Tokens)"]
            ExtStores[("Distributed Vector Databases<br/>Qdrant / Vespa / Milvus / OpenSearch / Solr / Luxir")]
            TokenFilter["Payload Pre-filtering<br/>security_allowed_read IN (user, groups)"]
            ExtStores --- TokenFilter
        end
    end

    Embedding --> PGV
    Embedding --> ExtStores

    subgraph QueryEnforcement["Secure Query & Inference Layer"]
        MCP["OpenCrawling Secure MCP Server (:8080/mcp)<br/>Identity Propagation: User ID + Group Authorities"]
        Agents["AI Assistants / Spring AI RAG / Copilots"]
        
        Agents -->|Authenticated Query| MCP
        MCP -->|Pushdown SQL EXISTS| PGV
        MCP -->|Pre-filtered ANN Query| ExtStores
    end

🏛️ 2. Core Architectural Patterns

OpenCrawling supports two complementary zero-trust strategies to balance real-time permission agility against distributed scalability.

Pattern A: Collocated Database Pattern (Zero-Lag Query Pushdown)

In enterprise data architectures, repositories frequently alter permissions, reassign tasks, update role memberships, or modify directory ACLs. In traditional vector architectures, updating document permissions requires re-indexing millions of vector chunks in an external database, creating severe distributed synchronization lag, race conditions, and write amplification.

To solve this, OpenCrawling generalizes the Collocated Database Pattern across all repository connectors by stamping unambiguous collocated entity reference keys into each document's metadata:

Repository Family	Connectors	Stamped Entity Reference Keys	Collocated Relational Target
BPMN Workflows	Flowable, Camunda, APS	`_proc_inst_id`, `_scope_id`, `_scope_type`	`ACT_RU_IDENTITYLINK`, `ACT_HI_IDENTITYLINK`
Enterprise Content (ECM)	Alfresco ACS, OASIS CMIS	`alfresco_node_id`, `cmis_object_id`, `cmis_repository_id`	`alf_node`, `alf_access_control_entry`, `alf_authority`
Local / Mounted Filesystem	Filesystem	`file_path`, `file_owner`, `file_group`, `file_mode_octal`	POSIX bitmask & `os_user_groups` directory table
Data Lakehouse	Apache Iceberg	`iceberg_table_name`, `iceberg_record_id`, `iceberg_snapshot_id`	Catalog governance & tenant entitlement table
Relational Databases	JDBC	`table_name`, `primary_key`, `tenant_id`	Database row-level security (RLS) & authorization table
Distributed Web Crawling	StormCrawler	`web_domain`, `web_scheme`, `stormcrawler_topology`	Domain/intranet realm white-list table

When pgvector resides in the same PostgreSQL database as the repository (or has access via PostgreSQL Foreign Data Wrappers postgres_fdw or shared schemas), authorization is evaluated dynamically at query execution time via native SQL pushdowns:

1. Workflow Engines Pushdown (Flowable, Camunda, APS):

SELECT id, content, embedding <=> :queryEmbedding AS distance
FROM vector_store_1024
WHERE (
    -- Runtime identity check for active workflows
    EXISTS (
        SELECT 1 FROM ACT_RU_IDENTITYLINK ru
        WHERE ru.PROC_INST_ID_ = (metadata->>'flowable_proc_inst_id')
          AND (ru.USER_ID_ = :currentUserId OR ru.GROUP_ID_ IN (:currentUserGroups))
    )
    -- Historical identity check for completed workflows
    OR EXISTS (
        SELECT 1 FROM ACT_HI_IDENTITYLINK hi
        WHERE hi.PROC_INST_ID_ = (metadata->>'flowable_proc_inst_id')
          AND (hi.USER_ID_ = :currentUserId OR hi.GROUP_ID_ IN (:currentUserGroups))
    )
)
ORDER BY distance ASC LIMIT :topK;

2. Alfresco Content Services Pushdown:

SELECT v.id, v.content, v.embedding <=> :queryEmbedding AS distance
FROM vector_store_1024 v
JOIN alf_node n ON n.uuid = (v.metadata->>'alfresco_node_id')
JOIN alf_access_control_list acl ON acl.id = n.acl_id
JOIN alf_access_control_entry ace ON ace.acl_id = acl.id
JOIN alf_authority auth ON auth.id = ace.authority_id
WHERE (
    auth.recipient = :currentUserId 
    OR auth.recipient IN (:currentUserGroups) 
    OR auth.recipient = 'GROUP_EVERYONE'
)
AND ace.allowed = true
ORDER BY distance ASC LIMIT :topK;

3. POSIX Filesystem Pushdown:

SELECT v.id, v.content, v.embedding <=> :queryEmbedding AS distance
FROM vector_store_1024 v
WHERE (
    -- Owner read bit (0400 = 256)
    (v.metadata->>'file_owner' = :currentUserId 
     AND (v.metadata->>'file_mode_octal')::int & 256 != 0)
    -- Group read bit (0040 = 32)
    OR (v.metadata->>'file_group' IN (:currentUserGroups) 
        AND (v.metadata->>'file_mode_octal')::int & 32 != 0)
    -- World / Other read bit (0004 = 4)
    OR ((v.metadata->>'file_mode_octal')::int & 4 != 0)
)
ORDER BY distance ASC LIMIT :topK;

Key Benefits of Pattern A:

Zero Synchronization Lag: Changes in workflow assignments, folder ACLs, or POSIX permissions take effect in vector search immediately without updating vector records.
Zero Row Explosion: No duplicated permission rows or redundant chunk updates when permissions change.
Atomic Revocation: If an account is removed from an authority group in Alfresco, Camunda, or the OS, their RAG access is revoked on the very next query.

Pattern B: Distributed Database Pattern (OIS Metadata Pre-Filtering)

When deploying to standalone, cloud-native vector search engines (such as Qdrant, Vespa, Milvus, OpenSearch 2.x/3.x, Apache Solr, or Luxir), OpenCrawling maps the document's SecurityConfig directly into structured vector payload fields:

{
  "security_allowed_read": ["jdoe", "GROUP_FINANCE", "GROUP_LEGAL", "GROUP_EVERYONE"],
  "security_denied_read": ["contractor_temp"],
  "security_inheritance": true
}

At query time, the caller's identity token is evaluated using boolean payload filters:

Condition: (security_allowed_read CONTAINS :userId OR security_allowed_read INTERSECTS :userGroups OR security_allowed_read CONTAINS 'GROUP_EVERYONE') AND NOT (security_denied_read CONTAINS :userId)

🔌 3. Connector Security Mapping Implementations

Each repository connector features a dedicated security mapper implementing OIS translation rules.

A. Alfresco Content Services (ACS) Connector

Mapper Class: AlfrescoSecurityMapper
API Ingestion: Queries the Core REST API requesting ?include=properties,aspectNames,permissions.
Access Control Entry (ACE) Mapping:
- Extracts both locally assigned permissions (entry.permissions.locallySet) and inherited permissions (entry.permissions.inherited).
- Distinguishes user accounts from group authorities:
  - Prefixes GROUP_ and ROLE_ are categorized as PermissionRule.identityType = "group".
  - All other account names are mapped to PermissionRule.identityType = "user".
- Public Access Normalization: Automatically identifies GROUP_EVERYONE, guest, and ROLE_ANONYMOUS as public access grants.
- Inheritance Flag: Stored in SecurityConfig.inheritanceEnabled based on entry.permissions.isInheritanceEnabled.
Collocated Metadata Attributes:
- alfresco_node_id: Target node UUID.
- alfresco_node_type: Content model type (cm:content, cm:folder, custom types).
- alfresco_aspects: Array of applied aspects (e.g. cm:titled, cm:versionable).
- alfresco_identity_users: Comma-separated list of authorized reader users.
- alfresco_identity_groups: Comma-separated list of authorized reader groups.
- alfresco_is_inherited: Boolean flag for permission inheritance.

B. OASIS CMIS Repository Connector

Mapper Class: CmisSecurityMapper
Binding Ingestion: Traverses CMIS Browser or AtomPub bindings, parsing the native cmis:acl element.
Permission Mapping:
- Resolves CMIS basic permissions (cmis:read, cmis:write, cmis:all) and vendor-specific permission extensions (e.g. Alfresco Consumer, Collaborator; Documentum BROWSE, READ).
- Maps cmis:isExact and direct vs. inherited ACE flags.
- Public Access Normalization: Recognizes anyone and anonymous principals.
Collocated Metadata Attributes:
- cmis_object_id: Repository object ID.
- cmis_object_type: Object type (cmis:document, cmis:folder).
- cmis_identity_users: Allowed reader user list.
- cmis_identity_groups: Allowed reader group list.

C. Flowable BPMN Repository Connector

Mapper Class: FlowableSecurityMapper
REST Ingestion: Queries /history/historic-process-instances/{id}/identitylinks.
Identity Link Resolution:
- Maps process instance starters, owners, assignees, and candidate groups into PermissionRule objects.
- Translates candidate groups to identityType = "group" and candidate users to identityType = "user".
Collocated Metadata Attributes:
- flowable_proc_inst_id: Flowable historic process instance ID.
- flowable_scope_id: Process instance scope ID.
- flowable_scope_type: Set to "processInstance".
- flowable_identity_users: Authorized participant user names.
- flowable_identity_groups: Authorized candidate group keys.

D. Camunda Platform 7 Repository Connector

Mapper Class: CamundaSecurityMapper
Engine Ingestion: Queries Camunda Engine REST /history/identity-link-log and process instance starter metadata.
Identity Link Resolution:
- Filters historic assignees, owners, and candidate group associations.
- Translates candidate groups into OIS group permission rules.
Collocated Metadata Attributes:
- camunda_proc_inst_id: Camunda historic process instance ID.
- camunda_scope_id: Process instance scope ID.
- camunda_scope_type: Set to "processInstance".
- camunda_identity_users: Authorized users.
- camunda_identity_groups: Authorized groups.

E. Alfresco Process Services (APS) Connector

Mapper Class: ApsSecurityMapper
Enterprise API Ingestion: Scans /historic-process-instances/query and related task user assignments (/tasks/query).
Workflow Attachment Security: Inherits parent workflow instance candidate groups and assignees for embedded attachment documents.
Collocated Metadata Attributes:
- aps_proc_inst_id: APS process instance ID.
- aps_scope_id: Scope ID.
- aps_scope_type: Set to "processInstance".

F. Local & Mounted Filesystem Connector

Connector Class: FileSystemRepositoryConnector
POSIX Security Ingestion: Evaluates POSIX file attributes (PosixFileAttributeView), extracting file owner, group, permissions string, and numeric octal modes.
Rule Mapping:
- Translates PosixFilePermission.OWNER_READ to PermissionRule(identityType = "user", access = "read").
- Translates PosixFilePermission.GROUP_READ to PermissionRule(identityType = "group", access = "read").
- Translates PosixFilePermission.OTHERS_READ to PermissionRule(identityType = "public", access = "read").
Collocated Metadata Attributes:
- file_path: Canonical absolute file path.
- file_name: File name.
- file_size: Size in bytes.
- file_owner: User owner account name.
- file_group: Primary POSIX group name.
- file_permissions: Symbolic permission string (e.g. rw-r--r--).
- file_mode_octal: Four-digit octal bitmask (e.g. 0644).
- file_identity_users: Owner user for token filtering.
- file_identity_groups: Owner group for token filtering.

G. Apache Iceberg Lakehouse Connector

Connector Class: IcebergRepositoryConnector
Catalog Ingestion: Scans Parquet data files across Iceberg tables.
Collocated Metadata Attributes:
- iceberg_table_name: Full catalog table identifier (e.g. warehouse.finance.invoices).
- iceberg_record_id: Primary record identifier or deterministic row hash.
- iceberg_table_location: Storage location URI (e.g. s3://warehouse/invoices).
- iceberg_snapshot_id: Active snapshot ID when the row was crawled.

H. Apache StormCrawler Repository Connector

Connector Class: StormCrawlerRepositoryConnector
Web Crawl Ingestion: Streams fetched web pages from Apache Storm Nimbus topologies.
Collocated Metadata Attributes:
- web_domain: Hostname / domain of the crawled URL (e.g. docs.example.com).
- web_scheme: URL protocol scheme (http / https).
- canonical.url: Canonical page URL.
- stormcrawler_topology: Deployed Storm topology name.

I. Relational Database (JDBC) Repository Connector

Connector Class: JdbcRepositoryConnector
Security Mapper: JdbcSecurityMapper
Column-to-Permission Mapping:
- userColumns: Extracts user identities from configured database columns (e.g. owner_id, created_by, assignee) and creates PermissionRule entries (type: "user", permission: defaultPermission).
- groupColumns: Extracts organizational group or role identities (e.g. department_id, security_group, role) and creates PermissionRule entries (type: "group", permission: defaultPermission).
- tenantColumn: Stamped as tenant_id metadata attribute to guarantee strict multi-tenant context isolation.
- If no security columns are defined or matched, JdbcSecurityMapper marks the record as SecurityConfig.createPublic() with legacy ACL token public.
Collocated Metadata Attributes:
- table_name: Name of the source database table or query identifier.
- primary_key: Source database primary key.
- tenant_id: Multi-tenant organization identifier.

Collocated Query Pushdown:

SELECT id, content, embedding <=> :queryEmbedding AS distance
FROM vector_store_1024
WHERE (
    -- Zero-trust column-based check and tenant match
    (metadata->>'tenant_id' = :currentTenantId)
    AND (
        security_allowed_read ? :currentUserId
        OR security_allowed_read ?| array[:currentUserGroups]
        OR security_allowed_read ? 'public'
    )
)
ORDER BY distance ASC
LIMIT :topK;

J. Doxis Repository Connector

Connector Class: DoxisRepositoryConnector
Security Mapper: DoxisSecurityMapper
Doxis ACE-to-Permission Mapping:
- VIEW_DOCUMENT_CONTENTS (document) and VIEW_FOLDER_CONTENTS (e-file / Akte) with authorizationVariant: GRANT map to access: "read".
- VIEW_* with authorizationVariant: DENY maps to access: "deny" (takes precedence).
- UPDATE_DOCUMENT and UPDATE_FOLDER with GRANT map to access: "write".
- Group everybody maps to public (or <customer>/everybody with tenant isolation).
- Evaluates users, organizational groups, and roles via DoxisSchema#principalById.
Hierarchical E-File Inheritance:
- A document filed inside an electronic dossier (Akte) inherits its parent e-file's ACEs. DoxisSecurityMapper merges e-file permissions with document-level instance ACEs, preserving effective access rights (inheritanceEnabled: true).
Fail-Secure Fallbacks (Deny by Default):
- In Doxis 4 CSB, class-level rights ("All instances") are not readable via REST API. Documents lacking instance ACEs receive configured fallbackPrincipals (e.g. group:Legal) or an empty ACL (deny by default). They are never marked public.
Tenant Isolation:
- With tenantIsolation: true, all user/group identities are qualified by the customer (e.g. ACME/Legal, ACME/elena.weber), completely isolating tenant spaces in shared vector stores and MCP server queries.

Collocated Query Pushdown:

SELECT id, content, embedding <=> :queryEmbedding AS distance
FROM vector_store_1024
WHERE (
    (metadata->>'doxis.customer' = :currentTenantId)
    AND (
        security_allowed_read ? :currentUserId
        OR security_allowed_read ?| array[:currentUserGroups]
        OR security_allowed_read ? 'public'
    )
)
ORDER BY distance ASC
LIMIT :topK;

K. gRPC Push Repository Connector

Connector Class: org.opencrawling.push.grpc.GrpcPushRepositoryConnector
Security Translator & Interceptor: org.opencrawling.push.grpc.OisDocumentTranslator, org.opencrawling.push.grpc.GrpcSecurityInterceptor
Protobuf-to-OIS Permission Mapping:
- SecurityConfig.allow_sids and SecurityConfig.owner map to PermissionRule(identity, "read").
- SecurityConfig.deny_sids maps to PermissionRule(identity, "deny") with explicit deny precedence.
- Legacy flat OisDocumentPayload.acl entries are merged into the effective allowed read SIDs while filtering out any identity present in deny_sids.
- If no SIDs are supplied on an UPSERT payload, defaults safely to "public" only when unconfigured, or enforces strict SID lists when SecurityConfig is present.
Transport-Level Authentication & mTLS:
- GrpcSecurityInterceptor enforces Authorization: Bearer <token> using constant-time byte comparison (MessageDigest.isEqual) on all ingestion RPCs (PushDocument, StreamDocuments, BiDirectionalStream).
- Supports TLS and Mutual TLS (ClientAuth.REQUIRE) via X.509 client certificate verification on the shaded Netty server.
Multi-Tenant Isolation:
- SecurityConfig.tenant_id is automatically stamped into metadata.put("tenant_id", tenantId), isolating tenant vector namespaces during MCP and hybrid vector queries.

L. Hyland OpenArch Repository & Output Connectors

Connector Classes: org.opencrawling.openarch.repository.OpenArchRepositoryConnector, org.opencrawling.openarch.output.OpenArchOutputConnector
Security Mapper: org.opencrawling.openarch.output.security.OpenArchSecurityMapper
Bidirectional OpenArch sys_acl ↔ OIS Mapping:
- Harvesting (fromOpenArchAcl / OpenArchRepositoryConnector):
  - OpenArch Read maps to access: "read".
  - Write and ReadWrite map to access: "write".
  - Delete and DeleteChild map to access: "delete".
  - ManageSecurity, ManageLock, and All map to access: "admin".
  - Any ACE with "granted": false maps to access: "deny", preserving explicit deny rules.
  - user.username (or user.id, or flat username) maps to identityType: "user"; group.groupname (or group.id) maps to identity: "group:<name>" with identityType: "group".
- Ingestion (toOpenArchAcl in OpenArchOutputConnector):
  - When syncAcls: true, OIS PermissionRules are converted into OpenArch sys_acl ACEs stamped with "creator": "opencrawling".
  - read → Read (granted: true), write → Write (granted: true), delete → Delete (granted: true), admin → ManageSecurity (granted: true), and deny → Read (granted: false).
  - Identities with identityType: "group" or prefixed with group: / GROUP_ emit "group": {"groupname": "<name>"} (stripping group:); user identities emit "user": {"username": "<name>"} (stripping user:).
Dedicated Guide: See Hyland OpenArch Integration.

M. M-Files Repository Connector

Connector Class: org.opencrawling.mfiles.MFilesRepositoryConnector
Security Mapper: org.opencrawling.mfiles.MFilesSecurityMapper
Fail-Closed MFWS & VAF Permission Mapping:
- MFWS Permissions (GET /objects/{type}/{id}/{version}/permissions):
  - Evaluates the object's effective AccessControlList (CustomComponent plus active AutomaticComponents from workflow states or named ACLs).
  - Restrictive Component Intersection: M-Files combines active ACL components restrictively, so an identity receives access: "read" (mfiles:user:<id> or mfiles:group:<id>) only when every active component (IsActive != false) grants ReadPermission: 1 (or "Allow", or bit 1 of AccessFlags).
  - Explicit Deny Precedence: Any other numeric ReadPermission code or "Deny" maps to access: "deny" for that identity and overrides allows across all components.
  - Pseudo-Users: Metadata-driven pseudo-user entries (IsPsudoUser: true or HasConcreteUserOrGruopID: false, e.g., "Project manager") grant nothing over MFWS; use vaf crawl mode or fallback-principals.
- Vault Application Framework (vaf Crawl Mode):
  - Consumes server-resolved aclTokens and denyTokens (mfiles:user:<id>, mfiles:group:<id>, mfiles:role:<id>) where metadata-driven pseudo-users and component intersections are evaluated inside the vault.
- Principal Formatting & Multi-Vault Isolation:
  - principal-format: name resolves user and group IDs via built-in M-Files value lists 6 (Users) and 16 (User Groups); qualify-principals: true prefixes every identity with <vaultGuid>/ to isolate multi-vault vector stores.
  - Groups in public-group-ids (e.g., 1 for "All internal users") map to public. Unreadable permissions fall back to fallback-principals or nobody (deny by default).
Dedicated Guide: See M-Files Repository Connector.

N. Microsoft SharePoint Online Repository Connector

Connector Class: org.opencrawling.sharepoint.SharePointRepositoryConnector
Security Mapper: org.opencrawling.sharepoint.SharePointSecurityMapper
Fail-Closed Microsoft Graph Permission Mapping (GET /drives/{driveId}/items/{itemId}/permissions):
- Readable Roles Only: Only roles granting content read access (read, write, owner, sp.full control, sp.contribute, sp.read) are accepted; metadata-only or custom roles (such as sp.restricted view) are ignored.
- Person Grants (grantedToV2.user / grantedToIdentitiesV2[].user + siteUser):
  - Emits both the lower-case email/UPN (extracted from user.email or the SharePoint membership claim i:0#.f|membership|<upn>) and the Entra object ID (user.id), or the object ID alone when user-identity: id.
  - Display names and site-local numeric siteUser.id values are never used as principals.
- Entra ID Security & M365 Groups:
  - Direct group grants (grantedToV2.group) and federated group claims in siteUser.loginName (c:0t.c|tenant|<guid>, c:0o.c|federateddirectoryclaimprovider|<guid>) map to identityType: "group" using the Entra group object ID (<guid>).
- SharePoint Site Groups (grantedToV2.siteGroup):
  - Emitted as site-qualified group principals sharepoint-group:<siteId>:<siteGroup.id> so numeric site-group IDs never collide across sites.
- Broad Access Grants (Fail-Closed by Default):
  - Tenant-wide Everyone (c:0(.s|true), Everyone except external users (spo-grid-all-users), organization sharing links (scope: organization), and anonymous sharing links (scope: anonymous) grant nothing unless explicitly mapped via everyone-identity, everyone-except-external-identity, organization-link-identity, or anonymous-link-identity.
  - Application identities (grantedToV2.application) are ignored, and items with no readable permissions fall back to fallback-principals or nobody (never public by default).
Dedicated Guide: See SharePoint Online Repository Connector.

📦 4. Unified Open Ingestion Standard (OIS) Envelope

Regardless of whether content originates from an ECM repository or a BPMN engine, OpenCrawling encapsulates the document and its access control into the standard OIS envelope circulating through Kafka:

{
  "id": "e4f8b91a-7b3c-4d5e-9f0a-1b2c3d4e5f6a",
  "uri": "alfresco://Company Home/Sites/legal/Contracts/Agreement-2026.pdf",
  "content": "MASTER SERVICES AGREEMENT\nBetween Company A and Company B...",
  "metadata": {
    "name": "Agreement-2026.pdf",
    "mimeType": "application/pdf",
    "sizeInBytes": 1048576,
    "alfresco_node_id": "e4f8b91a-7b3c-4d5e-9f0a-1b2c3d4e5f6a",
    "alfresco_node_type": "cm:content",
    "alfresco_aspects": ["cm:titled", "cm:author"],
    "alfresco_identity_users": "alice,bob",
    "alfresco_identity_groups": "GROUP_LEGAL,GROUP_DIRECTORS",
    "alfresco_is_inherited": true,
    "cm:title": "Master Services Agreement 2026"
  },
  "security": {
    "inheritanceEnabled": true,
    "permissions": [
      {
        "identity": "alice",
        "identityType": "user",
        "access": "read"
      },
      {
        "identity": "bob",
        "identityType": "user",
        "access": "read"
      },
      {
        "identity": "GROUP_LEGAL",
        "identityType": "group",
        "access": "read"
      },
      {
        "identity": "GROUP_DIRECTORS",
        "identityType": "group",
        "access": "read"
      }
    ]
  }
}

🔒 5. Query-Time Enforcement with the Secure MCP Server

The OpenCrawling Secure MCP Server (oc-mcp-server) serves as the authorized context gateway for AI agents, Copilots, and enterprise RAG pipelines. Powered by McpAclEvaluator and the pluggable McpVectorSearchBackend SPI architecture, it enforces Zero-Trust boundaries across all 8 target vector database engines (PostgreSQL/pgvector, Vespa, Qdrant, Milvus, OpenSearch 2.x, OpenSearch 3.x, Apache Solr, Luxir).

How Zero-Trust Identity is Enforced:

Identity Context & Parameter Propagation: Incoming tool requests to secureVectorSearch, getDocumentContent, or listAccessibleSources pass the authenticated user principal identity and active group/role authorities:

{
  "name": "secureVectorSearch",
  "arguments": {
    "query": "financial budget analysis",
    "userPrincipal": "alice@company.com",
    "userRoles": "finance,accounting",
    "maxResults": 5
  }
}

Pluggable Pushdown Query Pre-Filtering (McpVectorSearchBackend): The MCP server translates security predicates into native backend filter expressions at query time:
- In PostgreSQL (pgvector): Injects SQL clauses (acl = 'public' OR acl = :user_principal OR acl = ANY(:user_roles)) and utilizes JdbcTemplate with JSONPath expressions to resolve claim-check URIs and metadata.
- In Vespa: Formulates Vespa YQL queries pushing explicit security_denied_read exclusions directly into BM25 + vector hybrid rank profiles.
- In Distributed Vector Stores (Qdrant, Milvus, OpenSearch 2.x/3.x, Solr, Luxir): Constructs native filter expressions targeting security_allowed_read and tenant boundaries.
Authoritative In-Memory Defense-in-Depth (McpAclEvaluator): Regardless of database-level optimizations, every candidate match is authoritatively re-checked in memory:
- Explicit DENY Precedence: If a chunk has an explicit deny access rule matching the user principal or any assigned roles, it is immediately discarded. Denials strictly supersede any group or public grants.
- Multi-Schema Evaluation: Evaluates structured OIS PermissionRule objects (metadata["security"]), flattened arrays (security_allowed_read, security_denied_read), or legacy flat ACL strings (acl, acl_ss).
Guaranteed Zero Data Leakage: Chunks for which the caller is neither a listed user nor a member of an authorized group are pruned before any text enters the LLM context window.

🎛️ 6. Admin UI Configuration Controls

In the OpenCrawling Admin UI (oc-admin-ui), every enterprise connector includes an explicit toggle for zero-trust security synchronization:

Connector	UI Field Label	JSON Property Key	Default
Alfresco Content Services	`Extract Zero-Trust ACLs`	`includeAcls`	`true`
OASIS CMIS	`Extract Access Control Lists (ACLs)`	`includeAcls`	`true`
Flowable BPMN	`Extract Security & Identity Links (ACLs)`	`includeAcls`	`true`
Camunda Platform 7	`Extract Security & Identity Links (ACLs)`	`includeAcls`	`true`
Alfresco Process Services	`Extract Security & Identity Links (ACLs)`	`includeAcls`	`true`

When includeAcls is toggled false, documents are ingested with empty permission rules for high-throughput public indexing scenarios.

🧪 7. Integration Testing & Verification

Automated test scripts verify end-to-end ACL provenance, collocated metadata persistence, and query-time MCP authorization:

# Verify Secure MCP Server Zero-Trust ACL enforcement & explicit deny precedence
./scripts/test-mcp-decoupled.sh
# Or via master test runner:
./run-integration-tests.sh mcp

# Verify Alfresco zero-trust ACLs and collocated node attributes
./scripts/test-alfresco-decoupled.sh

# Verify OASIS CMIS ACL mapping
./scripts/test-cmis-decoupled.sh

# Verify Flowable collocated identity links and SQL pushdown
./scripts/test-flowable-decoupled.sh

# Verify Camunda collocated identity links and SQL pushdown
./scripts/test-camunda-decoupled.sh

Manual SQL Verification in `pgvector`:

-- Verify collocated entity keys and identity attributes across BPMN, ECM, Filesystem, Lakehouse, and Web
SELECT id,
       metadata->>'alfresco_node_id'         AS alfresco_id,
       metadata->>'cmis_object_id'           AS cmis_id,
       metadata->>'flowable_proc_inst_id'     AS flowable_id,
       metadata->>'camunda_proc_inst_id'      AS camunda_id,
       metadata->>'file_path'                AS file_path,
       metadata->>'file_mode_octal'          AS file_mode,
       metadata->>'iceberg_table_name'       AS iceberg_table,
       metadata->>'web_domain'               AS web_domain
FROM vector_store_1024
LIMIT 10;

🗺️ Documentation

Home
Quick Start
VM & Bare-Metal Distribution Installation
Local Development & Fast Docker Builds
User Guide
Architecture
Zero-Trust Security & ACL Management
Document Lifecycle Actions & Tombstones
Apache Tika 4 Text Extraction
Configuration Guide
Connectors Reference
Alfresco Content Services (ACS) Connector
Alfresco Process Services (APS) Connector
Flowable Repository Connector
Camunda Repository Connector
OASIS CMIS Repository Connector
Doxis Repository Connector
M-Files Repository Connector
SharePoint Online Repository Connector
Hyland OpenArch Integration (Repository, Output & MCP)
Relational Database (JDBC) Connector
Apache StormCrawler Repository Connector
gRPC Push Repository Connector
Apache Solr Output Connector
Luxir Output Connector
Apache SeaTunnel Output Connector
Apache Ozone Output Connector (Migration Mode)
Connector Execution Modes
Model Context Protocol
Apache Ozone Claim Check Store
Lakehouse Audit Reporting & Ledger
AI-Powered Observability (AIOps)
Auto-Narrativization Copilot
Maven Connector Archetypes
Java Client SDK
OpenCrawling CLI
OpenAPI Specification & Swagger UI
gRPC Internal Transport
Migration Guide (ManifoldCF)

🧪 Testing

Integration Test — Decoupled Pipeline
M-Files Integration Test
SharePoint Online Integration Test
Hyland OpenArch Integration Test
Local Docker Build Test
Secure MCP Server Integration Test
Narrativization Integration Test
Reporting Integration Test

Zero Trust Security and ACL Management

Zero-Trust Security & Enterprise ACL Management

📌 1. Executive Summary

🏛️ 2. Core Architectural Patterns

Pattern A: Collocated Database Pattern (Zero-Lag Query Pushdown)

1. Workflow Engines Pushdown (Flowable, Camunda, APS):

2. Alfresco Content Services Pushdown:

3. POSIX Filesystem Pushdown:

Key Benefits of Pattern A:

Pattern B: Distributed Database Pattern (OIS Metadata Pre-Filtering)

🔌 3. Connector Security Mapping Implementations

A. Alfresco Content Services (ACS) Connector

B. OASIS CMIS Repository Connector

C. Flowable BPMN Repository Connector

D. Camunda Platform 7 Repository Connector

E. Alfresco Process Services (APS) Connector

F. Local & Mounted Filesystem Connector

G. Apache Iceberg Lakehouse Connector

H. Apache StormCrawler Repository Connector

I. Relational Database (JDBC) Repository Connector

J. Doxis Repository Connector

K. gRPC Push Repository Connector

L. Hyland OpenArch Repository & Output Connectors

M. M-Files Repository Connector

N. Microsoft SharePoint Online Repository Connector

📦 4. Unified Open Ingestion Standard (OIS) Envelope

🔒 5. Query-Time Enforcement with the Secure MCP Server

How Zero-Trust Identity is Enforced:

🎛️ 6. Admin UI Configuration Controls

🧪 7. Integration Testing & Verification

Manual SQL Verification in pgvector:

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

🗺️ Documentation

🧪 Testing

🤝 Community

Clone this wiki locally

Manual SQL Verification in `pgvector`: