Skip to content

SharePoint Online Repository Connector

Jump to bottom
Piergiorgio Lucidi edited this page Oct 27, 2026 · 1 revision

SharePoint Online Repository Connector

SharePoint Online Java Spring Boot OIS Compliance Claim Check

Module: oc-sharepoint-repository-connector
Connector Class: org.opencrawling.sharepoint.SharePointRepositoryConnector
Client Class: org.opencrawling.sharepoint.GraphClient
Document Builder: org.opencrawling.sharepoint.SharePointDocumentBuilder
Security Mapper: org.opencrawling.sharepoint.SharePointSecurityMapper
Principals Helper: org.opencrawling.sharepoint.SharePointPrincipals
Properties Class: org.opencrawling.sharepoint.SharePointProperties
Crawl State: org.opencrawling.sharepoint.SharePointCrawlState
Spring Bean: @Component (activated when spring.opencrawling.connector.type=sharepoint or SPRING_OPENCRAWLING_REPOSITORY_CONNECTOR_TYPE=sharepoint)

📌 Overview

The SharePoint Online Repository Connector (oc-sharepoint-repository-connector) crawls Microsoft SharePoint Online document libraries through the Microsoft Graph v1.0 API (https://graph.microsoft.com/v1.0) and streams files into the OpenCrawling pipeline as Open Ingestion Standard (OIS) documents—complete with binary content streams, SharePoint list item column metadata (sharepoint_field_<InternalName>), incremental @odata.deltaLink state and deletion tombstones, and fail-closed Zero-Trust permissions (SharePointSecurityMapper).

Important

Microsoft 365 & Entra ID Requirement: The connector requires a Microsoft 365 tenant with SharePoint Online and an Entra ID app registration granted read access (Sites.Read.All or Sites.Selected) to the target site(s). OpenCrawling does not include Microsoft 365.

🏗️ Architecture & Flow

graph TD
    subgraph M365["Microsoft 365 & SharePoint Online"]
        Entra["Microsoft Entra ID (OAuth 2.0 client_credentials)"]
        Graph["Microsoft Graph v1.0 (/sites, /drives, /delta, /permissions)"]
        Drives["Document Libraries (driveItems, Folders, Files)"]
        ListItem["SharePoint List Item Columns (listItem/fields)"]
        CDN["Pre-Authenticated Download URLs (@microsoft.graph.downloadUrl)"]

        Entra -->|Bearer Token| Graph
        Drives & ListItem --> Graph
        Drives --> CDN
    end

    subgraph OpenCrawling["OpenCrawling Ingestion Runtime (Java 25)"]
        Client["GraphClient (Token Cache, ISV User-Agent, Shared Throttle Gate)"]
        Connector["SharePointRepositoryConnector (Delta & Tree Crawl Modes)"]
        CrawlState["SharePointCrawlState (deltaLink, eTag & Permission Fingerprint)"]
        VThreads["Virtual Threads (StructuredTaskScope + Semaphore)"]
        DocBld["SharePointDocumentBuilder (OIS Metadata & List Item Fields)"]
        SecMap["SharePointSecurityMapper (Fail-Closed Entra, Site Group & Link ACLs)"]
        TempStream["TempFileInputStream (Size-Capped Spool & Auto-Delete)"]
        ClaimCheck["Claim-Check Store (Local / Apache Ozone / S3)"]
        Tika["Apache Tika 4 Text Extraction"]

        Graph <-->|REST JSON + Retry-After| Client
        CDN -->|No Bearer Token Leaked| Client
        Client --> Connector
        Connector <--> CrawlState
        Connector --> VThreads
        VThreads --> DocBld
        VThreads --> SecMap
        VThreads --> TempStream
        TempStream --> ClaimCheck
        ClaimCheck --> Tika

        OIS["OIS Document (UPSERT / Tombstone DELETE)"]
        DocBld --> OIS
        SecMap --> OIS
        Tika --> OIS

        OIS --> Kafka["Kafka Pipeline (opencrawling-ingestion -> chunks -> embedded)"]
        Kafka --> Sinks[("Vector Stores: PGVector / Milvus / Qdrant / Solr / Vespa / OpenSearch / Luxir / OpenArch")]
    end

    subgraph AI["AI Applications & Zero-Trust MCP"]
        Sinks --> SecureMcp["OpenCrawling Secure MCP Server (McpAclEvaluator)"]
        SecureMcp -->|Zero-Trust User/Group & Site-Group ACL Enforcement| Agent["Enterprise RAG Assistant / Agent"]
    end
Loading

✅ Prerequisites

  1. A Microsoft Entra ID App Registration in the target tenant with:
    • A Client ID (client-id) and Client Secret (client-secret), or a pre-issued bearer token (access-token) for short-lived tests.
    • Application permissions (not delegated) with admin consent:
      • Sites.Read.All — read all sites in the tenant; or
      • Sites.Selected — read only sites explicitly granted to the app via POST /sites/{siteId}/permissions (recommended for least-privilege production crawls).
  2. A Target Site:
    • Either a Site URL (site-url, e.g., https://contoso.sharepoint.com/sites/Engineering, resolved via GET /sites/{hostname}:/{server-relative-path}) or a Graph Site ID (site-id, e.g., contoso.sharepoint.com,<spsite-guid>,<spweb-guid>).
  3. Network Access from the crawler to https://login.microsoftonline.com and https://graph.microsoft.com, plus the SharePoint tenant host (https://<tenant>.sharepoint.com) from which @microsoft.graph.downloadUrl serves file bytes.
  4. Connection Verification: Run Test Connection in the Admin UI or oc connector check --name <name> --type repository before starting a crawl. Both acquire a token, resolve the site, and list its document libraries (GET /sites/{siteId}/drives).

🚀 Key Features

  • App-Only OAuth 2.0 (client_credentials):
    • Acquires tokens from https://login.microsoftonline.com/{tenantId}/oauth2/v2.0/token for https://graph.microsoft.com/.default, caches them in memory, and refreshes them before expiry or once on HTTP 401.
  • Site & Document Library Selection:
    • Resolves sites by site-id or site-url.
    • Crawls every documentLibrary drive on the site by default, or only libraries matching drive-ids or drive-names (e.g., Documents, Policies).
  • Two Complementary Crawl Modes (crawl-mode):
    1. delta (default): Walks GET /drives/{driveId}/root/delta across @odata.nextLink pages and persists the final @odata.deltaLink per drive (SharePointCrawlState). Subsequent runs with incremental-enabled: true request only changes since that delta link:
      • New or modified files $\rightarrow$ action: "UPSERT" (skipped when eTag and the SHA-256 permission fingerprint match the stored state).
      • Deleted items (deleted facet or @removed) $\rightarrow$ action: "DELETE" tombstones. Deleting a folder cascades DELETE tombstones to every file previously recorded under that folder path.
      • Moved or renamed items / folders $\rightarrow$ updates tracked paths and skips content re-ingestion when the eTag and permissions are unchanged.
      • Expired delta token (410 Gone) $\rightarrow$ follows the Location header (or restarts from /root/delta), reconciles every live item against stored state, emits UPSERT only for changed items, and emits DELETE tombstones for items that disappeared while the token was expired.
    2. tree (Folder Traversal & Permission-Only Reconciliation): Walks GET /drives/{driveId}/root/children (or /root:/{path}:/children when root-folder-path is set), recursing into subfolders when recursive: true. With incremental-enabled: true, compares each file's eTag and permission fingerprint against stored state so permission-only updates are re-emitted and vanished files produce DELETE tombstones.
  • SharePoint List Item Columns (include-custom-fields: true):
    • Calls GET /drives/{driveId}/items/{itemId}/listItem?$expand=fields and emits custom SharePoint columns as sharepoint_field_<InternalName> (configurable via field-prefix), reducing person/lookup/managed-metadata objects to their display labels and skipping built-in SharePoint system columns (@odata.*, _UIVersionString, Edit, DocIcon, etc.).
  • Secure Binary Download via Claim Check:
    • Streams file bytes from the pre-authenticated @microsoft.graph.downloadUrl without forwarding the Graph bearer token (falling back to /drives/{driveId}/items/{itemId}/content 302 redirect if the pre-signed URL has expired).
    • Spools content to a size-capped temporary file (TempFileInputStream, bounded by max-file-size-bytes, default 50 MB) that deletes itself on stream close, and hands the stream to the OpenCrawling Claim-Check store and Apache Tika.
  • Shared Throttling Gate & Resilience:
    • Sends the Microsoft-recommended User-Agent format (ISV|OpenCrawling|OpenCrawling-SharePoint-Connector/1.0) and a unique client-request-id UUID on every request.
    • Shares a single throttle gate across all concurrent Virtual Thread workers: when any request receives 429 Too Many Requests, 503 Service Unavailable, or 504 Gateway Timeout, all workers pause until the server's Retry-After window (with a 1-second floor) has elapsed, then retry with exponential backoff and jitter.
    • Server errors (500, 502, 507) fail the item and prevent the @odata.deltaLink from advancing past an unindexed file.
  • Java 25 Structured Concurrency:
    • Processes drive items concurrently via StructuredTaskScope.open() and ObservabilityTask.observed(...), bounded by a Semaphore (parallelism, default 4).

⚙️ Configuration Properties

Configure the connector via Spring Boot properties (spring.opencrawling.connector.type=sharepoint and spring.opencrawling.connector.sharepoint.*) or through the Admin UI / connectors.json (org.opencrawling.sharepoint.SharePointRepositoryConnector, camelCase keys).

Spring Property (spring.opencrawling.connector.sharepoint.*) connectors.json Key Default Description
graph-base-url graphBaseUrl https://graph.microsoft.com/v1.0 Microsoft Graph v1.0 base URL
tenant-id tenantId — Required (unless token-url or access-token is set). Microsoft Entra ID tenant ID or domain (contoso.onmicrosoft.com)
client-id, client-secret clientId, clientSecret — Required (unless access-token is set). Entra app registration credentials
scope scope https://graph.microsoft.com/.default OAuth 2.0 scope for the client-credentials grant
token-url tokenUrl (derived from tenant-id) Explicit OAuth 2.0 token endpoint override
access-token accessToken — Pre-issued bearer token (testing only; bypasses token endpoint)
site-id siteId — Graph site ID (hostname,spsite-id,spweb-id). Either site-id or site-url is required
site-url siteUrl — SharePoint site URL (https://contoso.sharepoint.com/sites/Engineering)
drive-ids driveIds — Document library drive IDs to crawl (empty = all documentLibrary drives on the site)
drive-names driveNames — Document library display names to crawl (Documents, Policies)
root-folder-path rootFolderPath / Subfolder path within each drive (/Contracts/2026); filters delta items to this prefix
crawl-mode crawlMode delta delta (@odata.deltaLink change tracking) or tree (folder traversal)
recursive recursive true Recurse into subfolders in tree mode
include-content includeContent true Download binary file content (false emits metadata only)
max-file-size-bytes maxFileSizeBytes 52428800 Maximum file size to download in bytes (50 MB; larger files emit metadata only)
include-custom-fields includeCustomFields true Fetch listItem?$expand=fields and emit sharepoint_field_<InternalName>
field-prefix fieldPrefix sharepoint_field_ Metadata key prefix for SharePoint list item columns
include-acls includeAcls true Fetch /items/{id}/permissions (false marks documents public)
fallback-principals fallbackPrincipals — Fallback readers when permissions are unreadable (group:<id>, user:<upn>, public); empty = nobody
user-identity userIdentity email email (emit lower-case email/UPN and Entra object ID when both exist) or id (Entra object ID only)
everyone-identity everyoneIdentity — Principal emitted for the tenant-wide Everyone claim (c:0(.s|true). Empty = ignored (fail-closed)
everyone-except-external-identity everyoneExceptExternalIdentity — Principal emitted for Everyone except external users (spo-grid-all-users). Empty = ignored (fail-closed)
organization-link-identity organizationLinkIdentity — Principal emitted for organization-scoped sharing links (scope: organization). Empty = ignored (fail-closed)
anonymous-link-identity anonymousLinkIdentity — Principal emitted for anonymous sharing links (scope: anonymous, e.g., public). Empty = ignored (fail-closed)
emit-deletes emitDeletes true Emit DELETE tombstones for removed files and vanished tree items
incremental-enabled incrementalEnabled false Persist @odata.deltaLink, eTag, and permission fingerprints under state-directory
state-directory stateDirectory data/sharepoint-state Directory storing per-site/per-drive incremental state JSON files
page-size pageSize 200 Page size ($top) for Graph list and delta requests
parallelism parallelism 4 Concurrent files processed via Java 25 Virtual Threads
timeout-seconds timeoutSeconds 30 Per-request HTTP timeout in seconds
max-retries maxRetries 4 Retries on 429, 502–504, and transient network errors
max-documents maxDocuments 0 Stop after N documents (0 = unlimited; useful for test probes)
user-agent userAgent ISV|OpenCrawling|OpenCrawling-SharePoint-Connector/1.0 User-Agent header sent to Microsoft Graph

Tip

Scan Path Override: Passing a non-default scan-path (e.g., /Legal/NDAs) scopes the crawl to that folder for that run without modifying the persisted full-drive incremental state file.

📄 Example application.yml 

spring:
  opencrawling:
    repository-connector-type: sharepoint
    connector:
      type: sharepoint
      sharepoint:
        tenant-id: "11111111-2222-3333-4444-555555555555"
        client-id: "aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee"
        client-secret: "${SHAREPOINT_CLIENT_SECRET}"
        site-url: "https://contoso.sharepoint.com/sites/Engineering"
        drive-names:
          - "Documents"
          - "Specifications"
        crawl-mode: "delta"
        include-content: true
        max-file-size-bytes: 52428800
        include-custom-fields: true
        include-acls: true
        user-identity: "email"
        fallback-principals: "group:99999999-0000-1111-2222-333333333333"
        incremental-enabled: true
        emit-deletes: true
        page-size: 200
        parallelism: 4

📦 OIS Document Mapping & Payloads

1. File Document (action: "UPSERT")

Each file is emitted with a stable URI sharepoint://<siteId>/drives/<driveId>/items/<itemId>:

{
  "id": "sharepoint://contoso.sharepoint.com,11111111-2222-3333-4444-555555555555,66666666-7777-8888-9999-000000000000/drives/b!drive123/items/01ABCDEF23456789",
  "action": "UPSERT",
  "acl": "alice@contoso.com,aaaa1111-bbbb-2222-cccc-333333333333,dddd4444-eeee-5555-ffff-666666666666,sharepoint-group:contoso.sharepoint.com,11111111-2222-3333-4444-555555555555,66666666-7777-8888-9999-000000000000:7",
  "metadata": {
    "name": ["Architecture-Decision-Record-012.pdf"],
    "title": ["ADR 012 - Event Driven Ingestion"],
    "mimeType": ["application/pdf"],
    "sharepoint.siteId": ["contoso.sharepoint.com,11111111-2222-3333-4444-555555555555,66666666-7777-8888-9999-000000000000"],
    "sharepoint.siteName": ["Engineering"],
    "sharepoint.siteWebUrl": ["https://contoso.sharepoint.com/sites/Engineering"],
    "sharepoint.driveId": ["b!drive123"],
    "sharepoint.driveName": ["Documents"],
    "sharepoint.driveType": ["documentLibrary"],
    "sharepoint.itemId": ["01ABCDEF23456789"],
    "sharepoint.name": ["Architecture-Decision-Record-012.pdf"],
    "sharepoint.path": ["/Architecture/Architecture-Decision-Record-012.pdf"],
    "sharepoint.parentPath": ["/Architecture"],
    "sharepoint.parentId": ["01PARENTFOLDERID"],
    "sharepoint.webUrl": ["https://contoso.sharepoint.com/sites/Engineering/Shared%20Documents/Architecture/Architecture-Decision-Record-012.pdf"],
    "sharepoint.eTag": ["\"{D9B3F1E0-1234-4BCD-8EF0-112233445566},3\""],
    "sharepoint.cTag": ["\"c:{D9B3F1E0-1234-4BCD-8EF0-112233445566},2\""],
    "sharepoint.size": ["482910"],
    "sharepoint.mimeType": ["application/pdf"],
    "sharepoint.quickXorHash": ["aGVsbG8gd29ybGQ="],
    "sharepoint.created": ["2026-04-10T10:30:00Z"],
    "sharepoint.createdBy": ["Alice Smith"],
    "sharepoint.createdByEmail": ["alice@contoso.com"],
    "sharepoint.lastModified": ["2026-09-27T15:20:00Z"],
    "sharepoint.lastModifiedBy": ["Bob Jones"],
    "sharepoint.lastModifiedByEmail": ["bob@contoso.com"],
    "sharepoint.listItemId": ["42"],
    "sharepoint.contentType": ["Document"],
    "sharepoint_identity_users": ["alice@contoso.com", "aaaa1111-bbbb-2222-cccc-333333333333"],
    "sharepoint_identity_groups": [
      "dddd4444-eeee-5555-ffff-666666666666",
      "sharepoint-group:contoso.sharepoint.com,11111111-2222-3333-4444-555555555555,66666666-7777-8888-9999-000000000000:7"
    ],
    "sharepoint_field_Title": ["ADR 012 - Event Driven Ingestion"],
    "sharepoint_field_Department": ["Platform Engineering"],
    "sharepoint_field_Confidentiality": ["Internal"],
    "sharepoint_field_ReviewStatus": ["Approved"]
  },
  "security": {
    "inheritanceEnabled": true,
    "permissions": [
      { "identity": "alice@contoso.com", "identityType": "user", "access": "read" },
      { "identity": "aaaa1111-bbbb-2222-cccc-333333333333", "identityType": "user", "access": "read" },
      { "identity": "dddd4444-eeee-5555-ffff-666666666666", "identityType": "group", "access": "read" },
      { "identity": "sharepoint-group:contoso.sharepoint.com,11111111-2222-3333-4444-555555555555,66666666-7777-8888-9999-000000000000:7", "identityType": "group", "access": "read" }
    ]
  }
}

2. Deletion Tombstone (action: "DELETE")

Emitted when a delta feed reports a deleted file or folder (or when a previously tracked file is absent during a tree crawl or 410 Gone delta resync):

{
  "id": "sharepoint://contoso.sharepoint.com,11111111-2222-3333-4444-555555555555,66666666-7777-8888-9999-000000000000/drives/b!drive123/items/01ABCDEF23456789",
  "action": "DELETE",
  "metadata": {
    "sharepoint.siteId": ["contoso.sharepoint.com,11111111-2222-3333-4444-555555555555,66666666-7777-8888-9999-000000000000"],
    "sharepoint.driveId": ["b!drive123"],
    "sharepoint.itemId": ["01ABCDEF23456789"],
    "sharepoint.path": ["/Architecture/Architecture-Decision-Record-012.pdf"],
    "sharepoint.status": ["DELETED"]
  }
}

🔒 Zero-Trust Security & ACL Mapping

For each file (include-acls: true), SharePointSecurityMapper reads GET /drives/{driveId}/items/{itemId}/permissions and maps Microsoft Graph permission resources into OIS PermissionRules on a strict fail-closed basis. SharePoint permissions are additive (no deny ACEs exist in Graph driveItem permissions), and only roles that grant content read access (read, write, owner, sp.full control, sp.contribute, sp.read) are accepted—metadata-only or custom roles such as sp.restricted view are ignored.

Graph Permission Facet Principal Emitted (identity) identityType Notes
Person (grantedToV2.user / grantedToIdentitiesV2[].user / grantedTo.user + siteUser) Lower-case email / UPN (from user.email or siteUser.loginName claim i:0#.f|membership|<upn>) and Entra object ID (user.id); or user.id only when user-identity: id user Display names and site-local numeric siteUser.id are never used as principals
Entra Security or M365 Group (grantedToV2.group, or siteUser claim c:0t.c|tenant|<guid> / c:0o.c|federateddirectoryclaimprovider|<guid>) Entra group object ID (<guid>) group Matches Entra group claims in user JWTs (groups claim)
SharePoint Site Group (grantedToV2.siteGroup, e.g., Engineering Members, ID 7) sharepoint-group:<siteId>:<siteGroup.id> group Site-qualified so numeric site group IDs never collide across sites
Specific-People Sharing Link (link.scope = "users") Each recipient in grantedToIdentitiesV2 / grantedToIdentities user / group Link itself grants nothing beyond its listed recipients
Tenant Everyone (c:0(.s|true) Value of everyone-identity (if configured) (as configured) Ignored by default (fail-closed)
Everyone except external users (c:0-.f|rolemanager|spo-grid-all-users/...) Value of everyone-except-external-identity (if configured) (as configured) Ignored by default (fail-closed)
Organization Sharing Link (link.scope = "organization") Value of organization-link-identity (if configured) (as configured) Ignored by default (fail-closed)
Anonymous Sharing Link (link.scope = "anonymous") Value of anonymous-link-identity (e.g., public, if configured) (as configured) Ignored by default (fail-closed)
Application Identity (grantedToV2.application) (Ignored) — App identities are not human readers
Empty or Unreadable Permissions fallback-principals (if configured), otherwise nobody — Never public by default

Querying with the Secure MCP Server (oc-mcp-server)

At query time, pass the user's UPN/email (or Entra object ID) as userPrincipal and their Entra group object IDs plus any resolved sharepoint-group:<siteId>:<groupId> memberships as userRoles:

{
  "name": "secureVectorSearch",
  "arguments": {
    "query": "event driven ingestion architecture",
    "userPrincipal": "alice@contoso.com",
    "userRoles": "dddd4444-eeee-5555-ffff-666666666666,sharepoint-group:contoso.sharepoint.com,11111111-2222-3333-4444-555555555555,66666666-7777-8888-9999-000000000000:7",
    "maxResults": 5
  }
}

🔄 Incremental Crawls & Permission-Only Changes

When incremental-enabled: true, SharePointCrawlState stores a JSON state file per (siteId, driveId) in state-directory (data/sharepoint-state/ by default) containing:

  • The last @odata.deltaLink URL.
  • A map of itemId $\rightarrow$ {uri, path, eTag, permissionFingerprint} for every live file, plus folderId $\rightarrow$ folderPath for folder move/delete resolution.

Note

Reconciling Permission-Only Changes: Microsoft Graph driveItem/delta only surfaces permission-only changes (where a file's ACL changes without its eTag changing) when the caller holds Sites.FullControl.All. With least-privilege Sites.Read.All or Sites.Selected, schedule a periodic crawl-mode: tree run with incremental-enabled: true: the connector walks the folder tree, compares each file's current SHA-256 permission fingerprint against the stored fingerprint, and re-emits only files whose content (eTag) or permissions actually changed.

🌐 Microsoft Graph v1.0 Endpoints Used

Operation Endpoint
Acquire Token POST https://login.microsoftonline.com/{tenantId}/oauth2/v2.0/token (grant_type=client_credentials)
Resolve Site by URL GET /sites/{hostname}:/{server-relative-path}
Get Site by ID GET /sites/{siteId}
List Document Libraries GET /sites/{siteId}/drives
Get Single Drive GET /drives/{driveId}
Delta Crawl GET /drives/{driveId}/root/delta?$top={pageSize} and stored @odata.deltaLink
Tree Crawl (Root) GET /drives/{driveId}/root/children?$top={pageSize}
Tree Crawl (Subfolder Path) GET /drives/{driveId}/root:/{path}: $\rightarrow$ GET /drives/{driveId}/items/{folderId}/children?$top={pageSize}
List Item Columns GET /drives/{driveId}/items/{itemId}/listItem?$expand=fields
Item Permissions GET /drives/{driveId}/items/{itemId}/permissions
File Content Pre-authenticated @microsoft.graph.downloadUrl (no Authorization header), fallback GET /drives/{driveId}/items/{itemId}/content

🖥️ Admin UI & CLI Integration

Admin UI (oc-admin-ui)

  1. Open Connectors $\rightarrow$ New Connector.
  2. Select Microsoft SharePoint Online (Microsoft Graph API) (org.opencrawling.sharepoint.SharePointRepositoryConnector).
  3. Configure the Entra Tenant ID, Client ID, Client Secret, Site URL (or Site ID), Crawl Mode (delta or tree), and optional Broad Access Grant mappings.
  4. Click Test Connection to validate the OAuth 2.0 token exchange, site resolution, and document library listing.

OpenCrawling CLI (oc-cli)

# Verify SharePoint Online site & drive reachability
oc connector check --name sharepoint-eng --type repository

🧪 Testing & Verification

The SharePoint Online connector ships with a 6-tier verification suite:

1. Unit, MockWebServer & Official Graph Docs Conformance Tests

mvn test -pl oc-sharepoint-repository-connector -am

Runs unit and MockWebServer tests plus GraphDocsExamplesTest, which parses verbatim JSON responses from Microsoft's official microsoftgraph/microsoft-graph-docs-contrib documentation (site-get.md, drive-list.md, driveitem-delta.md, driveitem-list-children.md, driveitem-list-permissions.md, listitem-get.md) through GraphClient, SharePointDocumentBuilder, and SharePointSecurityMapper.

2. Stateful Graph Mock + OpenAPI & Documentation Conformance

./scripts/test-sharepoint-repository-connector.sh

Starts the stateful Python Microsoft Graph v1.0 SharePoint Mock (oc-sharepoint-repository-connector/graph-mock/graph_mock.py — serving deterministic PDF, DOCX, and TXT files across two document libraries with cursor-based @odata.deltaLink, 410 Gone token expiry, throttling injection, and live mutation endpoints), runs SharePointMockServerTest, and validates every mock response against the published microsoftgraph/msgraph-metadata OpenAPI schemas (check_conformance.py --strict and documented_permissions.py).

3. Microsoft Dev Proxy Chaos & Throttling Verification

./scripts/test-sharepoint-devproxy.sh

Starts Microsoft Dev Proxy (GraphRandomErrorPlugin + RetryAfterPlugin at a 50% failure rate injecting real Graph 429 Too Many Requests, 500, 502, 503, 504, and 507 responses with Retry-After headers and request-id / client-request-id echoing) in front of graph_mock.py and runs SharePointDevProxyChaosTest.

4. Independent Graph Server + 16-User RBAC Oracle Verification

./scripts/test-sharepoint-independent-mock.sh

Runs the connector against the independent third-party sergey-shulyak/mock-graph-server Express server and executes compare_acl_oracle.py, verifying across a 16-user persona matrix (9 files × 16 users = 144 access decisions) that OpenCrawling's McpAclEvaluator produces 0 false allows and 0 false denies compared to an independent SharePoint RBAC evaluator.

5. Full Decoupled Pipeline Integration Test (Kafka + Tika + Ollama + pgvector + MCP)

./scripts/test-sharepoint-decoupled.sh
# Or build from local workspace source with BuildKit ~/.m2 cache:
LOCAL_BUILD=true ./scripts/test-sharepoint-decoupled.sh

Boots oc-sharepoint-repository-connector/docker/docker-compose-decoupled-with-sharepoint.yml and verifies end-to-end:

  1. Full crawl of all live files across both SharePoint libraries into pgvector (vector_store_1024), reconciled against /mock/stats.
  2. Custom SharePoint list item columns (sharepoint_field_*) and extracted text chunks.
  3. Zero-Trust ACL enforcement through the Secure MCP Server (secureVectorSearch, getDocumentContent, listAccessibleSources) for Entra users, Entra groups, SharePoint site groups (sharepoint-group:<siteId>:7), specific-people sharing links, and fail-closed unmapped organization/anonymous links.
  4. Live SharePoint mutations (updating a file, deleting a file, creating a new file) followed by an incremental @odata.deltaLink crawl that indexes the new file, updates the modified file, and removes the deleted file's chunks from pgvector.

6. Live SharePoint Online Tenant Probe (Read-Only)

SHAREPOINT_LIVE_PROBE=1 \
SHAREPOINT_TENANT_ID="<tenant-guid>" \
SHAREPOINT_CLIENT_ID="<app-client-id>" \
SHAREPOINT_CLIENT_SECRET="<app-client-secret>" \
SHAREPOINT_SITE_URL="https://<tenant>.sharepoint.com/sites/<site>" \
mvn test -pl oc-sharepoint-repository-connector -Dtest=SharePointLiveProbeTest -Dsurefire.failIfNoSpecifiedTests=false

⚠️ Known Limitations

  • SharePoint Lists & Site Pages: Only documentLibrary drives are crawled. Generic SharePoint lists without binary files and /sites/{id}/pages site pages are not crawled in this release.
  • SharePoint Site Group Membership Expansion: Microsoft Graph /items/{id}/permissions returns siteGroup IDs and display names, not member lists. The connector emits sharepoint-group:<siteId>:<groupId> so callers can supply site-group memberships in userRoles at query time, or teams can assign permissions via Entra ID security / M365 groups whose object IDs are already present in user tokens.
  • Sensitivity Labels / RMS Encryption: Files encrypted with Microsoft Purview Information Protection (RMS) are downloaded as ciphertext unless the app registration and downstream Tika extractor are configured to decrypt them.

🗺️ Documentation

🧪 Testing

🤝 Community

Clone this wiki locally