Skip to content

M Files Repository Connector

Jump to bottom
Piergiorgio Lucidi edited this page Oct 27, 2026 · 1 revision

M-Files Repository Connector

M-Files Java Spring Boot OIS Compliance Claim Check

Module: oc-mfiles-repository-connector
Connector Class: org.opencrawling.mfiles.MFilesRepositoryConnector
Client Class: org.opencrawling.mfiles.MFilesClient
Document Builder: org.opencrawling.mfiles.MFilesDocumentBuilder
Security Mapper: org.opencrawling.mfiles.MFilesSecurityMapper
Properties Class: org.opencrawling.mfiles.MFilesProperties
Crawl State: org.opencrawling.mfiles.MFilesCrawlState
Spring Bean: @Component (activated when spring.opencrawling.connector.type=mfiles or SPRING_OPENCRAWLING_REPOSITORY_CONNECTOR_TYPE=mfiles)

📌 Overview

The M-Files Repository Connector (oc-mfiles-repository-connector) crawls an M-Files vault (M-Files Corporation) through the M-Files Web Service (MFWS) REST API (<server>/REST) and streams its objects into the OpenCrawling pipeline as Open Ingestion Standard (OIS) documents—complete with typed property values, single-file and multi-file binary streams, and fail-closed Zero-Trust permissions.

It serves as the OpenCrawling counterpart of a ManifoldCF M-Files repository connector—using the same MFWS calls, the same mfiles:user:<id> / mfiles:group:<id> permission tokens, and the same optional Vault Application Framework (VAF) change feed—rebuilt on the OpenCrawling SPI with reactive scanning, Java 25 Virtual Threads (StructuredTaskScope), binary time-window splitting, and OIS deletion tombstones.

Important

Licence & Session Policy: The connector requires a licensed, running M-Files Server (or M-Files Cloud vault) with the M-Files Web Service (REST API) enabled and an account with read access to the vault. OpenCrawling does not include M-Files or any M-Files licence. Each crawl scan holds one MFWS session and always ends it (DELETE /session, with fallback to POST /session?_method=DELETE on IIS).

🏗️ Architecture & Flow

graph TD
    subgraph MFiles["M-Files Vault (On-Premises or Cloud)"]
        Vault["M-Files Document Vault"]
        MFWS["M-Files Web Service (MFWS REST API /REST)"]
        VAF["Vault Application Framework (Optional /vaf/...)"]
        Objects["Object Versions (Single-File, Multi-File, Non-Doc Objects)"]
        Props["Property Definitions & Typed Values (MFDataType)"]
        Perms["Effective ACL Components & Named ACLs"]
        
        Vault --> MFWS
        Vault --> VAF
        Objects & Props & Perms --> MFWS
    end

    subgraph OpenCrawling["OpenCrawling Ingestion Runtime (Java 25)"]
        Client["MFilesClient (Session, Token/Basic/OAuth2, Retries)"]
        Connector["MFilesRepositoryConnector"]
        TimeSplit["Recursive Time-Window Splitter (p21>= / p21<=)"]
        VThreads["Virtual Threads (StructuredTaskScope + Semaphore)"]
        DocBld["MFilesDocumentBuilder & MFilesValues"]
        SecMap["MFilesSecurityMapper (Fail-Closed Component Intersection)"]
        TempStream["TempFileInputStream (Size-Capped Spool & Auto-Delete)"]
        CrawlState["MFilesCrawlState (Incremental Watermark & File Lineage)"]
        ClaimCheck["Claim-Check Store (Local / Apache Ozone / S3)"]
        Tika["Apache Tika 4 Text Extraction"]

        MFWS <-->|REST JSON| Client
        VAF <-->|Change Feed & Normalized Objects| Client
        Client --> Connector
        Connector --> TimeSplit
        Connector --> CrawlState
        Connector --> VThreads
        VThreads --> DocBld
        VThreads --> SecMap
        VThreads --> TempStream
        TempStream --> ClaimCheck
        ClaimCheck --> Tika

        OIS["OIS Document (UPSERT / Tombstone DELETE)"]
        DocBld --> OIS
        SecMap --> OIS
        Tika --> OIS

        OIS --> Kafka["Kafka Pipeline (opencrawling-ingestion -> chunks -> embedded)"]
        Kafka --> Sinks[("Vector Stores: PGVector / Milvus / Qdrant / Solr / Vespa / OpenSearch / Luxir / OpenArch")]
    end

    subgraph AI["AI Applications & Zero-Trust MCP"]
        Sinks --> SecureMcp["OpenCrawling Secure MCP Server (McpAclEvaluator)"]
        SecureMcp -->|Zero-Trust ACL & Explicit Deny Enforcement| Agent["Enterprise RAG Assistant / Agent"]
    end
Loading

✅ Prerequisites

Before configuring the connector or starting a crawl, verify the following requirements:

  1. A licensed M-Files Server or M-Files Cloud Vault with the M-Files Web Service (https://<server>/REST) enabled and reachable from the OpenCrawling runtime.
  2. The Vault GUID (vault-guid, with or without braces {...}). Without a vault GUID, M-Files issues a server-level token that cannot read any vault.
  3. A Read-Only Crawl Account:
    • auth-type: token (default) or basic: an M-Files user or Windows account (domain). Empty credentials are rejected because M-Files responds with an anonymous token that fails every vault call.
    • auth-type: oauth2: an OAuth 2.0 / Microsoft Entra ID client accepted by the vault, or a pre-issued bearer access token.
    • To resolve permissions by name (principal-format: name), the account must also have read access to the Users (6) and User Groups (16) value lists.
  4. Connection Verification: Use the Admin UI Test Connection button or oc connector check --name <name> --type repository before the first crawl. Both authenticate, verify the vault GUID and object types (GET /session/vault, GET /structure/objecttypes), and cleanly close the session.

🚀 Key Features

  • Four Crawl Modes (crawl-mode):
    1. Search Crawl (search, default): Queries GET /objects per configured class (class-ids, as p100=<id>), restricted to object-type (0 = Document, -1 = all object types) and optional MFWS search conditions (search-query, e.g., p1020=5 or q=contract; plain text without = runs a quick search).
    2. All Objects (all_objects): Same as search without class or query restrictions.
    3. View Crawl (view): Traverses an M-Files view (view-id, e.g., 101 or v101/L5), recursively descending into sub-views, lookup/text property folders, and traditional folders (up to depth 32). Objects appearing in multiple folders are deduplicated and emitted once.
    4. Vault Application Framework Change Feed (vaf): Consumes the change feed (<vaf-base-path>/changes?since=<cursor>) and normalized object endpoint (<vaf-base-path>/object/{objType}/{objId}/{version}/normalized) of an M-Files VAF app (matching the ManifoldCF M-Files protocol), where effective permissions—including metadata-driven pseudo-users—are resolved inside the vault.
  • Complete Listings Beyond the MFWS Limit:
    • MFWS truncates search results at limit (batch-size, default 500) and offers no offset pagination.
    • Whenever a search returns MoreResults: true, the connector automatically re-queries in last-modified time windows (p21>= / p21<=), recursively splitting windows in half until every window fits without truncation, and deduplicating objects on window boundaries.
  • Flexible Authentication & Session Safety:
    • token (default): Exchanges credentials + VaultGuid + SessionID at POST /server/authenticationtokens for an X-Authentication token.
    • basic: Sends X-Username, X-Password (with DOMAIN\username when domain is set), and X-Vault headers.
    • oauth2: Sends Authorization: Bearer <token> and X-Vault using either a direct oauth2.access-token or a client-credentials grant (MFilesOAuth2TokenSource) with automatic refresh before token expiry.
    • Token Renewal & Host Pinning: Automatically renews the token once on HTTP 401/403. Download URLs from VAF payloads are only followed on the configured MFWS host so credentials never leak to external hosts.
  • Single-File, Multi-File, and Non-File Objects:
    • Single-File Documents (SFD): Emitted under mfiles://<vaultGuid>/objects/<type>/<id>.
    • Multi-File Documents (MFD): Every constituent file is emitted as its own OIS document (mfiles://<vaultGuid>/objects/<type>/<id>/files/<fileId>) with mfiles.parentId pointing to the parent object URI, making agreements and all attached exhibits independently searchable.
    • Non-File Objects: Metadata-only objects (e.g., Customers, Projects) are indexed with their typed properties.
  • Version Modes (version-mode):
    • latest_only (default): Emits the latest version under a versionless stable URI so new versions cleanly replace old ones downstream.
    • all_versions: Crawls GET /objects/{type}/{id}/history and emits older versions under …/versions/<n> (and …/versions/<n>/files/<fileId>).
  • Typed Property Normalization (MFilesValues):
    • Resolves GET /structure/properties and normalizes every property value by MFDataType under <property-prefix><camelCaseName> (e.g., Contract Value $\rightarrow$ mfiles_prop_contractValue), with lookup item IDs emitted under …_id.
  • Zero-Trust Security Mapping (MFilesSecurityMapper):
    • Fail-closed evaluation of GET /objects/{type}/{id}/{version}/permissions and VAF aclTokens/denyTokens (see Zero-Trust Security Mapping below).
  • Incremental Synchronization & Tombstone Deletions:
    • Tracks per-vault, per-scope watermarks and emitted child IDs in data/mfiles-state/.
    • Searches with d=include to emit action: "DELETE" tombstones for deleted objects and for files removed from multi-file documents.
  • Java 25 Virtual Threads & Resource Hygiene:
    • Processes object batches concurrently using StructuredTaskScope.open() and ObservabilityTask.observed(...), bounded by a Semaphore (parallelism, default 4).
    • Streams binary attachments to size-capped temporary files via TempFileInputStream, guaranteeing file deletion on stream close.

⚙️ Configuration Properties

The connector can be configured via Spring Boot properties (spring.opencrawling.connector.type=mfiles and spring.opencrawling.connector.mfiles.*) or dynamically through the Admin UI / connectors.json (org.opencrawling.mfiles.MFilesRepositoryConnector, camelCase keys).

Spring Property (spring.opencrawling.connector.mfiles.*) connectors.json Key Default Description
url url http://localhost/REST MFWS base URL, including /REST
vault-guid vaultGuid — Required. Target vault GUID, with or without braces
auth-type authType token Authentication mode: token, basic, or oauth2
username, password username, password — Required for token and basic auth
domain domain — Optional Windows domain for the account
oauth2.token-url, oauth2.client-id, oauth2.client-secret, oauth2.scope oauth2TokenUrl, oauth2ClientId, oauth2ClientSecret, oauth2Scope — OAuth2 / Entra ID client-credentials grant settings
oauth2.access-token oauth2AccessToken — Pre-issued OAuth2 bearer access token
crawl-mode crawlMode search Crawl mode: search, view, all_objects, or vaf
object-type objectType 0 Object type ID (0 = Document; -1 crawls all object types)
class-ids classIds — Class IDs to filter (one search per class); empty crawls all classes
search-query searchQuery — MFWS search conditions (e.g., p1020=5&q=contract); plain text without = runs a quick search
view-id viewId — Required in view mode: view ID (101) or view path (v101/L5)
version-mode versionMode latest_only latest_only or all_versions
include-files includeFiles true Download attached binary files
max-file-size-bytes maxFileSizeBytes 52428800 Maximum file size in bytes (50 MB; larger files emit metadata only)
include-properties includeProperties true Emit typed property values as <property-prefix><name>
property-prefix propertyPrefix mfiles_prop_ Prefix for normalized property metadata keys
include-acls includeAcls true Extract object permissions (false marks documents public)
fallback-principals fallbackPrincipals — Fallback readers when permissions are unreadable (group:X, user:Y, public); empty = nobody
public-group-ids publicGroupIds — M-Files user group IDs treated as public (e.g., 1 for "All internal users")
principal-format principalFormat token token (mfiles:user:<id>, mfiles:group:<id>) or name (resolved via value lists)
users-value-list, user-groups-value-list usersValueList, userGroupsValueList 6, 16 Value list IDs for users (6) and user groups (16) when principal-format: name
qualify-principals qualifyPrincipals false Prefix every identity with the vault GUID (<vaultGuid>/<identity>)
emit-deletes emitDeletes true Search deleted objects (d=include) and emit DELETE tombstones
incremental-enabled incrementalEnabled false Track watermark state, skip unchanged objects, and purge removed child IDs
modified-after modifiedAfter — Lower bound on last-modified timestamp (2026-01-01 or 2026-01-01T00:00:00Z)
clock-skew-seconds clockSkewSeconds 300 Incremental watermark safety overlap (seconds subtracted from scan start)
state-directory stateDirectory data/mfiles-state Directory storing per-vault/per-scope incremental JSON state files
vaf-base-path vafBasePath — Required in vaf mode (e.g., /vaf/indexer)
vaf-change-feed-path, vaf-normalized-path vafChangeFeedPath, vafNormalizedPath /changes, /object/{objType}/{objId}/{version}/normalized VAF endpoint paths
batch-size batchSize 500 Maximum results per MFWS search request (limit)
parallelism parallelism 4 Concurrent objects processed via Java 25 Virtual Threads
timeout-seconds timeoutSeconds 30 HTTP request timeout in seconds
max-retries maxRetries 3 Retries with backoff on HTTP 429, 502–504, and refused connections
max-documents maxDocuments 0 Stop after N objects (0 = unlimited; useful for test probes)

Tip

Scan Path Overrides: Passing a non-default scan-path (or job path) overrides the configured selection for that run: MFWS conditions containing = or plain text run a search, and a path starting with v<digits> (e.g., v101 or v101/L5) crawls that view. Path-override runs do not touch the incremental state file.

📄 Example application.yml

1. Token Authentication (Default)

spring:
  opencrawling:
    repository-connector-type: mfiles
    connector:
      type: mfiles
      mfiles:
        url: "https://mfiles.example.com/REST"
        vault-guid: "{C0B3B52E-6512-4BD8-8F2B-2A8E9A349201}"
        auth-type: "token"
        username: "mfiles_service_user"
        password: "${MFILES_PASSWORD}"
        crawl-mode: "search"
        object-type: 0
        class-ids:
          - 14  # Non-Disclosure Agreement
          - 22  # Master Services Agreement
        version-mode: "latest_only"
        include-files: true
        max-file-size-bytes: 52428800
        fallback-principals: "group:records-managers"
        incremental-enabled: true
        modified-after: "2026-01-01T00:00:00Z"
        batch-size: 500
        parallelism: 4
        timeout-seconds: 30

2. OAuth 2.0 / Microsoft Entra ID Authentication

spring:
  opencrawling:
    repository-connector-type: mfiles
    connector:
      type: mfiles
      mfiles:
        url: "https://mfiles.example.com/REST"
        vault-guid: "{C0B3B52E-6512-4BD8-8F2B-2A8E9A349201}"
        auth-type: "oauth2"
        oauth2:
          token-url: "https://login.microsoftonline.com/<tenant>/oauth2/v2.0/token"
          client-id: "opencrawling-client-id"
          client-secret: "${MFILES_CLIENT_SECRET}"
          scope: "<mfiles-app-scope>/.default"

📦 OIS Document Mapping & Payloads

1. Single-File Document (action: "UPSERT")

{
  "id": "mfiles://C0B3B52E-6512-4BD8-8F2B-2A8E9A349201/objects/0/48192",
  "action": "UPSERT",
  "acl": "mfiles:user:12,mfiles:group:5",
  "metadata": {
    "name": ["Global Vendor NDA - Final Signed.pdf"],
    "title": ["Global Vendor NDA - Final Signed"],
    "mimeType": ["application/pdf"],
    "mfiles.vaultGuid": ["C0B3B52E-6512-4BD8-8F2B-2A8E9A349201"],
    "mfiles.vaultName": ["Contracts"],
    "mfiles.objType": ["0"],
    "mfiles.objId": ["48192"],
    "mfiles.version": ["3"],
    "mfiles.isLatestVersion": ["true"],
    "mfiles.title": ["Global Vendor NDA - Final Signed"],
    "mfiles.classId": ["14"],
    "mfiles.className": ["Non-Disclosure Agreement"],
    "mfiles.singleFile": ["true"],
    "mfiles.fileCount": ["1"],
    "mfiles.fileId": ["7"],
    "mfiles.fileName": ["Global Vendor NDA - Final Signed.pdf"],
    "mfiles.fileSize": ["1482910"],
    "mfiles.created": ["2026-04-10T10:30:00Z"],
    "mfiles.createdBy": ["sarah.connor"],
    "mfiles.lastModified": ["2026-09-27T15:20:00Z"],
    "mfiles.lastModifiedBy": ["alex.vance"],
    "mfiles.state": ["Approved"],
    "mfiles_identity_users": ["mfiles:user:12"],
    "mfiles_identity_groups": ["mfiles:group:5"],
    "mfiles_prop_customer": ["Cyberdyne Systems"],
    "mfiles_prop_customer_id": ["5"],
    "mfiles_prop_contractValue": ["150000"],
    "mfiles_prop_effectiveDate": ["2026-05-01"],
    "mfiles_prop_tags": ["Legal", "Vendor"]
  },
  "security": {
    "inheritanceEnabled": false,
    "permissions": [
      { "identity": "mfiles:user:12", "identityType": "user", "access": "read" },
      { "identity": "mfiles:group:5", "identityType": "group", "access": "read" },
      { "identity": "mfiles:user:9", "identityType": "user", "access": "deny" }
    ]
  }
}
  • Multi-File Documents: Each file of a multi-file document 48193 is emitted under mfiles://<vaultGuid>/objects/0/48193/files/11, …/files/12, etc., with mfiles.parentId set to mfiles://<vaultGuid>/objects/0/48193.
  • Historical Versions (all_versions): Older versions receive /versions/<n> (e.g., mfiles://<vaultGuid>/objects/0/48192/versions/2 and …/versions/2/files/<fileId>).
  • Additional Metadata Keys: mfiles.displayId, mfiles.workflow, mfiles.checkedOut.

2. Deletion Tombstone (action: "DELETE")

Emitted for each document/file URI previously produced by a deleted object:

{
  "id": "mfiles://C0B3B52E-6512-4BD8-8F2B-2A8E9A349201/objects/0/48192",
  "action": "DELETE",
  "metadata": {
    "mfiles.vaultGuid": ["C0B3B52E-6512-4BD8-8F2B-2A8E9A349201"],
    "mfiles.objType": ["0"],
    "mfiles.objId": ["48192"],
    "mfiles.status": ["DELETED"]
  }
}

3. Typed Property Normalization (MFDataType)

MFDataType Example Input Normalized OIS Value
Date (5) 2026-05-01T00:00:00Z 2026-05-01
Timestamp (7) 2026-09-27T17:20:00+02:00 2026-09-27T15:20:00Z
Integer (2), Integer64 (11), Floating (3) 150000.00 150000
Boolean (8) true true / false
Lookup (9) {Item: 5, DisplayValue: "Cyberdyne Systems"} Cyberdyne Systems (and ID 5 under …_id)
MultiSelectLookup (10) Multiple lookup items All display values (deleted lookup items are dropped)
Text (1), MultiLineText (13), Time (6) String value Raw string value
Any other type — Server DisplayValue

🔒 Zero-Trust Security & ACL Mapping

The connector reads GET /objects/{type}/{id}/{version}/permissions (or VAF aclTokens / denyTokens) and maps permissions on a strict fail-closed basis:

M-Files Permission Construct OIS Security Rule
Entry (UserOrGroupID, IsGroup) with ReadPermission: 1 (or "Allow", or bit 1 of AccessFlags) access: "read" as mfiles:user:<id> or mfiles:group:<id>
Any other numeric ReadPermission code, or "Deny" access: "deny" for that identity (explicit deny wins; unverified codes can only restrict)
Multiple active ACL components (CustomComponent + AutomaticComponents from workflow states or named ACLs) Restrictive intersection: only identities allowed by every active component (IsActive != false) receive read
Pseudo-users (IsPsudoUser: true or HasConcreteUserOrGruopID: false, e.g., "Project manager") Grants nothing over MFWS (cannot be resolved without VAF); resolved 1-to-1 in vaf mode
Group ID listed in public-group-ids (e.g., 1 for "All internal users") Mapped to public
Unreadable permissions or 404 on /permissions Uses fallback-principals, or nobody (never public by default)
  • Name Resolution (principal-format: name): Resolves user and group IDs to names via built-in value lists 6 (Users) and 16 (User Groups), falling back to tokens if an ID is unknown.
  • Multi-Vault Qualification (qualify-principals: true): Prefixes every identity with <vaultGuid>/ so per-vault user/group IDs cannot collide across shared multi-vault vector indexes.

🌐 MFWS REST Endpoints Used

Operation MFWS Endpoint
Authentication POST /server/authenticationtokens (Username, Password, Domain, VaultGuid, SessionID) $\rightarrow$ {"Value": token} (X-Authentication)
Session End DELETE /session (falls back to POST /session?_method=DELETE when IIS rejects DELETE)
Vault Info GET /session/vault (Name, GUID)
Structure GET /structure/properties, GET /structure/objecttypes
Object Search GET /objects?o=<type>&p100=<class>&p21>=<from>&p21<=<to>&d=include&limit=<n> $\rightarrow$ {Items, MoreResults}
Views GET /views/<path>/items $\rightarrow$ {Items: [FolderContentItem], MoreResults}
Properties GET /objects/{type}/{id}/{version}/properties
Version History GET /objects/{type}/{id}/history
File Content GET /objects/{type}/{id}/{version}/files/{fileId}/content
Permissions GET /objects/{type}/{id}/{version}/permissions
Value Lists GET /valuelists/{id}/items (principal-format: name)
VAF <vaf-base-path>/changes, <vaf-base-path>/object/{objType}/{objId}/{version}/normalized

🖥️ Admin UI & CLI Integration

Admin UI (oc-admin-ui)

  1. Open Connectors $\rightarrow$ New Connector.
  2. Select M-Files Repository (Knowledge Work Automation) (org.opencrawling.mfiles.MFilesRepositoryConnector).
  3. Configure the MFWS URL, Vault GUID, Authentication (token, basic, or oauth2), and Crawl Mode (search, view, all_objects, or vaf).
  4. Click Test Connection to authenticate, verify the vault GUID and object types, and close the test session.

OpenCrawling CLI (oc-cli)

# Verify M-Files vault connectivity
oc connector check --name mfiles-vault --type repository

🧪 Testing & Verification

1. Unit & MockWebServer Tests

mvn test -pl oc-mfiles-repository-connector -am

Runs unit and MockWebServer tests against MFWS fixtures in src/test/resources/mfiles-mock-responses/ covering class searches, recursive time-window splitting, single- and multi-file documents, typed property values, restrictive ACL component intersection, explicit denies, tombstones, version modes, incremental state, views, VAF change feeds, all three auth modes, token renewal, retries, and size-capped downloads.

2. Standalone Connector Test with the Stateful MFWS Mock

./scripts/test-mfiles-repository-connector.sh

Probes MFILES_URL (read-only GET /server/status), starts the built-in MFWS Mock (oc-mfiles-repository-connector/mfiles-mock/mfws_mock.py — a stateful Python emulator serving a deterministic generated vault with real PDF, DOCX, and TXT files), and executes the full test suite plus MFilesMockServerTest over real HTTP.

3. Full Decoupled Pipeline Integration Test (9 Steps)

./scripts/test-mfiles-decoupled.sh

Boots oc-mfiles-repository-connector/docker/docker-compose-decoupled-with-mfiles.yml (MFWS mock, Kafka, Redis, PostgreSQL/pgvector, Ollama mxbai-embed-large, oc-crawler, oc-ingestion-consumer, oc-embedding-consumer, oc-writer-consumer, and oc-mcp-server) and validates:

  1. Full vault crawl reconciled against the mock's ground truth (/mock/stats): every live file indexed once, single-file and multi-file (mfiles.parentId) lineage, typed properties, and absence of deleted objects.
  2. Zero-Trust ACL component intersection and explicit deny rules verified in pgvector and via the Secure MCP Server (secureVectorSearch, getDocumentContent, listAccessibleSources over Streamable HTTP /mcp).
  3. Live vault mutations (removing a file from a multi-file document, deleting a document, creating a new NDA) followed by an incremental crawl that updates pgvector and purges removed/deleted chunks via tombstones.

4. Running Against a Real M-Files Server / Live Probe

To run the read-only live probe against a licensed M-Files server:

MFILES_LIVE_PROBE=1 \
MFILES_URL=https://<server>/REST \
MFILES_VAULT="{<vault-guid>}" \
MFILES_USER=<user> \
MFILES_PASSWORD=<password> \
mvn test -pl oc-mfiles-repository-connector -Dtest=MFilesLiveProbeTest -Dsurefire.failIfNoSpecifiedTests=false

🗺️ Documentation

🧪 Testing

🤝 Community

Clone this wiki locally