Skip to content

Documentum Repository Connector

Jump to bottom
textercabaceira edited this page Oct 31, 2026 · 1 revision

OpenText Documentum Repository Connector

OpenText Documentum Java Spring Boot OIS Compliance Claim Check

Module: oc-documentum-repository-connector
Connector Class: org.opencrawling.documentum.DocumentumRepositoryConnector
Client Class: org.opencrawling.documentum.DocumentumClient
DQL Builder: org.opencrawling.documentum.DocumentumDql
Document Builder: org.opencrawling.documentum.DocumentumDocumentBuilder
Security Mapper: org.opencrawling.documentum.DocumentumSecurityMapper
Properties Class: org.opencrawling.documentum.DocumentumProperties
Crawl State: org.opencrawling.documentum.DocumentumCrawlState
Spring Bean: @Component (activated when spring.opencrawling.connector.type=documentum; the Docker stack sets SPRING_OPENCRAWLING_CONNECTOR_TYPE=documentum)

📌 Overview

The OpenText Documentum Repository Connector (oc-documentum-repository-connector) crawls a repository (docbase) of an OpenText Documentum Content Server through Documentum REST Services (/dctm-rest) and streams its documents into the OpenCrawling pipeline as Open Ingestion Standard (OIS) documents—with their attributes (including the single and repeating attributes of custom types), primary content or a preferred rendition, folder paths, versions and fail-closed Zero-Trust permissions.

Important

Licence & Prerequisites: The connector requires a licensed, running OpenText Documentum Content Server with Documentum REST Services deployed and reachable from OpenCrawling. OpenCrawling does not include OpenText Documentum or any OpenText licence. OpenCrawling connects only to a Documentum deployment you already operate and license, and using Documentum this way (automated reading and indexing into a search or AI system) must be allowed by that licence. The connector is strictly read-only: it sends GET requests only and never creates, modifies or deletes anything in Documentum.

Note

Verification status: no free Documentum system exists (the Developer Edition ended with 6.x, and the official container images need an OpenText support account), so the connector has not yet been run against a real Content Server. It is verified against a Documentum REST Services mock that conforms to OpenText's published OpenAPI document for the REST API and is driven by OpenText's own REST client, with a DQL engine and Documentum's permission evaluation (see Testing & Verification). The points that could not be confirmed are checked when the connector connects, handled both ways, or resolved so that a wrong guess hides documents rather than exposing them (see Known Limitations below).

🏗️ Architecture & Flow

graph TD
    subgraph Documentum["OpenText Documentum"]
        REST["Documentum REST Services (/dctm-rest)"]
        Repo["Repository (docbase)"]
        Docs["dm_document & Custom Subtypes (CURRENT / Latest Major / All Versions)"]
        Content["Primary Content & Renditions (LOCAL enclosure)"]
        Folders["Cabinets & Folders (FOLDER ... DESCEND)"]
        ACLs["Permission Sets (dm_acl), dm_user, dm_group"]
        Audit["dm_audittrail (dm_destroy / dm_prune, optional)"]

        Repo --> REST
        Docs & Content & Folders & ACLs & Audit --> REST
    end

    subgraph OpenCrawling["OpenCrawling Crawler (Java 25)"]
        Client["DocumentumClient (GET only, Retries, Same-Host Downloads)"]
        Dql["DocumentumDql (Keyset Cursors, Server-Time-Zone Date Literals)"]
        Connector["DocumentumRepositoryConnector (DQL / Folder Crawl Modes)"]
        CrawlState["DocumentumCrawlState (Watermark, vstamps, ACL Stamps, Deletion Tracking)"]
        VThreads["Virtual Threads (StructuredTaskScope + ObservabilityTask, bounded parallelism)"]
        DocBld["DocumentumDocumentBuilder (OIS Metadata & Custom Attributes)"]
        SecMap["DocumentumSecurityMapper (Permits, Restrictions as Denies, Required Groups Excluded)"]
        TempStream["Temporary-File Content Stream (Size-Capped, Auto-Deleted)"]

        REST <-->|"GET ?dql=... (application/vnd.emc.documentum+json)"| Client
        REST -->|"GET objects/{id}, contents/content, LOCAL enclosure"| Client
        Dql --> Connector
        Client --> Connector
        Connector <--> CrawlState
        Connector --> VThreads
        VThreads --> DocBld
        VThreads --> SecMap
        VThreads --> TempStream

        OIS["OIS Document (UPSERT / DELETE Tombstone)"]
        DocBld --> OIS
        SecMap --> OIS
        TempStream --> OIS
    end

    OIS --> ClaimCheck["Claim-Check Store (Local / Apache Ozone)"]
    ClaimCheck --> Kafka["Kafka (opencrawling-documents -> Tika + chunking -> opencrawling-chunks -> embedding -> opencrawling-embedded)"]
    Kafka --> Sinks[("Vector Stores: PGVector / Milvus / Qdrant / Solr / Vespa / OpenSearch / Luxir / OpenArch")]
Loading

One Incremental Run

sequenceDiagram
    participant C as Connector
    participant R as Documentum REST Services
    participant S as Crawl State (JSON)
    participant P as Pipeline (Kafka)

    Note over C,R: Connect: refuse to crawl rather than guess
    C->>R: GET /repositories/{repo}
    C->>R: DQL: root folder exists (folder mode)
    C->>R: DQL page of batch-size rows (the server may refuse the size)
    C->>R: DQL: one cabinet again by its exact r_modify_date (date literal check)
    C->>R: GET currentuser (audit tracking: View Audit privilege)
    C->>S: load watermark, offsets, permission set stamps, known documents
    loop keyset pages: r_modify_date >= watermark
        C->>R: DQL listing (ORDER BY r_modify_date, r_object_id)
        C->>R: DQL: permission sets and users of the page (cached)
        par per document, bounded parallelism
            alt object, i_vstamp and mapped permissions unchanged
                C->>C: skip
            else changed or new
                C->>R: GET objects/{id}
                C->>R: GET contents/content?media-url-policy=local
                C->>R: GET LOCAL enclosure (content bytes)
                C->>P: OIS UPSERT
            end
        end
    end
    C->>R: DQL: folders changed below the root (folder mode), listed again
    C->>R: DQL: i_vstamp of the permission sets in use, documents of edited ones listed again
    C->>R: DQL: id sweep (reconcile) or dm_audittrail (audit)
    C->>P: OIS DELETE tombstones
    C->>S: save: advance the watermarks only if nothing failed or was cut short
Loading

✅ Prerequisites

  1. A licensed OpenText Documentum Content Server with Documentum REST Services deployed (https://<host>/dctm-rest) and reachable from the OpenCrawling runtime.
  2. The repository name (docbase), as listed at https://<host>/dctm-rest/repositories.
  3. An account that can read the documents to crawl. Through DQL a non-superuser only sees the objects it has at least BROWSE on, and can only download content it has READ on, so a superuser (or an account with READ on everything in scope) is the usual choice.
    • auth-type: basic (default): user name and password; a Documentum login ticket (DM_TICKET=...) works as the password.
    • auth-type: oauth2: client-credentials grant against OpenText Directory Services (OTDS) or another identity provider in front of REST Services, or a ready access token (bearer).
    • Kerberos (SPNEGO) is not supported.
  4. For deletion-tracking: audit only: a superuser or an account with the View Audit extended privilege (checked when the connector connects), and dm_destroy / dm_prune audited for the crawled types. The default reconcile tracking needs neither.

🚀 Key Features

  • Documentum REST Services, as OpenText's own REST clients use them: DQL through the repository's dql link, objects, content metadata with media-url-policy=local, and content from the LOCAL enclosure on the REST server itself. GET requests only, so no CSRF token is needed.
  • DQL and folder crawl modes: per object type (subtypes included), an optional DQL condition or a full DQL statement; in folder mode a cabinet or folder tree (FOLDER('/path', DESCEND)) or a single folder.
  • Version policies: CURRENT version, latest major version (n.0), or every version.
  • Content & renditions: primary content, or a preferred rendition (e.g. pdf) when present; size-capped temporary files.
  • Custom types: single and repeating attributes of custom subtypes as documentum_attr_* metadata.
  • Fail-closed Zero-Trust ACL mapping of permission sets: access permits, access restrictions as denies, dm_owner / dm_group / dm_world, required groups and required group sets left out.
  • Incremental crawls with keyset paging on r_modify_date, r_object_id, detection of edited permission sets, and deletion tracking by reconciliation or the audit trail.
  • Date-literal check on connect: DQL reads date literals in the Content Server's time zone; the connector proves its literals match before crawling, because a wrong time zone would skip documents silently.
  • Bounded concurrency on virtual threads, retries with back-off for 429 and 502–504 (honouring Retry-After), one OAuth2 token renewal on 401, credentials that never leave the REST server (content links naming a proxy's host are followed on the configured URL; ACS and BOCS hosts are refused).
  • Hardened for real servers: the page size is checked against the server's maximum on connect, IN lists stay under the servlet container's request line limit, daylight saving changes between runs skip nothing, and a document the account may browse but not read is indexed by metadata only instead of stalling every later run.
  • Safe incremental state: a failed or cut-short run advances nothing and forgets nothing, so no permission set edit and no deletion is ever lost.

⚙️ Configuration Properties

Spring properties live under spring.opencrawling.connector.documentum.* (kebab-case); a connector configuration in the Admin UI or connectors.json uses the camelCase key.

Property Key Default Description
url url http://localhost:8080/dctm-rest The REST Services context
repository-name repositoryName — The repository (docbase) name (required)
auth-type authType basic basic, oauth2 or bearer; kerberos is refused
username, password username, password — For basic
oauth2.token-url, oauth2.client-id, oauth2.client-secret, oauth2.scope oauth2TokenUrl, … — Client-credentials grant for oauth2
oauth2.access-token oauth2AccessToken — A ready token, for bearer (or oauth2)
crawl-mode crawlMode dql dql or folder
object-types objectTypes dm_document Types queried one after the other; subtypes included
where where — An extra DQL condition
dql-query dqlQuery — A full SELECT ... FROM <type> [WHERE ...]: its type and condition are used
root-cabinet-path rootCabinetPath / Cabinet or folder of folder mode
include-subfolders includeSubfolders true FOLDER(..., DESCEND), or the folder itself only
version-mode versionMode current_only current_only, latest_major or all_versions
include-content-stream includeContentStream true Download content
max-content-size-bytes maxContentSizeBytes 52428800 Larger content is indexed by metadata only; 0 = no limit
prefer-rendition preferRendition — A format indexed instead of the primary content when present, e.g. pdf
include-acls includeAcls true Map permission sets; false makes every document public
world-identity worldIdentity dm_world Group identity for dm_world; empty grants nothing
world-public worldPublic false Map dm_world to public instead
principal-format principalFormat name name (user_name) or login (user_login_name)
required-group-documents requiredGroupDocuments exclude exclude, or fallback (fallback principals only)
fallback-principals fallbackPrincipals — Readers of documents nobody can read through their permission set
include-custom-attributes includeCustomAttributes true Attributes beyond dm_sysobject's
include-system-attributes includeSystemAttributes false The remaining dm_sysobject attributes too
include-folder-paths includeFolderPaths true Resolve i_folder_id to folder paths
attribute-prefix attributePrefix documentum_attr_ Prefix of attribute metadata keys
deletion-tracking deletionTracking reconcile reconcile, audit or none
audit-deletion-enabled auditDeletionEnabled false Shorthand for deletion-tracking: audit
audit-events auditEvents dm_destroy,dm_prune Audit events that mean a version is gone
incremental-enabled incrementalEnabled true Watermark and skip-unchanged
modified-after modifiedAfter — Only objects modified since this date or instant
server-time-zone serverTimeZone auto Time zone of DQL date literals; auto uses each date's offset from REST; a region such as Europe/Berlin follows daylight saving; an invalid value is an error
clock-skew-seconds clockSkewSeconds 300 Margin subtracted from the watermark
batch-size batchSize 100 Rows per DQL page for every query; at most 1000 and at most the server's rest.paging.max.size (checked on connect)
parallelism parallelism 4 Documents processed at once
timeout-seconds timeoutSeconds 30 HTTP timeout
max-retries maxRetries 3 Retries of throttled or failed requests
max-documents maxDocuments 0 Index at most this many changed documents per run (0 = no limit); unchanged ones do not count, so successive incremental runs work through a large repository
state-directory stateDirectory data/documentum-state Incremental crawl state

📄 Example application.yml 

spring:
  opencrawling:
    connector:
      type: documentum
      documentum:
        url: "https://dctm.example.com/dctm-rest"
        repository-name: "corp_repo"
        auth-type: "basic"
        username: "dctm_crawler"
        password: "${DOCUMENTUM_PASSWORD}"
        crawl-mode: "folder"
        root-cabinet-path: "/Regulatory/Submissions"
        object-types: "my_submission"
        where: "a_status = 'Approved'"
        version-mode: "current_only"
        prefer-rendition: "pdf"
        deletion-tracking: "reconcile"
        batch-size: 100

Choosing the Modes

Setting Choose When
crawl-mode dql (default) The scope is a set of types and conditions anywhere in the repository (object-types, where, or dql-query).
folder The scope is a cabinet or folder tree (root-cabinet-path). Incremental runs then also notice folders moved into the tree.
version-mode current_only (default) Search and AI over what users see today: one OIS document per document, the CURRENT version.
latest_major Minor versions are drafts: one OIS document per document, its highest n.0 version in scope.
all_versions Records, audit or eDiscovery use: one OIS document per version, with its own id.
deletion-tracking reconcile (default) Sees every way a document leaves: destroyed, pruned, moved out of scope, its CURRENT version destroyed. One id sweep per run (one DQL page per batch-size objects in scope).
audit Very large repositories where a full id sweep per run is too expensive: reads dm_audittrail since the last run instead. Needs auditing of dm_destroy / dm_prune and the View Audit privilege; does not see documents moved out of scope.
none Append-only indexing; deletions need a full run.
principal-format name (default) The search side identifies users by their Documentum user_name.
login It identifies them by user_login_name (e.g. a directory account name).
world-identity / world-public dm_world group (default) Documents readable by dm_world are readable by members of that group in the index.
world-public: true Anyone who can query the index may read them (only when the index is not shared beyond the repository's users).

📦 OIS Document Mapping & Payloads

Document Ids

Every Documentum version is its own object (r_object_id); i_chronicle_id (the first version's id) ties a version tree together. Ids are stable per chronicle, so a new version replaces the previous one downstream:

Case OIS id
current_only, latest_major documentum://<repository>/documents/<i_chronicle_id>
all_versions documentum://<repository>/documents/<i_chronicle_id>/versions/<r_object_id>

Metadata

Key Value
name, title, mimeType object_name, title (or the name), the indexed content's MIME type
documentum.repository, documentum.objectId, documentum.chronicleId Repository, the version's r_object_id, i_chronicle_id
documentum.objectName, documentum.title, documentum.subject, documentum.type, documentum.status, documentum.authors, documentum.keywords Document attributes (documentum.status is a_status, e.g. Approved)
documentum.creator, documentum.created, documentum.modifier, documentum.modified, documentum.owner, documentum.group, documentum.lockOwner People and dates
documentum.versionLabels, documentum.versionLabel, documentum.isCurrentVersion Version labels (implicit first, then symbolic)
documentum.contentFormat, documentum.contentSize, documentum.indexedFormat, documentum.renditionFormat Primary format and size; the format indexed; the rendition, when one was indexed
documentum.aclDomain, documentum.aclName The permission set
documentum.folderPath Every folder the object is linked into
documentum_attr_<name> Attributes of custom types; repeating attributes as several values, dates as ISO instants
documentum_identity_users, documentum_identity_groups The readers

A DELETE tombstone carries documentum.status: DELETED and documentum.id, the key of the chronicle (or version) in the crawl state.

Sample Document

The connector's real output for a document of the DCTM REST mock: an oc_submission (custom type with single and repeating attributes) in version 2.1, linked into two folders, with the permission set of the worked example below:

{
  "id": "documentum://corp/documents/090277b680000169",
  "uri": "documentum://corp/documents/090277b680000169",
  "lastModified": "2026-01-06T08:39:13Z",
  "action": "UPSERT",
  "metadata": {
    "name": ["Submission 047"],
    "title": ["Submission 047 title"],
    "mimeType": ["text/plain"],
    "documentum.repository": ["corp"],
    "documentum.objectId": ["090277b68000016c"],
    "documentum.chronicleId": ["090277b680000169"],
    "documentum.objectName": ["Submission 047"],
    "documentum.title": ["Submission 047 title"],
    "documentum.subject": ["Subject of 47"],
    "documentum.type": ["oc_submission"],
    "documentum.creator": ["Rui Costa"],
    "documentum.created": ["2026-01-06T08:39:13Z"],
    "documentum.modifier": ["Rui Costa"],
    "documentum.modified": ["2026-01-06T08:39:13Z"],
    "documentum.owner": ["Rui Costa"],
    "documentum.group": ["regulatory"],
    "documentum.aclDomain": ["dmadmin"],
    "documentum.aclName": ["oc_finance_restricted"],
    "documentum.versionLabels": ["2.1", "CURRENT"],
    "documentum.versionLabel": ["2.1"],
    "documentum.isCurrentVersion": ["true"],
    "documentum.authors": ["Rui Costa"],
    "documentum.keywords": ["oc", "k2"],
    "documentum.contentFormat": ["crtext"],
    "documentum.contentSize": ["121"],
    "documentum.indexedFormat": ["crtext"],
    "documentum.folderPath": ["/Corporate/Contracts", "/Regulatory/Submissions"],
    "documentum_identity_users": ["rui costa"],
    "documentum_identity_groups": ["finance", "emea"],
    "documentum_attr_submission_id": ["SUB-0047"],
    "documentum_attr_agency": ["FDA"],
    "documentum_attr_product_codes": ["P47", "P48"],
    "documentum_attr_approved": ["false"],
    "documentum_attr_page_estimate": ["57"],
    "documentum_attr_review_dates": ["2026-02-08T00:00:00Z"]
  },
  "acl": "rui costa,finance,emea",
  "security": {
    "inheritanceEnabled": false,
    "permissions": [
      { "identity": "rui costa", "identityType": "user", "displayName": "rui costa", "access": "read" },
      { "identity": "finance", "identityType": "group", "displayName": "finance", "access": "read" },
      { "identity": "emea", "identityType": "group", "displayName": "emea", "access": "read" },
      { "identity": "mallory grey", "identityType": "user", "displayName": "mallory grey", "access": "deny" }
    ]
  }
}

Deletion Tombstone (action: "DELETE")

When the document is destroyed (or leaves the scope), the next run emits a tombstone with the same id; the writer purges its chunks, and no content, text or embedding is produced for it:

{
  "id": "documentum://corp/documents/090277b680000169",
  "uri": "documentum://corp/documents/090277b680000169",
  "action": "DELETE",
  "metadata": {
    "documentum.repository": ["corp"],
    "documentum.id": ["090277b680000169"],
    "documentum.status": ["DELETED"]
  }
}

🔒 Zero-Trust Security & ACL Mapping

A document's permission set (acl_domain, acl_name) is read with DQL from dm_acl (cached, and re-read when its i_vstamp changes). Documentum gives a user the most permissive basic permit among the entries that match them, lowers it by any matching access restriction, and takes it away when they are outside a required group or required group set.

Permission set entry (r_permit_type) OIS rule
Access permit (0) of READ (3) or more read for the accessor
Access permit below READ (NONE, BROWSE) nothing
Access restriction (3) of READ or less (takes READ away) deny for the accessor (wins over every grant)
dm_owner, dm_group the document's owner_name (user, or group when the owner is one), group_name
dm_world the world-identity group, or public with world-public; a restriction on dm_world leaves only the fallback principals
Required group (6), required group set (7) the document is left out (exclude, default) or indexed for the fallback principals only (fallback)
Extended (1, 4) and application (2, 5) permits and restrictions, alias accessors (%name) nothing
A restriction whose accessor has no identity (login format: an unknown user, or a login several login domains share) the fallback principals only: dropping it would grant that user
No reader fallback-principals, or nobody. Never public

Identities are lower-cased user and group names (principal-format: login uses user_login_name; a login name that users of several user_login_domains share identifies nobody and is never used). Where the mapping differs from Documentum it only restricts: superusers, dm_read_all / dm_browse_all members and any implicit owner rights are not represented.

Worked Example

The document of the sample above, owned by Rui Costa, with the permission set dmadmin / oc_finance_restricted:

Accessor r_permit_type r_accessor_permit Documentum OIS rule
dm_world 0 access permit 1 NONE grants nothing —
dm_owner 0 access permit 7 DELETE Rui Costa (the owner) may read rui costa user read
finance (group) 0 access permit 3 READ members may read finance group read
emea (group) 0 access permit 3 READ members may read emea group read
Mallory Grey 3 access restriction 3 READ READ taken away, although Mallory is in emea mallory grey user deny

In the index, members of finance or emea and Rui Costa find the document; Mallory Grey does not, because a deny wins over her group's grant; everyone else finds nothing. Had the permission set also had a required group (r_permit_type 6), the document would have been left out of the index altogether (required-group-documents: exclude), since "member of every required group" cannot be written as OIS rules.

The tests check this for every document and every user of the mock: the documents the OIS rules let a user read are exactly those Documentum's own evaluation lets them READ.

🔄 Incremental Crawls & Delete Detection

  • Keyset paging: listings select single-valued attributes, ordered by r_modify_date, r_object_id, and continue with r_modify_date > t OR (r_modify_date = t AND r_object_id > id) until a page comes back empty. REST page numbers re-run the query and shift when objects change; r_modify_date has one-second precision.
  • Date literals: written in the offset the REST server gave each date (server-time-zone: auto) or the configured zone, and checked on connect against a cabinet's exact r_modify_date. With auto the state records the offset of the watermark, so a daylight saving change between runs skips nothing.
  • Watermark: r_modify_date >= <watermark>; documents whose object, i_vstamp and mapped permissions are unchanged are skipped. The watermark advances only after a run without failures.
  • Edited permission sets change no document: each run compares the i_vstamp of the permission sets in use and lists the documents of a changed one again.
  • Moved folders: moving a folder saves only the folder, so in folder mode each run lists again the folders below the root saved since the watermark.
  • Content the account may browse but not read (HTTP 403), or only offered by another server, is indexed by metadata only with a warning, so it never holds the watermark back.
  • Failed and cut-short runs advance nothing and forget nothing: when a document cannot be crawled, or max-documents stops the run, the watermark, the audit watermark and the permission set stamps stay as they were, and every document the state knows stays known. The next run retries, so no permission set edit and no deletion is lost.
  • max-documents counts the documents fetched and indexed, not the unchanged ones, so successive incremental runs work through a large repository in steps.
  • Deletions: the first run tombstones everything previously known that it did not find. Later runs use reconcile (object ids in scope against the state) or audit (dm_audittrail events since the last run, read a day back). A document whose indexed object is gone is listed again by chronicle: another version in scope is re-indexed (destroying the CURRENT version makes its antecedent CURRENT without modifying it), otherwise it is tombstoned.

🌐 Documentum REST Requests

Request Use
GET <url>/repositories/<repository> Connection check
GET <url>/repositories/<repository>?dql=...&page=1&items-per-page=<batch> Listings, permission sets, users, folder paths, id sweeps, audit trail, date-literal check
GET .../currentuser Privileges, for audit tracking
GET .../objects/<id> Every attribute of a changed document
GET .../objects/<id>/contents/content?page=0&media-url-policy=local[&format=<rendition>] Content metadata and its LOCAL enclosure
GET <enclosure> The content bytes, from the REST server

🖥️ Admin UI & CLI Integration

Admin UI: OpenText Documentum (REST Services) in the repository connector list, with every setting above and a Test Connection button. It runs the same checks as a crawl's start: the repository, the root folder (folder mode), the page size and the date-literal time zone, and reports the server's time zone offset.

CLI:

oc connector check --name dctm-corp --type repository

🧪 Testing & Verification

OpenText Documentum has no free container image, so the tests run against the DCTM REST mock (oc-documentum-repository-connector/documentum-mock): a Documentum REST Services emulator (Python standard library only) with a DQL engine over typed object tables, in the response shapes of a REST 16.7 server and OpenText's REST clients. It also implements Documentum's permission evaluation, and the security tests compare the connector's mapping with it user by user. Its README lists every assumption.

1. Unit, MockWebServer & Mock Scenario Tests

mvn test -pl oc-documentum-repository-connector -am

DocumentumMappingTest (settings, DQL statements and date literals, every permission rule), DocumentumClientTest (MockWebServer, responses in src/test/resources/documentum-mock-responses) and, with DCTM_MOCK_URL set, DocumentumMockServerTest: full crawls reconciled with the mock's ground truth; for every user, the documents the OIS rules let them read are exactly those Documentum lets them READ; incremental runs after check-ins, edited permission sets, destroyed documents and versions, folder moves; failed and cut-short runs; and what a real server may do differently (see below). The scenarios also pass against a repository of 3,000 documents in a daylight saving time zone.

2. Standalone Mock Server Test

./scripts/test-documentum-repository-connector.sh

Checks the mock against OpenText's published OpenAPI document, runs the module tests against it as a server at +02:00, then again at -05:00 with every DQL page silently capped at 5 rows.

3. Mock Conformance: OpenText's OpenAPI Document, OpenText's Own Client & Independent Audits

With no Documentum system available, the mock is held to everything OpenText publishes:

  • OpenAPI document: OpenText publishes the Documentum REST API as an OpenAPI 3 document on developer.opentext.com ("Documentum D2", the D2 REST Services, which embed the core REST resources). documentum-mock/conformance/check_spec.py downloads it (it is not redistributed), crawls the mock the way a client does and checks every exchange: documented path and method, query parameters, status, media type and JSON schema. No deviation, as a superuser and as an ordinary user; the places where the document does not describe the core server are reported as warnings with the reason.
  • OpenText's own client: OpenText's Python REST client navigates the mock unmodified (home, repositories, DQL and its next links, objects, contents and downloads, cabinets → folders → documents, users, groups, types, formats, versions).
  • Constants and real responses: the DFC 26.4 javadoc confirms its permit and privilege constants, and real REST 16.7 responses its shapes.
  • Independent audits: they found what the mock got wrong and what a real server may do that it did not show. Each of those is now a mock setting (POST /mock/config, /mock/folders/move) with a connector test that fails without the connector's handling: a smaller page maximum, an 8 KB request line limit, links naming a proxy's host, content redirects, daylight saving, login names shared across login domains, folder moves, content the account may browse but not read.

4. Decoupled Pipeline End-to-End Test

./scripts/test-documentum-decoupled.sh
# LOCAL_BUILD=true builds from docker/local (BuildKit Maven cache); MCP_PORT and OC_IMAGE_TAG move the MCP port and tag the images

Crawl → Kafka → Tika → embeddings → pgvector → MCP secure search, then an incremental crawl after a new version, a destroyed document, a new document and an edited permission set.

Running the Stack Against a Real Documentum System

First run checklist, as the connector has not met a real Content Server yet:

  1. Use a dedicated read-only account (a superuser is simplest: it reads everything, and the mapping never grants more than the permission sets do).
  2. Run Test Connection in the Admin UI first: it proves the credentials, the repository, the page size and the date-literal time zone.
  3. Start with include-content-stream: false and max-documents: 100, and compare a few documents' metadata and permission rules with Documentum Administrator.
  4. Then enable content, remove the limit (or raise it for a phased first load), and keep deletion-tracking: reconcile until the audit trail's setup is confirmed.
  5. Watch the log for the warnings listed under Troubleshooting.

Set DOCUMENTUM_URL, DOCUMENTUM_REPOSITORY, DOCUMENTUM_USERNAME, DOCUMENTUM_PASSWORD (optionally DOCUMENTUM_AUTH_TYPE, DOCUMENTUM_OBJECT_TYPES, DOCUMENTUM_WHERE, DOCUMENTUM_CRAWL_MODE, DOCUMENTUM_ROOT_CABINET_PATH, DOCUMENTUM_VERSION_MODE, DOCUMENTUM_PREFER_RENDITION, DOCUMENTUM_MAX_DOCUMENTS, DOCUMENTUM_FALLBACK_PRINCIPALS, DOCUMENTUM_SERVER_TIME_ZONE) before docker compose -f oc-documentum-repository-connector/docker/docker-compose-decoupled-with-documentum.yml up. Start with DOCUMENTUM_MAX_DOCUMENTS on a large repository.

📈 Performance & Sizing

Work Requests
Listing One DQL page per batch-size objects in scope (since the watermark on incremental runs)
Permission sets and users One DQL query per page of listed rows, for those not cached yet in the run
Folder paths One DQL query per changed document whose folders are not cached yet in the run
An unchanged document None beyond its listing row
A changed or new document GET objects/{id}, GET contents/content (twice when a preferred rendition is missing), GET of the content
reconcile One DQL page per batch-size objects in scope (the id sweep)
audit One DQL page per batch-size audit events since the last run (read a day back)
  • First load of a large repository: set max-documents (e.g. 50000) and run the job repeatedly; each run indexes the next documents and the watermark is only set once a run completes. Then remove the limit.
  • parallelism (default 4) bounds the documents fetched at once; raise it when the REST server's latency, not its load, limits the crawl. Every request is a separate authenticated HTTP call; there is no session to keep.
  • batch-size (default 100, at most the server's rest.paging.max.size, 1000 by default): larger pages mean fewer DQL round trips for listings and sweeps.
  • Memory: the crawl state (one entry per document) and the reconcile id sweep are held in memory, a few hundred bytes per document; content is streamed through size-capped temporary files.
  • The mock-backed scenario tests pass against a repository of 3,000 documents in a daylight saving time zone.

🛠️ Troubleshooting

Message (log or Test Connection) Cause What to do
Documentum reads DQL date literals in another time zone than ...: cabinet ... is not found as DATE(...) DQL reads date literals in the Content Server's time zone, which differs from the one the connector writes Set server-time-zone to the Content Server's zone, preferably a region such as Europe/Berlin
Documentum date literals could not be verified (no cabinet visible) (warning) The account cannot see any cabinet, so the check could not run Give the account BROWSE on a cabinet, or set server-time-zone explicitly
Documentum listing of ... made no progress ...; is server-time-zone right? A date literal is read in another zone As above
Documentum refused a DQL page of N items (...). Lower batch-size ... batch-size is above the REST server's rest.paging.max.size Lower batch-size
Documentum REST repository answered HTTP 401 E_BAD_CREDENTIALS Wrong user name, password or expired login ticket Fix the credentials; with OAuth2, check the client and the token URL
OAuth2 token endpoint ... answered HTTP ... The identity provider refused the client credentials Check oauth2.client-id, oauth2.client-secret, oauth2.scope
Documentum folder '...' was not found, or the account cannot see it. Wrong root-cabinet-path, or no BROWSE on it Fix the path or the account's permissions
deletion-tracking 'audit' needs an account that can read dm_audittrail ... The account is neither a superuser nor has View Audit Grant View Audit, or use deletion-tracking: reconcile
Documentum object ...: the crawl account may browse it but not read its content (HTTP 403); indexed by metadata only. (warning) The account has BROWSE but not READ on the document Grant READ (or use a superuser). The content is indexed the next time the document changes, or on a full run (delete the state file)
Refusing to send Documentum credentials to <host>: the content link is not on the REST server. (warning) The server offers the content only through an ACS or BOCS host Indexed by metadata only; make REST Services serve LOCAL content
Documentum login name '...' of '...' belongs to users of several login domains ... (warning) principal-format: login and a login name shared across user_login_domains Use principal-format: name, or make the login names unique
Documentum permission set .../... was not found; its documents get no readers. (warning) The permission set cannot be read Its documents fall back to fallback-principals or nobody; check the account can read dm_acl
auth-type 'kerberos' is not supported ... SPNEGO cannot be negotiated by the connector Use basic (a login ticket works as the password) or oauth2
server-time-zone '...' is not a time zone ... An abbreviation such as CEST, or a typo Use auto, an offset such as +02:00, or a region such as Europe/Berlin

⚠️ Known Limitations

  • Never run against a real Documentum system. Unverified points are checked on connect (the time zone of DQL date literals, the page size), handled both ways (oversized items-per-page refused or capped), kept clear of (IN lists of at most 100 ids, under the servlet container's request line limit), or treated as opaque (REST error codes: only HTTP statuses are relied on).
  • Kerberos (SPNEGO) is not supported.
  • Required groups and required group sets cannot be expressed as OIS rules: such documents are left out by default.
  • Multi-page content and virtual documents: only page 0 is indexed; virtual document children are indexed on their own.
  • Audit tracking sees destroyed and pruned versions only, and needs dm_destroy registered for auditing (not checked); reconcile also sees documents leaving the scope.
  • The repeated hour when clocks go back: a date literal in it is ambiguous to DQL, so a listing crossing it then can skip documents saved in the earlier hour (once a year; a full run or reconcile recovers them).
  • Large repositories: the crawl state and the reconcile sweep are held in memory, a few hundred bytes per document.
  • Folders, cabinets and non-sysobject types are not crawled.

🗺️ Documentation

🧪 Testing

🤝 Community

Clone this wiki locally