Skip to content

FileNet Repository Connector

Jump to bottom
textercabaceira edited this page Oct 30, 2026 · 2 revisions

IBM FileNet P8 Repository Connector

IBM FileNet Java Spring Boot OIS Compliance Claim Check

Module: oc-filenet-repository-connector
Connector Class: org.opencrawling.filenet.FileNetRepositoryConnector
Client Class: org.opencrawling.filenet.FileNetClient
Document Builder: org.opencrawling.filenet.FileNetDocumentBuilder
Security Mapper: org.opencrawling.filenet.FileNetSecurityMapper
Properties Class: org.opencrawling.filenet.FileNetProperties
Crawl State: org.opencrawling.filenet.FileNetCrawlState
Spring Bean: @Component (activated when spring.opencrawling.connector.type=filenet; the Docker stack sets SPRING_OPENCRAWLING_CONNECTOR_TYPE=filenet)

📌 Overview

The IBM FileNet P8 Repository Connector (oc-filenet-repository-connector) crawls an object store of an IBM FileNet P8 Content Platform Engine through the Content Services GraphQL API (CSGQL) (/content-services-graphql/graphql) and streams its documents into the OpenCrawling pipeline as Open Ingestion Standard (OIS) documents—with their class properties, content elements, folder filings, versions and fail-closed Zero-Trust permissions.

Important

Licence & Prerequisites: The connector requires a licensed, running IBM FileNet P8 Content Platform Engine (FileNet Content Manager, on-premises or in IBM Cloud Pak for Business Automation) with the Content Services GraphQL API deployed and reachable from OpenCrawling. OpenCrawling does not include IBM FileNet P8 or any IBM licence. OpenCrawling connects only to a FileNet deployment you already operate and license, and using FileNet this way (automated reading and indexing into a search or AI system) must be allowed by that licence. The connector is strictly read-only: it never creates, modifies or deletes anything in FileNet.

Note

Verification status: no free FileNet system exists, so the connector has not yet been run against a real Content Engine. It is verified against IBM's published GraphQL schema, IBM's own Content Services MCP server client and a GraphQL mock of the API (see Testing & Verification). The points that could not be confirmed are configurable or handled so that a wrong guess hides documents rather than exposing them (see Known Limitations below).

🏗️ Architecture & Flow

graph TD
    subgraph FileNet["IBM FileNet P8 Content Platform Engine"]
        CSGQL["Content Services GraphQL API (/content-services-graphql/graphql)"]
        OS["Object Store (e.g., OS1)"]
        Docs["Documents & Version Series (Current / Released / All)"]
        Elems["Content Elements (ContentTransfer / ContentReference)"]
        Folders["Folder Filings (INSUBFOLDER / INFOLDER)"]
        ACLs["Effective Permissions (ALLOW / DENY access masks) & ActiveMarkings"]
        Audit["Audit Log (DeletionEvents, optional)"]

        OS --> CSGQL
        Docs & Elems & Folders & ACLs & Audit --> CSGQL
    end

    subgraph OpenCrawling["OpenCrawling Crawler (Java 25)"]
        Client["FileNetClient (XSRF Double-Submit, Retries, Same-Host Downloads)"]
        Connector["FileNetRepositoryConnector (Query / Folder Crawl Modes)"]
        CrawlState["FileNetCrawlState (Watermark, Fingerprints, Deletion Tracking)"]
        VThreads["Virtual Threads (StructuredTaskScope + ObservabilityTask, bounded parallelism)"]
        DocBld["FileNetDocumentBuilder (OIS Metadata & Element Lineage)"]
        SecMap["FileNetSecurityMapper (Fail-Closed Masks, Deny Wins, Markings)"]
        TempStream["Temporary-File Content Stream (Size-Capped, Auto-Deleted)"]

        CSGQL <-->|GraphQL POST + ECM-CS-XSRF-Token| Client
        CSGQL -->|GET relative downloadUrl| Client
        Client --> Connector
        Connector <--> CrawlState
        Connector --> VThreads
        VThreads --> DocBld
        VThreads --> SecMap
        VThreads --> TempStream

        OIS["OIS Document (UPSERT / DELETE Tombstone)"]
        DocBld --> OIS
        SecMap --> OIS
        TempStream --> OIS
    end

    OIS --> ClaimCheck["Claim-Check Store (Local / Apache Ozone)"]
    ClaimCheck --> Kafka["Kafka (opencrawling-documents -> Tika + chunking -> opencrawling-chunks -> embedding -> opencrawling-embedded)"]
    Kafka --> Sinks[("Vector Stores: PGVector / Milvus / Qdrant / Solr / Vespa / OpenSearch / Luxir / OpenArch")]
Loading

✅ Prerequisites

  1. IBM FileNet P8 Content Platform Engine (on-premises or IBM Cloud Pak for Business Automation) with the Content Services GraphQL API deployed (https://<host>/content-services-graphql/graphql).
  2. Object Store: symbolic name (e.g., OS1) or id.
  3. Authentication Credentials:
    • Basic Auth (auth-type: basic, default): directory user and password.
    • OAuth 2.0 (auth-type: oauth2): client-credentials grant against the platform's identity provider (Cloud Pak IAM / Zen) (oauth2.token-url, oauth2.client-id, oauth2.client-secret). The token is renewed once after an HTTP 401.
    • Bearer Token (auth-type: bearer): a ready access token (oauth2.access-token).
    • LTPA Token (auth-type: ltpa): a WebSphere Liberty LtpaToken2 cookie value.
  4. Permissions in FileNet: the account needs read access to the documents and the right to read their permissions (both part of FileNet's default "view" level), so the connector can index who may read each document. Documents the account can't see are not crawled.
  5. Deletion Auditing (optional): only for deletion-tracking: audit, deletion auditing must be enabled on the crawled document classes so Content Engine records DeletionEvents. The default reconcile mode needs no auditing.

🚀 Key Features

  • Content Services GraphQL, as IBM's own clients use it: searches are documents(from, where) paged with moreDocuments(token); id sweeps and audit events use repositoryRows(sql). Requests carry a client-generated ECM-CS-XSRF-Token as header and cookie (the API's double-submit check). GraphQL errors returned with HTTP 200 are raised as errors. Every GraphQL document the connector sends is validated against IBM's published schema.
  • Flexible Crawl Scopes:
    • query mode: per document class (subclasses included), with an optional CE SQL where condition, or a full CE SQL statement (sql-query).
    • folder mode: a folder subtree (This INSUBFOLDER '<path>') or a single folder (This INFOLDER '<path>').
  • Version Policies: the current version of each document (default), its latest released (major) version, or every version.
  • Multi-Element Content Support: every ContentTransfer element is downloaded to a temporary file and becomes its own OIS document, linked to the document (filenet.parentId, filenet.elementSequenceNumber). An external ContentReference keeps its location (filenet.contentLocation). Documents without content are indexed by their metadata; elements over max-content-size-bytes too.
  • Credentials Stay on the FileNet Host: content is fetched from each element's relative downloadUrl, resolved against /content-services-graphql; a download URL pointing to another host is refused. Secrets are never logged.
  • Fail-Closed Zero-Trust Security: FileNet's effective Permissions (direct, default, inherited, template, proxy and marking entries) are evaluated against read-access-mask (default 131201, FileNet's view level). Any deny of a read right wins. Documents with active markings are left out by default.
  • Incremental Crawls & Deletion Tracking:
    • A DateLastModified watermark, skipping documents whose version, update sequence number and permissions are unchanged; the watermark only advances after a run without failures.
    • Deletions are detected by reconciliation (the ids in scope compared with the crawl state, default) or the audit log (DeletionEvent queries), emitting OIS DELETE tombstones.
  • Java 25 Structured Concurrency: bounded parallelism on virtual threads (StructuredTaskScope + ObservabilityTask.observed), with retries and back-off for HTTP 429 and 502–504, honouring Retry-After.

⚙️ Configuration Properties

Spring Boot properties reside under spring.opencrawling.connector.filenet.* (kebab-case); the Admin UI and connectors.json use the camelCase keys.

Property (YAML) Admin UI / JSON Key Default Description
url url http://localhost:9080/content-services-graphql/graphql Content Services GraphQL endpoint; downloads resolve against /content-services-graphql.
object-store objectStore (Required) Object store symbolic name (e.g. OS1) or id.
auth-type authType basic basic, oauth2, bearer or ltpa.
username username — User for basic.
password password — Password for basic (masked in the UI, never logged).
oauth2.token-url oauth2TokenUrl — OAuth 2.0 token endpoint for oauth2.
oauth2.client-id oauth2ClientId — OAuth 2.0 client id.
oauth2.client-secret oauth2ClientSecret — OAuth 2.0 client secret.
oauth2.scope oauth2Scope — Optional OAuth 2.0 scope.
oauth2.access-token oauth2AccessToken — A ready access token for bearer (or oauth2).
ltpa-token ltpaToken — LtpaToken2 cookie value for ltpa.
crawl-mode crawlMode query query (by class / CE SQL) or folder (by folder).
document-classes documentClasses Document Comma-separated classes, one search each, subclasses included.
where where — Extra CE SQL condition, e.g. [DocumentTitle] LIKE 'Loan%'.
sql-query sqlQuery — Full CE SQL statement; its FROM class and WHERE condition replace document-classes (the API builds the SELECT itself).
root-folder-path rootFolderPath / Root folder when crawl-mode=folder.
include-subfolders includeSubfolders true true: the subtree (This INSUBFOLDER); false: the folder only (This INFOLDER).
version-mode versionMode current current, released (alias latest_major) or all (alias all_versions).
include-content-stream includeContentStream true Download content.
max-content-size-bytes maxContentSizeBytes 52428800 (50 MB) Larger elements are indexed by metadata only.
include-acls includeAcls true Map permissions; when false, documents are public.
include-custom-properties includeCustomProperties true Emit class properties as <property-prefix><name>.
include-system-properties includeSystemProperties false Also emit Content Engine system properties.
include-folder-paths includeFolderPaths true Read the folders each document is filed in.
property-prefix propertyPrefix filenet_prop_ Prefix of property metadata keys.
read-access-mask readAccessMask 131201 Access rights a grantee needs to be a reader (FileNet's view level).
authenticated-users-identity authenticatedUsersIdentity #authenticated-users Group the #AUTHENTICATED-USERS grantee maps to; empty grants nothing.
authenticated-users-public authenticatedUsersPublic false Map #AUTHENTICATED-USERS to public instead.
principal-format principalFormat name name (grantee names lower-cased) or short (cn=Finance,… → finance, CORP\jdoe → jdoe).
fallback-principals fallbackPrincipals — Identities granted read when no permission grants anyone; empty means nobody.
marked-documents markedDocuments exclude exclude: documents with active markings, or whose markings can't be read, are not indexed; index: index them with their MARKING denies applied.
deletion-tracking deletionTracking reconcile reconcile, audit or none (deletion-tracking-enabled: false is none).
deletion-event-sql deletionEventSql SELECT [Id], [SourceObjectId], [DateCreated] FROM DeletionEvent WHERE [DateCreated] > {since} The audit query.
incremental-enabled incrementalEnabled true Keep a crawl state between runs.
modified-after modifiedAfter — Lower bound on DateLastModified (ISO date or timestamp).
clock-skew-seconds clockSkewSeconds 300 The watermark is set this far before the scan start.
batch-size batchSize 100 Search page size.
parallelism parallelism 4 Documents processed at the same time.
timeout-seconds timeoutSeconds 30 Per-request timeout.
max-retries maxRetries 3 Retries for HTTP 429 and 502–504.
max-documents maxDocuments 0 (unlimited) Stop after this many documents; for test runs.
state-directory stateDirectory data/filenet-state Incremental crawl state, one file per object store and scope.

A job path starting with / (e.g. /Mortgages/Commercial) runs a one-off, non-incremental crawl of that folder subtree; default uses the configuration.

📄 Example application.yml 

spring:
  opencrawling:
    connector:
      type: filenet
      filenet:
        url: https://cpe.example.com/content-services-graphql/graphql
        object-store: OS1
        auth-type: basic
        username: p8crawler
        password: "${FILENET_PASSWORD}"
        crawl-mode: query
        document-classes: FinancialReport,Contract
        where: "[DateCreated] >= 20240101T000000Z"
        version-mode: current
        batch-size: 100
        parallelism: 8
        include-acls: true
        read-access-mask: 131201
        marked-documents: exclude
        deletion-tracking: reconcile
        max-content-size-bytes: 52428800

📦 OIS Document Mapping & Payloads

Document Ids & Multi-Element Decomposition

A FileNet Document can hold zero, one or several content elements, and each version is its own object. Ids are stable per version series, so a new version replaces the previous one downstream instead of leaving it indexed (GUID braces removed):

Case OIS id
Document (first element, or no content) filenet://<object-store>/documents/<version-series-id>
Element 1+ of the same document filenet://<object-store>/documents/<version-series-id>/elements/<n> (with filenet.parentId)
version-mode: all filenet://<object-store>/documents/<version-series-id>/versions/<version-id> (elements below it the same way)

Each element carries the document's class properties, folder filings, version fields and permissions, plus its own content type, retrieval name and sequence number.

Metadata

Key Value
name, title, mimeType Retrieval name (or the document name), the document name, the element's content type
filenet.objectStore, filenet.id, filenet.versionSeriesId Object store, the version id, the version series id
filenet.name, filenet.class, filenet.creator, filenet.dateCreated, filenet.lastModifier, filenet.dateLastModified Document fields
filenet.majorVersion, filenet.minorVersion, filenet.versionLabel, filenet.isCurrentVersion, filenet.versionStatus Version fields
filenet.contentSize, filenet.elementCount, filenet.elementSequenceNumber, filenet.retrievalName, filenet.elementSize, filenet.contentLocation, filenet.parentId Content element fields (contentLocation for external references, parentId for elements 1+)
filenet.folderPath Every folder the document is filed in
filenet_prop_<camelCaseName> Class properties by symbolic name (ContractNumber → filenet_prop_contractNumber); lists as several values, dates as ISO instants; object-valued and binary properties are skipped
filenet_identity_users, filenet_identity_groups The readers

A DELETE tombstone carries filenet.status: DELETED.

Sample Document

{
  "id": "filenet://OS1/documents/4B68E982-F5F4-4DF2-A39C-A5796A84F3E1",
  "uri": "filenet://OS1/documents/4B68E982-F5F4-4DF2-A39C-A5796A84F3E1",
  "lastModified": "2026-03-15T11:45:00Z",
  "action": "UPSERT",
  "metadata": {
    "name": ["Q1_Report.pdf"],
    "title": ["Q1 2026 Financial Overview"],
    "mimeType": ["application/pdf"],
    "filenet.objectStore": ["OS1"],
    "filenet.id": ["{0D1C2B3A-4E5F-4061-8293-A4B5C6D7E8F9}"],
    "filenet.versionSeriesId": ["{4B68E982-F5F4-4DF2-A39C-A5796A84F3E1}"],
    "filenet.class": ["FinancialReport"],
    "filenet.majorVersion": ["1"],
    "filenet.minorVersion": ["0"],
    "filenet.versionLabel": ["1.0"],
    "filenet.isCurrentVersion": ["true"],
    "filenet.folderPath": ["/Finance/2026", "/Executive/Briefings"],
    "filenet.elementCount": ["1"],
    "filenet.elementSequenceNumber": ["0"],
    "filenet_prop_fiscalQuarter": ["Q1"],
    "filenet_identity_users": ["jdoe"],
    "filenet_identity_groups": ["cn=finance,ou=groups,dc=example,dc=com"]
  },
  "acl": "cn=finance,ou=groups,dc=example,dc=com,jdoe",
  "security": {
    "inheritanceEnabled": false,
    "permissions": [
      { "identity": "mallory", "identityType": "user", "displayName": "mallory", "access": "deny" },
      { "identity": "cn=finance,ou=groups,dc=example,dc=com", "identityType": "group", "displayName": "cn=finance,ou=groups,dc=example,dc=com", "access": "read" },
      { "identity": "jdoe", "identityType": "user", "displayName": "jdoe", "access": "read" }
    ]
  }
}

🔒 Zero-Trust Security & ACL Mapping

FileNet's Permissions list is already effective: direct and default entries plus those inherited from the security parent (SOURCE_PARENT), a security template, a security proxy (PROXY) and markings (MARKING). FileNetSecurityMapper combines each grantee's ALLOW and DENY access masks:

FileNet permissions OIS rule
The grantee's allowed rights cover read-access-mask (default 131201: read, view content, read permissions) and none of them is denied read
The grantee is denied any right in read-access-mask, even view content alone deny (wins over every grant)
Allowed rights that don't cover the mask (e.g. read without view content) nothing
#AUTHENTICATED-USERS the authenticated-users-identity group, or public when authenticated-users-public: true
#CREATOR-OWNER, entries without a grantee name, role and delegated permissions nothing
A grantee of type UNKNOWN (an unresolved account, often a SID) kept verbatim as a group
No reader fallback-principals, or nobody. Never public

Identities are the grantee names lower-cased (principal-format: short reduces distinguished names and DOMAIN\user). Users are identityType: user, groups group.

Deny precedence. Content Engine ranks entries by source (a direct allow can override an inherited deny). OpenCrawling lets every deny win, so where it differs from FileNet it can only restrict access. The ManifoldCF FileNet connector, by contrast, ignores partial denies.

🏷️ Markings

A FileNet marking (a security label from a marking set) can take rights away from users the permissions otherwise allow. Whether Content Engine reports that restriction as MARKING deny entries in a document's Permissions has not been confirmed on a real system; if it doesn't, indexing marked documents from their permissions alone would expose them.

So by default (marked-documents: exclude) the connector requests each document's ActiveMarkings and leaves the document out when it:

  • has an active marking;
  • has a MARKING permission entry; or
  • doesn't report ActiveMarkings at all (logged once per run).

A document marked after it was indexed is removed on the next run, and indexed again once its markings are removed. Marked documents are missing from search, never exposed. Once the behaviour is confirmed on a Content Engine, marked-documents: index indexes them with every MARKING deny applied.

🔄 Incremental Crawls & Delete Detection

The crawl state (state-directory, one JSON file per object store and scope) holds a DateLastModified watermark and, per document, the version id, update sequence number and permission fingerprint last indexed.

  • A run searches [DateLastModified] > <watermark> and skips documents whose version, update sequence number and permissions are unchanged. The watermark only advances after a run without failures.

  • A content element a document no longer has gets a tombstone.

  • Deletions:

    • reconcile (default) lists the version ids in scope (SELECT [Id] FROM <class> WHERE <scope>) and compares them with the state;
    • audit reads DeletionEvents created since the last run.

    A document whose tracked version is gone is searched again by version series: if another version is now in scope (the current version was deleted, a version was demoted) it is re-indexed, otherwise it is tombstoned.

  • Not seen by an incremental run (schedule a periodic full crawl): permissions inherited from a folder whose own permissions changed, and documents that leave the scope without being modified.

🌐 Content Services GraphQL Operations

GraphQL Operation Purpose
_apiInfo(repositoryIdentifier) / objectStore(identifier) Connection check
documents(repositoryIdentifier, from, where, pageSize) Searches, fetching document fields, content elements, folder filings, permissions and properties in one round trip
moreDocuments(token) The next page of a search
moreFolders(token) Folder filings beyond the first page (50 by default)
repositoryRows(repositoryIdentifier, sql, pageSize) CE SQL id sweeps (reconcile) and DeletionEvent queries (audit)
moreRepositoryRows(token) The next page of those
GET <downloadUrl> Content of one element, relative to /content-services-graphql, with the same credentials and XSRF header

🖥️ Admin UI & CLI Integration

  • Admin UI Form: IBM FileNet P8 (Content Platform Engine) in the repository connector list, with every setting above (including "Documents with Markings"), credential masking, and a Test Connection button that reads _apiInfo and the object store, so wrong credentials, a rejected XSRF check or an unknown object store fail before the first crawl.
  • OpenCrawling CLI: 
    # Test the connection of a saved FileNet connector configuration
oc connector check --name filenet-p8 --type repository

🧪 Testing & Verification

IBM FileNet P8 has no free container image, so the tests run against the CSGQL mock (oc-filenet-repository-connector/filenet-mock), a real GraphQL server (graphql-core) over a hand-written subset of IBM's published Content Services GraphQL schema. Two checks keep the mock and the connector honest:

  • Conformance with IBM's schema (filenet-mock/conformance.py): the mock's schema is a strict subset of IBM's (same types, fields, arguments and enum values), and every GraphQL document the connector sends validates against IBM's full schema. IBM's schema is downloaded at a pinned commit and is not part of the repository (IBM licence).
  • Cross-check with IBM's own client: IBM's Content Services MCP server (Apache-2.0) was run against the mock. Authentication, the XSRF double submit, downloadUrl resolution, error reporting and paging agree with the connector.

1. Unit Tests

mvn test -pl oc-filenet-repository-connector -am

FileNetMappingTest (settings, ids, properties, security, markings) and FileNetClientTest (the protocol against MockWebServer: XSRF double submit, Basic/bearer/LTPA, OAuth2 renewal, retries, GraphQL errors, relative and off-host downloads). FileNetMockServerTest runs too when CSGQL_MOCK_URL is set.

2. Standalone Mock Server Test

./scripts/test-filenet-repository-connector.sh

Creates the mock's Python environment, runs the conformance check, starts the mock and runs FileNetMockServerTest against it: full crawls reconciled with the mock's ground truth, every permission pattern, folder/class/SQL/version modes, incremental runs after check-ins, deletions, permission changes and new documents (reconcile and audit), and marked documents left out, removed when marked later and indexed again when unmarked.

3. Decoupled Pipeline End-to-End Test

./scripts/test-filenet-decoupled.sh
# LOCAL_BUILD=true builds from docker/local (BuildKit Maven cache); MCP_PORT and OC_IMAGE_TAG move the MCP port and tag the images

Spins up the CSGQL mock (filenet-mock-decoupled-filenet), postgres-vector-decoupled-filenet, kafka-decoupled-filenet, redis-stack-decoupled-filenet, ollama-decoupled-filenet and the decoupled OpenCrawling services:

  • Every content element of an unmarked document is indexed exactly once, with its lineage, class properties, folder paths and permission rules; the marked document is left out.
  • MCP secure search honours permissions: a group member finds a document, a user denied only view content and an outsider find nothing.
  • An incremental crawl brings a new version, a deletion and a new document into pgvector.

Running the Stack Against a Real FileNet System

Set FILENET_URL, FILENET_OBJECT_STORE, FILENET_USERNAME and FILENET_PASSWORD (optionally FILENET_AUTH_TYPE, FILENET_DOCUMENT_CLASSES, FILENET_WHERE, FILENET_CRAWL_MODE, FILENET_ROOT_FOLDER_PATH, FILENET_MAX_DOCUMENTS, FILENET_FALLBACK_PRINCIPALS, FILENET_MARKED_DOCUMENTS) before:

docker compose -f oc-filenet-repository-connector/docker/docker-compose-decoupled-with-filenet.yml up -d

Start with FILENET_MAX_DOCUMENTS on a large object store.

⚠️ Known Limitations

  • Not yet run against a real FileNet system. Still to confirm on a Content Engine, each configurable or handled defensively:
    • the integer values of individual access rights (read-access-mask defaults to IBM's documented composite view level, 131201);
    • how markings show in a document's Permissions (marked documents are excluded by default);
    • the DeletionEvent property names (only for deletion-tracking: audit; deletion-event-sql is configurable);
    • the exact downloadUrl format (treated as opaque).
  • Source-ranked precedence: every deny wins, so the mapping can only be stricter than Content Engine.
  • Incremental runs don't see permissions inherited from a folder whose permissions changed, or documents that leave the scope without being modified; schedule a periodic full crawl.
  • One class per search: the API takes a single class in from (subclasses included), so several classes are searched one after the other; joins aren't supported.
  • Only documents are crawled: annotations, custom objects and folders themselves are not.

🗺️ Documentation

🧪 Testing

🤝 Community

Clone this wiki locally