Skip to content

Doxis Repository Connector

Jump to bottom
Piergiorgio Lucidi edited this page Oct 17, 2026 · 1 revision

Doxis Repository Connector

Doxis Java Spring Boot OIS Compliance Claim Check

Module: oc-doxis-repository-connector
Client Module: oc-doxis-client
Connector Class: org.opencrawling.doxis.DoxisRepositoryConnector
Client Class: org.opencrawling.doxis.client.DoxisClient
Document Builder: org.opencrawling.doxis.DoxisDocumentBuilder
Security Mapper: org.opencrawling.doxis.DoxisSecurityMapper
Spring Bean: @Component (activated when spring.opencrawling.connector.type=doxis or CONNECTOR_TYPE=doxis)

📌 Overview

The Doxis Repository Connector enables high-throughput, security-aware crawling and ingestion of enterprise documents, electronic files (e-files / Akten), custom metadata descriptors, and zero-trust access control policies from Doxis 4 (by SER Group) into OpenCrawling.

Compatible with Doxis Content Service Bus (CSB) REST API (contract version 14.4.1 or higher), the connector bridges Doxis ECM with OpenCrawling’s Open Ingestion Standard (OIS), Java 25 Virtual Threads (StructuredTaskScope), Apache Tika extraction, Spring AI embeddings, and the Secure Model Context Protocol (MCP) Server.

Important

Licence & Session Policy: The connector requires a valid, licensed Doxis 4 installation (SER Group) whose licence allows API / technical-user sessions. OpenCrawling does not ship Doxis or any Doxis licence. A crawl scan uses exactly one CSB session for its entire run and logs out at completion (guaranteed via finally blocks); the connection check opens and closes a single session.

🏗️ Architecture & Flow

graph TD
    subgraph Doxis["Doxis Intelligent Content Automation (SER Group)"]
        CSB["Doxis Content Service Bus (CSB 14.4+)"]
        WebAPI["CSB REST Gateway (:8080/restws/.../api/v1)"]
        Docs["Document Repository & Archives"]
        eFiles["Electronic Files (Akten) Hierarchies"]
        Descriptors["Typed Metadata Descriptors"]
        Security["Doxis ACL Engine (Users, Groups, Roles)"]
        
        CSB --> WebAPI
        Docs & eFiles & Descriptors & Security --> CSB
    end

    subgraph OpenCrawling["OpenCrawling Ingestion Runtime (Java 25)"]
        DoxisClient["DoxisClient (Shared CSB REST Client)"]
        Connector["DoxisRepositoryConnector"]
        VThreads["Virtual Threads (StructuredTaskScope)"]
        SecMap["DoxisSecurityMapper (ACE -> OIS Permissions)"]
        DocBld["DoxisDocumentBuilder (Descriptors & Normalization)"]
        TempStream["TempFileInputStream (Eager Spool & Auto-Delete)"]
        ClaimCheck["Binary Content Stream (Claim Check Store)"]
        Tika["Apache Tika Text Extraction"]
        CrawlState["DoxisCrawlState (Incremental JSON State)"]
        
        WebAPI <-->|REST JSON (Basic / Ticket / OAuth2)| DoxisClient
        DoxisClient --> Connector
        Connector --> CrawlState
        Connector --> VThreads
        VThreads --> SecMap
        VThreads --> DocBld
        VThreads --> TempStream
        TempStream --> ClaimCheck
        ClaimCheck --> Tika
        
        OIS["OIS Document (UPSERT / Tombstone DELETE)"]
        DocBld --> OIS
        SecMap --> OIS
        Tika --> OIS
        
        OIS --> Kafka["Kafka Topic (opencrawling-raw-documents)"]
        Kafka --> Ingestion["oc-ingestion-consumer (Chunking & Embeddings)"]
        Ingestion --> Sinks[("Vector Stores: PGVector / Milvus / Qdrant / Solr / Vespa / Luxir")]
    end

    subgraph AI["AI Applications & MCP Agents"]
        Sinks --> SecureMcp["OpenCrawling Secure MCP Server"]
        SecureMcp -->|Zero-Trust ACL Filtering| Agent["Enterprise RAG Assistant / Agent"]
    end
Loading

🚀 Key Features

  • Dual Crawl Modes:
    1. Search Crawl Mode (SEARCH): Runs a CQL statement (SELECT * FROM <repository> [WHERE <searchQuery>]) paged through GET /documents/searchResults/{searchId} using 1-based offsets. Search result sets are always closed on the server (DELETE /documents/searchResults/{searchId}). The listing terminates based on distinct document IDs, preventing infinite loops or skipped documents.
    2. Folder / E-File Crawl Mode (FOLDER): Traverses an electronic dossier (Akte) starting from root-folder-id, recursing through folder nodes (folderNodes), child nodes (childrenFolderNodes), and sub-e-files (searchHitsCompoundEntityWsTO). Documents linked across multiple nodes are deduplicated and emitted once.
  • Flexible Authentication:
    • Basic Auth (basic): Username, password, customer/tenant, and role (POST /login).
    • Session Ticket (ticket): Existing CSB session ticket (POST /loginBySessionTicket).
    • OAuth2 / OIDC (oauth2): Direct access token or automatic client-credentials grant exchange with token refresh (POST /loginOIDCWithAccessToken).
    • Session Renewal: Expired CSB sessions during long crawls are transparently re-authenticated once.
  • Open Ingestion Standard (OIS) Document Model:
    • Normalizes document IDs to doxis://<customer>/<repository>/documents/<uuid> (older versions get /versions/<n>).
    • Maps metadata fields: name, title (from ObjectName), mimeType, doxis.documentId, doxis.customer, doxis.repository, doxis.documentClass, doxis.version, doxis.createdDate, doxis.createdBy, doxis.modifiedDate, doxis.modifiedBy, doxis.parentFolderId, and doxis.parentFolderName.
  • Typed Descriptor Normalization:
    • Custom descriptors are normalized into standard formats:
      • DATE: Formatted to YYYY-MM-DD (e.g. 20230222 $\rightarrow$ 2023-02-22).
      • DATETIME: Formatted to ISO-8601 (YYYY-MM-DDTHH:mm:ss).
      • INTEGER, LONGINTEGER: Parsed as clean integer strings.
      • FLOATINGPOINT: Normalized with standard decimal points.
      • BOOL: Normalized to true/false.
    • If normalized, original raw values are retained under doxis.raw.<name>.
    • Reserved key protection: unprefixed descriptors matching OIS keys (title, mimeType, id, acl) are mapped to doxis.attr.<name>.
  • Zero-Trust Access Control (ACL) Mapping:
    • Evaluates Access Control Entries (ACEs) for users, groups, and roles.
    • VIEW_DOCUMENT_CONTENTS and VIEW_FOLDER_CONTENTS map to read (or deny if DENY variant); UPDATE_* maps to write.
    • Merges e-file (Akte) parent ACEs with document-level ACEs (effective permission inheritance).
    • Fail-Secure Fallbacks: Class-level permissions not exposed via REST fall back to fallback-principals or default to empty ACL (deny by default); documents are never made public by mistake.
    • Tenant Isolation: When tenant-isolation: true, all identities are qualified with the customer name (e.g., DX4/Legal), preventing privilege bleed across multi-tenant vector indexes.
  • Claim-Check Binary Streaming with Clean Lifecycle:
    • Downloads document content objects into temporary files using TempFileInputStream, which guarantees deletion of the temp file on close().
    • Enforces max-content-size-bytes threshold (default 50 MB) to skip massive binaries while indexing metadata.
    • Emits external content links (doxis.contentLink) without content streams.
  • Incremental Synchronization & Tombstone Deletes:
    • State is atomically tracked in data/doxis-state/<customer>-<repository>.json.
    • Unchanged documents (matching modificationDate) are skipped without making per-document REST calls.
    • Emits action: "DELETE" tombstones for logically deleted documents (logicalDeleted: true) and physically deleted documents that disappeared from complete search listings.
  • High-Throughput Concurrency with Java 25:
    • Uses Java 25 Virtual Threads (StructuredTaskScope) with ObservabilityTask.observed(...) for distributed tracing context propagation.
    • Concurrency bounded by parallelism (default 2) via Semaphore to prevent session exhaustion on shared Doxis servers.

⚙️ Configuration Properties

Spring Property (spring.opencrawling.connector.doxis.*) connectors.json Key Default Description
url url http://localhost:8080/restws/publicws/rest/api/v1 Base URL of Doxis CSB REST API
auth-type authType basic Authentication mode: basic, ticket, or oauth2
customer-name customerName — Required. CSB customer (tenant), e.g. DX4
username, password username, password — Credentials for basic auth
session-ticket sessionTicket — Session ticket for ticket auth
oauth2.token-url, oauth2.client-id, oauth2.client-secret, oauth2.scope oauth2TokenUrl, oauth2ClientId, oauth2ClientSecret, oauth2Scope — OAuth2 client credentials grant settings
oauth2.access-token oauth2AccessToken — Direct OAuth2 bearer access token
role role admins Role to assume during session
repository-id repositoryId — Required. Target DMS repository UUID or name
crawl-mode crawlMode search Crawl mode: search or folder
document-classes documentClasses — Document classes to filter (names or UUIDs)
search-query searchQuery — CQL WHERE filter, e.g. OBJECTNAME LIKE 'Contract*'
root-folder-id rootFolderId — E-file UUID (required in folder mode)
include-subfolders includeSubfolders true In folder mode: descend into child nodes and sub-e-files
version-mode versionMode latest_only latest_only (current version) or all_versions
include-content-stream includeContentStream true Download binary content streams
max-content-size-bytes maxContentSizeBytes 52428800 Maximum binary payload size in bytes (50 MB)
include-descriptors includeDescriptors true Ingest custom metadata descriptors
descriptor-prefix descriptorPrefix — Prefix for descriptor keys (e.g. doxis_desc_)
include-acls includeAcls true Extract document and e-file ACLs
fallback-principals fallbackPrincipals — Fallback readers for documents without instance ACEs (e.g. group:Legal)
tenant-isolation tenantIsolation false Qualify identities with customer name (e.g., DX4/Legal)
emit-logical-deletes emitLogicalDeletes true Emit DELETE tombstones for logically deleted documents
incremental incremental false Search mode: track state and skip unchanged documents
state-directory stateDirectory data/doxis-state Local path where incremental state JSON files are stored
batch-size batchSize 100 Number of hits per search page
parallelism parallelism 2 Number of documents processed concurrently
timeout-seconds timeoutSeconds 120 CSB HTTP connection and read timeout
max-retries maxRetries 2 Retries on 429 (Retry-After), 502–504
max-documents maxDocuments 0 Maximum documents to crawl (0 = unlimited)

📄 Example application.yml 

spring:
  opencrawling:
    repository-connector-type: doxis
    connector:
      type: doxis
      doxis:
        url: "https://csb.enterprise.local/restws/publicws/rest/api/v1"
        customer-name: "DX4"
        auth-type: "basic"
        username: "crawler_service"
        password: "${DOXIS_PASSWORD}"
        role: "admins"
        repository-id: "DB1"
        crawl-mode: "search"
        search-query: "OBJECTNAME LIKE 'MSA*'"
        document-classes:
          - "ContractDocument"
          - "LegalDossier"
        version-mode: "latest_only"
        include-content-stream: true
        max-content-size-bytes: 52428800
        include-descriptors: true
        descriptor-prefix: "doxis_desc_"
        include-acls: true
        fallback-principals: "group:Legal"
        tenant-isolation: true
        emit-logical-deletes: true
        incremental: true
        state-directory: "data/doxis-state"
        batch-size: 100
        parallelism: 2
        timeout-seconds: 120

📦 OIS Document Payloads

1. Ingestion Document (action: "UPSERT")

{
  "id": "doxis://DX4/DB1/documents/3f2a9c1e-5b7d-4e8a-9c0f-1a2b3c4d5e6f",
  "action": "UPSERT",
  "metadata": {
    "name": ["CON-2026-0042.pdf"],
    "title": ["Master Services Agreement 2026"],
    "mimeType": ["application/pdf"],
    "doxis_desc_ObjectName": ["Master Services Agreement 2026"],
    "doxis_desc_ObjectNumberExternal": ["CTR-2026-0042"],
    "doxis_desc_ObjectDate": ["2026-03-15"],
    "doxis.raw.ObjectDate": ["20260315"],
    "doxis.documentId": ["3f2a9c1e-5b7d-4e8a-9c0f-1a2b3c4d5e6f"],
    "doxis.customer": ["DX4"],
    "doxis.repository": ["DB1"],
    "doxis.documentClass": ["ContractDocument"],
    "doxis.version": ["1"],
    "doxis.isLatestVersion": ["true"],
    "doxis.createdDate": ["2026-03-15T11:20:00.000Z"],
    "doxis.createdBy": ["johann.schmidt"],
    "doxis.modifiedDate": ["2026-09-28T14:15:30.000Z"],
    "doxis.modifiedBy": ["elena.weber"],
    "doxis.parentFolderId": ["9e8d7c6b-5a49-4382-b1a0-f9e8d7c6b5a4"],
    "doxis.parentFolderName": ["Acme Corp Master File"],
    "doxis.fileName": ["CON-2026-0042.pdf"],
    "doxis.fileSize": ["2097152"]
  },
  "security": {
    "inheritanceEnabled": true,
    "permissions": [
      { "identity": "DX4/Legal", "identityType": "group", "access": "read" },
      { "identity": "DX4/elena.weber", "identityType": "user", "access": "write" },
      { "identity": "DX4/restricted.user", "identityType": "user", "access": "deny" }
    ]
  }
}

2. Deletion Tombstone (action: "DELETE")

{
  "id": "doxis://DX4/DB1/documents/3f2a9c1e-5b7d-4e8a-9c0f-1a2b3c4d5e6f",
  "action": "DELETE",
  "metadata": {
    "doxis.documentId": ["3f2a9c1e-5b7d-4e8a-9c0f-1a2b3c4d5e6f"],
    "doxis.status": ["DELETED"],
    "doxis.customer": ["DX4"],
    "doxis.repository": ["DB1"]
  }
}

🖥️ Admin UI & CLI Integration

Admin UI (oc-admin-ui)

In the Admin UI:

  1. Navigate to Connectors $\rightarrow$ New Connector.
  2. Select Doxis Repository (Intelligent Content Automation) (org.opencrawling.doxis.DoxisRepositoryConnector).
  3. Fill in the CSB URL, Customer, Repository, and Auth parameters.
  4. Click Test Connection to execute an end-to-end diagnostic session ping.

OpenCrawling CLI (oc-cli)

Verify connectivity from the terminal:

oc connector check --name Doxis_Repository --type repository

🧪 Testing & Verification

  1. Unit & MockWebServer Tests:

    mvn test -pl oc-doxis-client,oc-doxis-repository-connector -am

    Runs mock HTTP test suites covering authentication, search paging, e-file traversal, ACL mapping, descriptor normalization, and incremental tombstone handling.

  2. Connector Test Script:

    ./scripts/test-doxis-repository-connector.sh

    Probes a Doxis CSB endpoint (unauthenticated GET /customers) if reachable, and executes the Maven test suite.

  3. Read-Only Live Probe Test:

    export DOXIS_LIVE_PROBE=1
export DOXIS_URL="https://csb.example.com/restws/publicws/rest/api/v1"
export DOXIS_CUSTOMER="DX4"
export DOXIS_USER="crawler"
export DOXIS_PASSWORD="password"
export DOXIS_ROLE="admins"
export DOXIS_REPOSITORY="DB1"
mvn test -pl oc-doxis-repository-connector -Dtest=DoxisLiveProbeTest

🗺️ Documentation

🧪 Testing

🤝 Community

Clone this wiki locally