Skip to content

Doxis Output Connector

Piergiorgio Lucidi edited this page Nov 17, 2026 · 2 revisions

Doxis Output Connector

Doxis Java Spring Boot OIS Compliance Claim Check

Module: oc-doxis-output-connector
Optional Content-Link Module: oc-doxis-blueline-content-link (Maven profile -Pdoxis-blueline)
Shared Client Module: oc-doxis-client (shared with the Doxis Repository Connector)
Connector Class: org.opencrawling.doxis.output.DoxisOutputConnector (OutputConnector SPI / Doxis_Output)
Decoupled Kafka Writer: org.opencrawling.doxis.output.messaging.DoxisStoreWriterConsumer (topic opencrawling-documents, group opencrawling-doxis-writer)
Configuration Properties: org.opencrawling.doxis.output.config.DoxisOutputProperties (spring.opencrawling.output.doxis.*)
UI Settings Adapter: org.opencrawling.doxis.output.config.DoxisConnectorSettings (doxis* keys)
Spring Activation: @ConditionalOnProperty(name = "spring.opencrawling.output.type", havingValue = "doxis")


📌 Overview

The Doxis Output Connector archives crawled OpenCrawling documents into a Doxis 4 (SER Group) DMS repository through the Doxis Content Service Bus (CSB) REST API (/restws/publicws/rest/api/v1, verified against CSB 14.4.1).

It writes the target document type, typed metadata descriptors, Zero-Trust OIS Access Control Lists (ACLs), electronic file (Akte) filings, document versions, and deletion tombstones. By strictly separating the metadata plane from the content plane, DoxisOutputConnector plans how each binary reaches Doxis solely from document metadata — enabling streamed uploads for normal files, reference-only archiving, or zero-copy external content links (CONTENT_LINK) for multi-terabyte assets.

Important

Licence & Session Policy: Archiving into Doxis requires a valid, unexpired Doxis 4 licence (SER Group) for the target CSB whose licence permits API / technical-user sessions. OpenCrawling includes neither Doxis nor any Doxis licence, nor does it bundle SER proprietary client JARs. CSB counts technical-user sessions against your licence:

  • A direct UI/CLI/API job creates a per-job connector (ConnectorFactory) holding one CSB session for the run and automatically logs out (POST /logout) when the job finishes via ConnectorFactory.release().
  • A Test Connection check (ConnectorCheckerService) opens a short-lived session and closes it immediately.
  • Plan concurrent job session capacity with your Doxis administrator.

📋 Prerequisites

Verify all items below before configuring DoxisOutputConnector or starting an archiving job:

  1. Valid Doxis 4 Licence & CSB REST API:

    • Doxis 4 CSB 14.4 or later (verified on 14.4.1) with restws enabled and reachable at <scheme>://<csb-host>:<port>/restws/publicws/rest/api/v1.
    • Known customer (tenant) short name (listable via GET /customers).
    • API / technical-user session capacity permitted by your SER licence.
    • For zero-copy CONTENT_LINK: separately licensed SER Doxis Java (Blueline) client JARs (compiled for Java 17 bytecode, e.g. 14.4.1-1), loaded externally via content-link.client-lib-dir.
  2. Technical User & Role Permissions (e.g. role: admins when the user's default role lacks write privileges):

    Operation Required Doxis Right
    API Sign-in (POST /login) Login allowed; "change password at next login" must be disabled (otherwise API login is blocked)
    Document creation & versioning createDocument on target document classes
    Per-document ACLs (security.mode: DOCUMENT) Instance rights enabled on target document classes
    E-file filing (PRIMARY_PARENT) Document - Change primary parent object (SET_PRIMARY_PARENT) on target document classes (otherwise SECU0015 … setPrimaryParent permission)
    E-file removal / deletion All instances - Read and All instances - Write (high-volume write right) on the e-file class (otherwise SECU0015 … high volume write permission)
    Creating folder nodes (RELATIONSHIP filing) CREATE_FOLDER on the e-file (granted automatically to the connector user when security.grant-connector-user=true)
    Resolving OIS identities to principals Read access to /users and /groups
  3. Repository Configuration (cubeDesigner) (cannot be modified via CSB REST API):

    • Target DMS repository (repository) exists and allows the configured document-type (otherwise creation fails with INSTANCE0014).
    • Target document type's allowedMimeTypes includes the mime types being archived (or application/octet-stream as fallback, otherwise SEDNA0204).
    • E-file filing requires a content repository for primary parent objects configured under DMS -> Databases (otherwise INSTANCE0207).
    • For CONTENT_LINK on large/external assets, configure a dedicated document class in Doxis Designer with Allow linking contents = True, content indexing off, Creation of representations disabled (no Rendition Server conversion rules), thumbnails/ML off, and File from file system -> File deletion -> Delete files = False.
  4. OpenCrawling Runtime:

    • Java 25 runtime (oc-runtime, or decoupled oc-writer-consumer).
    • Supply password via Admin UI secrets, environment variables (SPRING_OPENCRAWLING_OUTPUT_DOXIS_PASSWORD), or an external secret store — never commit secrets to source control.

🏗️ Architecture & Pipeline Execution

Unlike vector databases that consume embedded text chunks from opencrawling-embedded, DoxisOutputConnector archives source binaries and document metadata directly:

flowchart TD
    subgraph Sources["OpenCrawling Sources & Pipeline"]
        Repo["Source Repository Connector"]
        Orch["JobOrchestrator (Direct / UI Job Path)"]
        KafkaDocs["Kafka Topic: opencrawling-documents"]
        ClaimCheck["ClaimCheckStore (Lazy Read Only on UPLOAD)"]
        Ingest["IngestionConsumer (Skips Tika Above max-extract-bytes)"]
    end

    subgraph OutputCore["oc-doxis-output-connector"]
        Factory["ConnectorFactory / Release on Job End"]
        Conn["DoxisOutputConnector"]
        Writer["DoxisStoreWriterConsumer (Group: opencrawling-doxis-writer)"]
        Planner["ContentPlanner + PrefixLocatorResolver + ContentLinkResolver"]
        Mapper["DoxisDocumentMapper + DoxisAclMapper"]
        Filer["DoxisRecordFiler (E-Files & Folder Nodes)"]
        Client["DoxisClient (oc-doxis-client REST + CQL)"]
    end

    subgraph Blueline["Optional: -Pdoxis-blueline"]
        Loader["ContentLinkWriterLoader (Isolated Child-First ClassLoader)"]
        SOAP["BluelineContentLinkWriter (SER Java API via SOAP)"]
    end

    subgraph Doxis4["SER Doxis 4 CSB (14.4+)"]
        REST["CSB REST API (/restws/publicws/rest/api/v1)"]
        Transfer["CSB SOAP (/sedna-transfer-service-xf/services/)"]
        Store["DMS Repository (Documents, Versions, E-Files, ACLs)"]
    end

    Repo -->|Direct Mode| Orch
    Orch --> Conn
    Factory -->|Builds Per-Job Instance & Logs Out| Conn
    Repo -->|Decoupled Mode| KafkaDocs
    KafkaDocs -->|Parallel Consumer Group| Writer
    KafkaDocs -->|RAG Extraction When Enabled| Ingest
    Writer -->|Lazy Stream Only If UPLOAD| ClaimCheck
    Writer --> Conn
    Conn --> Planner
    Conn --> Mapper
    Conn --> Filer
    Conn -->|REST Metadata, Upload, ACLs, CQL, Verify, Delete| Client
    Client --> REST
    Planner -->|CONTENT_LINK| Loader
    Loader --> SOAP
    SOAP --> Transfer
    REST --> Store
    Transfer --> Store
Loading

Dual Execution Paths

  1. Per-Job / Direct Path (JobController -> ConnectorFactory):
    • Selecting Doxis_Output (org.opencrawling.doxis.output.DoxisOutputConnector) in the Admin UI, CLI, or REST API resolves DoxisOutputProperties via DoxisConnectorSettings.fromConfiguration(config) (OutputKind.DOXIS).
    • JobController runs the job on a Java 25 Virtual Thread and calls connectorFactory.release(finalOutputConnector) in a finally block so the per-job CSB session logs out cleanly when the job finishes (leaving any shared runtime bean untouched).
  2. Decoupled Kafka Path (DoxisStoreWriterConsumer):
    • Activated when spring.opencrawling.output.type=doxis and opencrawling.consumer.writer.enabled=true.
    • DoxisConnectionInitializer validates login, repository, document type, and external-id attribute asynchronously at startup (failures warn and retry lazily on the first document).
    • DoxisStoreWriterConsumer consumes IngestionMessage events directly from opencrawling-documents (not opencrawling-embedded) in its own isolated consumer group (spring.opencrawling.output.doxis.consumer-group, default opencrawling-doxis-writer), running in parallel with text extraction. Any archiving failure is rethrown (IllegalStateException, #333 / PR #336) so the Kafka container's DefaultErrorHandler retries the record with exponential back-off and routes exhausted failures to opencrawling-documents.DLT (retaining its claim-check payload via DeadLetteredClaims, #332).
    • Binaries are wrapped in LazyInputStream: the ClaimCheckStore object is never opened or downloaded unless the document's ContentPlan selects UPLOAD.

Large Binary Bypass (opencrawling.ingestion.max-extract-bytes)

To archive multi-gigabyte or multi-terabyte files without exhausting memory during Tika text extraction:

  • IngestionConsumer enforces opencrawling.ingestion.max-extract-bytes (default 536870912 bytes / 512 MiB).
  • Files exceeding this threshold skip Apache Tika extraction and embedding completely, while IngestionMessage metadata still flows to DoxisStoreWriterConsumer on opencrawling-documents so Doxis archives the metadata, descriptors, ACLs, and content links safely.

🎯 Content Strategies (ContentPlanner)

ContentPlanner chooses how each binary reaches Doxis by inspecting document metadata only (sizeInBytes/contentLength, sha256/hashValue, file: stat, URI prefixes) without reading content streams upfront:

Strategy Behavior Use Case & Notes
AUTO (Default) Chooses per document: (1) PREDEFINED_LOCATOR if locator resolves (experimental); (2) UPLOAD if size is known and <= content.upload-max-bytes (or unknown size when fallback is UPLOAD); (3) CONTENT_LINK if link resolves and a ContentLinkWriter is loaded; (4) content.fallback (REFERENCE_ONLY by default). Best general-purpose default combining direct uploads for normal files with zero-copy links or references for oversized files.
UPLOAD Streams binary as multipart inputStream alongside documentParams (POST …/documents) or documentVersionParams (POST …/versions). Fails if known length exceeds content.upload-max-bytes (default 2147483648 / 2 GiB). Standard small-to-medium DMS archiving when binaries should be copied into Doxis storage.
CONTENT_LINK Zero-copy external UNC path or URL registered via oc-doxis-blueline-content-link (IDocument.addContentLinkPartDocument). Supported zero-copy mechanism (verified with a 5 TB UNC test on CSB 14.4.1). Multi-terabyte media/archives staying in place on NAS/file shares without copying bytes into Doxis. Requires content-link.client-lib-dir.
REFERENCE_ONLY Creates the document metadata record without binary content; optionally records source URI in reference-attribute. Lightweight metadata cataloging pointing back to source systems.
PREDEFINED_LOCATOR Experimental. Sends predefinedLocator, contentLength, and optional SHA-256 without inputStream. Off by default in Doxis storage adapters and unsupported by SER as a zero-copy path in CSB 14.4.1 (see Known Limitations). Experimental staging only where custom Doxis storage adapters explicitly enable predefined locators.

Read-Back Verification & Automatic Rollback

When content.verify=true (default):

  • After creating a new document or adding a version (UPLOAD, PREDEFINED_LOCATOR, or CONTENT_LINK), the connector calls GET …/dmsRepositories/{repo}/documents/{uuid}/versions?initializeRepresentations=true to inspect the default representation's content object.
  • Checks that a content object exists (catching silent NO_CONTENT writes), matches the expected byte length (for uploads/locators), compares fullFilename for UNC CONTENT_LINKs, and validates SHA-256 whenever CSB reports a non-blank hashValue.
  • If filing, ACL assignment, or verification fails on a newly created document, DoxisOutputConnector automatically rolls back the write by physically deleting the partial document (DELETE …/documents/{uuid}) so orphaned, empty, or unprotected records never remain in Doxis.

🔄 Upsert, Metadata Mapping & E-File (Akte) Filing

1. External ID Lookup & Conflict Resolution

  • External ID: document.id() is stored in external-id-attribute (default ObjectNumber) and searched before every write via CQL: SELECT * FROM <repoShortName> WHERE <field> = '<value>'.
  • Hash Truncation Protection: IDs exceeding the Doxis descriptor's length are hashed deterministically to h:<sha256Prefix> so lookups stay reliable.
  • conflict-resolution on existing documents:
    • NEW_VERSION (Default): Appends a new document version via POST …/documents/{uuid}/versions (note: for CONTENT_LINK, re-crawls refresh descriptors and keep the existing link instead of adding versions).
    • UPDATE_METADATA: Patches descriptors on the current version (updateAttributes) and syncs permissions without creating a new version or transferring content.
    • SKIP: Leaves existing documents untouched.
  • Unchanged Detection (content.change-marker-attribute): When configured (e.g. SystemId1), stores <lastModified>|<length> per version. On NEW_VERSION re-crawls, matching markers skip version creation cleanly.

2. Typed Descriptor Mapping (DoxisDocumentMapper)

  • title-attribute (default ObjectName) receives OIS title, name, file name, or document ID.
  • Optional reference-attribute stores document.uri().
  • attribute-mapping (<OIS key>: <Doxis attribute>, or key=Attribute in UI/JSON) converts values automatically using CSB schema metadata (DoxisSchema):
    • STRING: strips NUL bytes and truncates to descriptor length.
    • INTEGER, LONGINTEGER, FLOATINGPOINT, BOOL (true/1/y).
    • DATE / DATETIME: accepts epoch milliseconds, ISO-8601 instants/offsets, or YYYY-MM-DD dates.
    • Multi-value descriptors preserve all list values; single-value descriptors take the first element.
    • Pseudo-keys id, uri, and lastModified map top-level RepositoryDocument fields directly.

3. E-File (Akte) Filing (DoxisRecordFiler)

filing.mode controls where newly created documents are filed:

filing.mode Behavior
METADATA (Default) Files into record UUID from metadata key filing.record-id-metadata-key (default doxisRecordId), falling back to filing.record-id.
NONE Do not file documents into any e-file.
FIXED Always file into filing.record-id.
SOURCE_FOLDER One e-file per parent folder extracted from document.uri(); looked up in filing.record-class by filing.record-key-attribute and auto-created if filing.auto-create=true.
KEY_METADATA One e-file per value of metadata key filing.record-key-metadata-key (e.g. customer ID), looked up/auto-created like SOURCE_FOLDER.

filing.method controls how the document attaches to the e-file:

  • PRIMARY_PARENT (Default): Calls PUT …/documents/{uuid}/primaryParent after creation (works for uploads and CONTENT_LINK).
  • RELATIONSHIP: Sends relationshipParams during REST creation into filing.folder-node-id (doxisFolderNodeId metadata overrides) or the document-capable STATIC folder node named filing.folder-node-name (default Documents), auto-created under the e-file's NODES_ONLY root when missing. Applies to uploads only; CONTENT_LINK falls back to PRIMARY_PARENT.
  • Stale cached e-file/node IDs are evicted automatically on HTTP 404 or Doxis INSTANCE* errors.

🛡️ Zero-Trust OIS Security Mapping & Deletion Tombstones

OIS-to-Doxis Permission Translation (DoxisAclMapper)

Identities in SecurityConfig are resolved via DoxisSchema to Doxis users (by name or login name) or groups; OIS public maps to Doxis group everybody.

OIS access Document Permission (DocumentAceParams) E-File (Akte) Permission
read GRANT VIEW_DOCUMENT_CONTENTS GRANT VIEW_FOLDER_CONTENTS
write GRANT VIEW_DOCUMENT_CONTENTS, UPDATE_DOCUMENT, VERSION_DOCUMENT GRANT VIEW_FOLDER_CONTENTS, UPDATE_FOLDER, EDIT_FOLDER_DESCRIPTORS
deny DENY VIEW_DOCUMENT_CONTENTS DENY VIEW_FOLDER_CONTENTS

Security Controls (spring.opencrawling.output.doxis.security.*):

  • mode: DOCUMENT (default, per-document ACEs), RECORD (e-file ACEs inherited by documents when class enables "Primary parent objects -> Pass down permissions"), DOCUMENT_AND_RECORD, or NONE (class ACL only; also forced if apply-security-acls=false).
  • grant-connector-user (default true): Grants the connector's own technical user VIEW_FOLDER_CONTENTS, UPDATE_FOLDER, EDIT_FOLDER_DESCRIPTORS, DELETE_FOLDER, CREATE_FOLDER, SET_PRIMARY_PARENT, and VIEW_REMOVED_FOLDER on e-files it creates so instance ACLs do not lock the connector out of managing its own e-files.
  • record-acl-sync: CREATE_ONLY (default) or ADDITIVE (adds missing ACEs when later or re-crawled documents enter an e-file; never removes existing ACEs).
  • remove-stale (default false): On document re-crawl, removes connector-managed document ACEs (VIEW_DOCUMENT_CONTENTS, UPDATE_DOCUMENT, VERSION_DOCUMENT) no longer present at source while leaving Doxis administrator-managed permissions untouched.
  • strict (default false): Fails and rolls back a newly created document if its document class rejects per-document permissions (SECU0050) or if any OIS identity cannot be resolved in Doxis; when false, logs warnings instead.

OIS Deletion Tombstones (DocumentAction.DELETE)

Lookups match archived documents by external-id-attribute and apply delete-mode:

  • LOGICAL (Default): Executes POST …/remove — reversible removal that preserves binaries on disk (recommended when using in-place / content-link storage or shared content repositories).
  • PHYSICAL: Executes DELETE …/documents/{uuid} — irrevocable deletion. (Verified on CSB 14.4.1: physical delete of a CONTENT_LINK document removes only the Doxis metadata record and leaves the external target file untouched.)

🔗 Zero-Copy Content Links (oc-doxis-blueline-content-link)

Because Doxis CSB REST API 14.4.x has no endpoint to attach external content links, OpenCrawling provides an optional companion module oc-doxis-blueline-content-link implementing ContentLinkWriter via SER's Doxis Java (Blueline) API (IDocument.addContentLinkPartDocument()).

  • Excluded from default build: SER client JARs are proprietary and not redistributed with OpenCrawling.
  • Isolated ClassLoading: ContentLinkWriterLoader loads oc-doxis-blueline-content-link and the SER client JARs from content-link.client-lib-dir in a child-first isolated class loader sharing only the writer interface, SLF4J, and JDK classes, preventing Spring/CXF/Jackson version conflicts with OpenCrawling.
  • SOAP Endpoint: Speaks to http://<csb-host>:<port>/sedna-transfer-service-xf/services/ (defaults to host and port from base-url; override via content-link.csb-host / content-link.csb-port).

Build & Assemble Blueline Client Directory

# 1. Install SER client jar from your Doxis installation (e.g. webCube WEB-INF/lib)
mvn install:install-file \
  -Dfile=/path/to/WEB-INF/lib/blueline-14.4.1-1.jar \
  -DgroupId=de.ser.doxis4.java-api \
  -DartifactId=blueline \
  -Dversion=14.4.1-1 \
  -Dpackaging=jar \
  -DgeneratePom=true

# 2. Build the optional module (Java 25; SER jars are Java 17 bytecode 14.4.1-1)
mvn -Pdoxis-blueline install -pl oc-doxis-blueline-content-link

# 3. Stage all SER client jars + OpenCrawling Blueline adapter into client-lib-dir
mkdir -p /opt/doxis-client-lib
cp /path/to/WEB-INF/lib/*.jar /opt/doxis-client-lib/
cp oc-doxis-blueline-content-link/target/oc-doxis-blueline-content-link-*.jar /opt/doxis-client-lib/

Required SER Maven coordinates referenced by the build/runtime: de.ser.doxis4.java-api:blueline, de.ser.doxis4.java-api:sednaclient-blueline, and de.ser.doxis4.csb.client:sednaclient-api-nodep.

Note

CSB Serving & Re-crawl Semantics: CSB does not stream CONTENT_LINK binaries over REST GET …/contentObjects/{id} (SEDNA0104); opening clients resolve UNC/URL paths directly. Re-crawling a CONTENT_LINK document updates descriptors and syncs ACLs while keeping the original link (no extra versions are created).


⚙️ Configuration Parameters

All Spring Boot properties bind under spring.opencrawling.output.doxis.*, and per-connector Admin UI / connectors.json entries map via DoxisConnectorSettings:

Spring Property (spring.opencrawling.output.doxis.*) UI / JSON Key (doxis*) Default Description
base-url doxisBaseUrl http://localhost:8080/restws/publicws/rest/api/v1 CSB REST API base URL
customer-name doxisCustomerName — Required. Doxis customer (tenant) short name
username doxisUsername — Required. Technical archiving user
password doxisPassword — Required. Technical user password
role doxisRole — Optional Doxis role for login (e.g. admins)
repository doxisRepository — Required. Target DMS repository name or UUID
document-type doxisDocumentType BaseDocument Document class name or UUID allowed in repository
external-id-attribute doxisExternalIdAttribute ObjectNumber Descriptor holding OIS ID for CQL upsert/delete lookup
title-attribute doxisTitleAttribute ObjectName Descriptor receiving document title
reference-attribute doxisReferenceAttribute — Optional descriptor storing source URI
attribute-mapping.<key> doxisAttributeMapping {} Map <OIS key> (id, uri, lastModified, or metadata) to Doxis descriptor
conflict-resolution doxisConflictResolution NEW_VERSION Duplicate handling: NEW_VERSION, UPDATE_METADATA, SKIP
delete-mode doxisDeleteMode LOGICAL Tombstone deletion: LOGICAL (POST …/remove) or PHYSICAL (DELETE …)
apply-security-acls doxisApplySecurityAcls true Enables OIS ACL mapping (setting false forces security.mode=NONE)
content.strategy doxisContentStrategy AUTO AUTO, UPLOAD, CONTENT_LINK, REFERENCE_ONLY, PREDEFINED_LOCATOR
content.upload-max-bytes doxisUploadMaxBytes 2147483648 Max binary size streamed for UPLOAD / AUTO (2 GiB)
content.fallback doxisContentFallback REFERENCE_ONLY AUTO fallback (REFERENCE_ONLY or UPLOAD)
content.verify doxisVerifyContent true Read-back verification of content object, length, and SHA-256
content.change-marker-attribute doxisChangeMarkerAttribute — Descriptor storing <lastModified>|<length> to skip unchanged versions
locator.metadata-key doxisLocatorMetadataKey doxisLocator Metadata key supplying explicit predefinedLocator
locator.uri-prefix doxisLocatorUriPrefix — Source URI prefix stripped to form locator
locator.prefix doxisLocatorPrefix "" Prefix prepended to derived locators
filing.mode doxisFilingMode METADATA (or FIXED if record-id set) NONE, FIXED, METADATA, SOURCE_FOLDER, KEY_METADATA
filing.method doxisFilingMethod PRIMARY_PARENT PRIMARY_PARENT or RELATIONSHIP
filing.record-id doxisFilingRecordId — Target e-file UUID for FIXED / fallback
filing.record-repository doxisFilingRecordRepository Target repository Repository hosting target e-files
filing.folder-node-id doxisFilingFolderNodeId — Explicit folder node UUID for RELATIONSHIP filing
filing.folder-node-name doxisFilingFolderNodeName Documents Auto-located/created STATIC folder node name for RELATIONSHIP
filing.record-id-metadata-key doxisFilingRecordIdMetadataKey doxisRecordId Metadata key specifying e-file UUID in METADATA mode
filing.folder-node-metadata-key doxisFilingFolderNodeMetadataKey doxisFolderNodeId Metadata key overriding folder-node-id per document
filing.record-key-metadata-key doxisFilingRecordKeyMetadataKey — Metadata key identifying e-file in KEY_METADATA mode
filing.record-class doxisFilingRecordClass — E-file class used when auto-creating records
filing.record-key-attribute doxisFilingRecordKeyAttribute ObjectNumberExternal Descriptor storing keyed e-file lookup key
filing.record-title-attribute doxisFilingRecordTitleAttribute ObjectName Descriptor receiving e-file display title
filing.auto-create doxisFilingAutoCreate true Auto-create missing e-files and STATIC folder nodes
security.mode doxisSecurityMode DOCUMENT DOCUMENT, RECORD, DOCUMENT_AND_RECORD, NONE
security.strict doxisSecurityStrict false Roll back new docs on SECU0050 or unresolvable identities
security.remove-stale doxisSecurityRemoveStale false Prune stale connector-managed document ACEs on re-crawl
security.record-acl-sync doxisSecurityRecordAclSync CREATE_ONLY E-file ACL maintenance: CREATE_ONLY or ADDITIVE
security.grant-connector-user doxisSecurityGrantConnectorUser true Grant connector user rights on e-files it creates
content-link.client-lib-dir doxisContentLinkClientLibDir — Directory with Blueline + SER JARs; enables CONTENT_LINK
content-link.csb-host doxisContentLinkCsbHost Host of base-url CSB SOAP host override for Blueline
content-link.csb-port doxisContentLinkCsbPort Port of base-url (0 -> 8080) CSB SOAP port override for Blueline
content-link.uri-prefix doxisContentLinkUriPrefix — URI prefix rewritten for CONTENT_LINK
content-link.link-prefix doxisContentLinkPrefix "" Replacement UNC or URL prefix (/ -> \ for UNC)
content-link.link-type doxisContentLinkType UNC External link format: UNC or URL
content-link.metadata-key doxisContentLinkMetadataKey doxisContentLink Metadata key supplying an explicit external link
content-link.document-type doxisContentLinkDocumentType Main document-type Dedicated Doxis class allowing content links
max-retries doxisMaxRetries 3 HTTP retry count on transport failures / 429 / 502–504
timeout-seconds doxisTimeoutSeconds 120 CSB HTTP client timeout in seconds
consumer-group — opencrawling-doxis-writer Kafka consumer group of DoxisStoreWriterConsumer

Tip

Metadata Extraction Bound: Configure opencrawling.ingestion.max-extract-bytes (default 536870912 / 512 MiB) when running oc-ingestion-consumer alongside DoxisStoreWriterConsumer so multi-gigabyte binaries bypass Tika parsing while still being archived in Doxis.


📄 Example application.yml

spring:
  opencrawling:
    output:
      type: doxis
      doxis:
        base-url: "http://csb.example.com:8080/restws/publicws/rest/api/v1"
        customer-name: faststarter
        username: crawler
        password: "${DOXIS_PASSWORD}"
        role: admins
        repository: D_TEXTER
        document-type: TX_MigratedDocument
        external-id-attribute: ObjectNumberExternal
        title-attribute: ObjectName
        reference-attribute: URL
        attribute-mapping:
          author: ObjectAuthors
          lastModified: ObjectDate
        content:
          strategy: AUTO
          upload-max-bytes: 2147483648
          verify: true
          change-marker-attribute: SystemId1
        content-link:
          client-lib-dir: /opt/doxis-client-lib
          uri-prefix: "file:///mnt/archive/"
          link-prefix: "\\\\fileserver\\archive\\"
          link-type: UNC
          document-type: TX_LargeExternalDocument
        filing:
          mode: SOURCE_FOLDER
          method: PRIMARY_PARENT
          record-class: TX_SourceFolder
          auto-create: true
        security:
          mode: RECORD
          record-acl-sync: ADDITIVE
          grant-connector-user: true

opencrawling:
  consumer:
    writer:
      enabled: true
  ingestion:
    max-extract-bytes: 536870912

🖥️ Admin UI & CLI Integration

  • Admin UI (oc-admin-ui): Registered as default output connector Doxis_Output — Doxis 4 ECM Archive (CSB REST API) (org.opencrawling.doxis.output.DoxisOutputConnector), exposing sections for connection credentials, descriptors, binary strategies, optional Blueline content links, e-file filing, and lifecycle/security controls.
  • Connection Check (oc-cli & UI Test Connection):
    oc connector check --name Doxis_Output --type output
    Logs in to CSB (/login), fetches /session/user and /dmsRepositories/{repository}, and immediately logs out (/logout).

🧪 Testing & Verification

1. Maven Unit & MockWebServer Suites

# Core output connector (77 tests against recorded CSB 14.4.1 fixtures & OkHttp MockWebServer)
mvn test -pl oc-doxis-client,oc-doxis-output-connector -am

# Optional Blueline content-link module (requires SER blueline jar installed locally)
mvn -Pdoxis-blueline test -pl oc-doxis-blueline-content-link

2. Live CSB Contract Smoke Test (scripts/test-doxis-connector.sh)

There is no open-source Doxis Docker container; use scripts/test-doxis-connector.sh against a reachable CSB instance. When CSB variables are unset, the script skips cleanly (exit 0):

DOXIS_BASE_URL=http://csb-host:8080/restws/publicws/rest/api/v1 \
DOXIS_CUSTOMER=faststarter DOXIS_USER=crawler DOXIS_PASSWORD=... \
DOXIS_ROLE=admins DOXIS_REPOSITORY=D_TEXTER \
DOXIS_EXTERNAL_ID_FIELD=OBJECTNUMBER2 \
DOXIS_WRITE_TEST=true \
DOXIS_DOCUMENT_TYPE_UUID=<type-uuid> \
DOXIS_EXTERNAL_ID_ATTRIBUTE_UUID=<attr-uuid> \
DOXIS_MIME_TYPE=text/plain \
./scripts/test-doxis-connector.sh

Verifies:

  1. CSB liveness (/sedna-transfer-service-xf/isAlive) and tenant presence (GET /customers).
  2. JWT /login, /session/user, repository inspection (shortName, informationObjectTypeIds), and CQL /documents/search.
  3. (Opt-in DOXIS_WRITE_TEST=true) Multipart document creation, read-back verification of stored byte length via /versions?initializeRepresentations=true, physical delete cleanup, and /logout.

⚠️ Known Limitations

Observations recorded against Doxis CSB 14.4.1 (2026-10):

  • PREDEFINED_LOCATOR is experimental and off by default: SER confirmed (2026-10-02) that predefinedLocator is not a supported zero-copy mechanism in 14.4.1; CSB either rejects predefinedLocator with content or silently stores NO_CONTENT (caught and rolled back by verify). Use oc-doxis-blueline-content-link (CONTENT_LINK) for supported zero-copy external binaries.
  • No REST content-link endpoint in CSB 14.4.x: CONTENT_LINK requires -Pdoxis-blueline plus licensed SER Java client JARs. Calls in BluelineContentLinkWriter are serialized per writer session.
  • SHA-256 read-back: CSB 14.4.1 does not return client-supplied hashValue on read-back; verify() validates exact byte length and validates SHA-256 only when CSB returns one.
  • RELATIONSHIP filing vs. PRIMARY_PARENT: Documents filed via RELATIONSHIP into a STATIC folder node do not set a primaryParent; content links always fall back to PRIMARY_PARENT.
  • Single-use HTTP upload retries: HTTP 429/502–504 exponential retries apply only to repeatable requests; non-reopenable binary upload streams are sent once.
  • Decoupled upload vs. claim-check cleanup: In decoupled Kafka mode, IngestionConsumer with claimcheck.cleanup-on-consume=true can delete claim-check payloads before DoxisStoreWriterConsumer uploads them. Use direct job execution, content links, or disable premature cleanup when archiving binaries from Kafka.
  • Effective non-admin inheritance check: Removing an already logically deleted e-file returns HTTP 500 in CSB 14.4.1; non-admin effective inheritance on e-file ACLs should be validated directly in your target Doxis security model.

🗺️ Documentation


🎓 Tutorials


🧪 Testing


🤝 Community

Clone this wiki locally