Skip to content

Hyland Content Innovation Cloud CIC Connector

Piergiorgio Lucidi edited this page Nov 9, 2026 · 2 revisions

Hyland Content Innovation Cloud (CIC) Connector

Hyland CIC Java Spring Boot OIS Compliance Claim Check

Issue / Pull Request: #129 / #255
Module: oc-hyland-cic-connector (bundled in oc-runtime, oc-crawler, and multi-service Docker images)
Repository Connector Class: org.opencrawling.hyland.cic.repository.HylandCicRepositoryConnector (Hyland_CIC_Repository)
Output Connector Class: org.opencrawling.hyland.cic.output.HylandCicOutputConnector (Hyland_CIC_Output)
Client Class: org.opencrawling.hyland.cic.client.HylandCicClient
Security Mapper: org.opencrawling.hyland.cic.security.HylandCicSecurityMapper
Properties Class: org.opencrawling.hyland.cic.config.HylandCicProperties
Spring Bean: @Component (activated when spring.opencrawling.connector.type=hyland-cic for repository, or spring.opencrawling.output.type=hyland-cic for output)


📌 Overview

The Hyland Content Innovation Cloud (CIC) Platform Connector (oc-hyland-cic-connector) delivers cloud-native, bi-directional integration between OpenCrawling and the Hyland Experience Platform (HXP) content microservices. It connects enterprise repositories hosted on HXP to downstream vector databases, search engines, and modern AI/RAG architectures, while also offering a high-performance output connector for automated document archival, versioning, and migration directly back into HXP dossiers and folders.

The connector fully implements the Open Ingestion Standard (OIS) SPIs:

  1. Pre-Crawl Discovery & Dry-Run Estimation: Implements DiscoverableRepositoryConnector to navigate folder trees, discover document hierarchies, and preview crawl volume and duration without downloading heavy binary streams.
  2. Single-Document Fetching & Scope Guarding: Implements DocumentFetchingRepositoryConnector to allow autonomous AI agents and BPM event-driven workflows to fetch individual documents by UUID with strict anti-probing boundary enforcement.
  3. Java 25 Virtual Threads Reactive Streaming: Non-blocking Project Reactor Flux<RepositoryDocument> streams powered by lightweight virtual threads (StructuredTaskScope) with claim-check offloading.
  4. Zero-Trust Access Control (ACL) Synchronization: Translates HXP Access Control Entries (ACEs) into canonical OIS SecurityConfig rules with tenant-aware principal qualification and explicit deny precedence.
  5. Bi-Directional Archival & Tombstone Lifecycle: High-throughput multipart document uploads, configurable conflict resolution (create-version, overwrite, skip), and automated deletion handling (DocumentAction.DELETE calling DELETE /api/content/documents/{id}).

Important

Enterprise Notice: This connector interacts with Hyland Content Innovation Cloud (HXP) REST services over TLS 1.3. OpenCrawling does not bundle proprietary Hyland software, runtime components, or tenant credentials. Connectivity requires an active Hyland Cloud subscription, valid tenant authorization, and OAuth2 credentials provisioned via the Hyland Identity Provider (IdP).


🏗️ Architecture & Flow

flowchart TD
    subgraph HXP["Hyland Content Innovation Cloud (HXP Microservices)"]
        IdP["Hyland IdP (OAuth2 Token Service /connect/token)"]
        Folders["Folder Hierarchy API (/api/content/folders)"]
        Docs["Document & Dossier API (/api/content/documents)"]
        Query["Content Query Service (/api/content/query)"]
        Download["Binary Streaming (/api/content/documents/{id}/content)"]
    end

    subgraph OC["OpenCrawling Runtime (Java 25 & Virtual Threads)"]
        Client["HylandCicClient (Token Cache, Rate Limit Backoff, HTTP 429 Retry)"]
        RepoConn["HylandCicRepositoryConnector (Discoverable & DocumentFetching)"]
        OutConn["HylandCicOutputConnector (Multipart Upload & Tombstones)"]
        SecMap["HylandCicSecurityMapper (Tenant Qualification & Deny Precedence)"]
        ClaimCheck["Claim-Check Store (Redis / Apache Ozone / AWS S3)"]
    end

    subgraph Pipeline["Decoupled Kafka Pipeline"]
        Kafka["Apache Kafka Topics (opencrawling-documents)"]
        Ingest["oc-ingestion-consumer (Apache Tika 4 + Chunking)"]
        Embed["oc-embedding-consumer (Ollama / OpenAI Embeddings)"]
        Vector["Output Sinks (pgvector / Vespa / Solr / Qdrant / Milvus / OpenSearch / Elasticsearch)"]
    end

    IdP <-->|"OAuth2 client_credentials"| Client
    Folders <-->|"GET /api/content/folders"| Client
    Docs <-->|"GET / POST / DELETE"| Client
    Query <-->|"POST /api/content/query"| Client
    Download -->|"TLS 1.3 Streaming"| Client

    Client --> RepoConn
    RepoConn -->|"OIS RepositoryDocument"| ClaimCheck
    RepoConn -->|"Publish Envelope"| Kafka
    Kafka --> Ingest --> Embed --> Vector
    OutConn <-->|"Multipart Upload & DELETE"| Client
Loading

⚙️ Core Capabilities

1. Repository Connector (HylandCicRepositoryConnector)

  • Pre-Crawl Discovery (explore): Recursively navigates HXP folder structures and document hierarchies up to configurable depth and node limits without downloading binary payloads. Returns rich metadata including item counts, sizes, mime types, and child counts.
  • Dry-Run Estimation (dryRun): Previews ingestion volume, calculates estimated storage and execution durations, and returns representative sample nodes for both folder hierarchy traversals and arbitrary content queries.
  • Scope Targeting & Boundary Validation (crawlStart): Validates starting folder paths or folder IDs against connector boundaries and live caller permissions, strictly enforcing anti-probing defense.
  • Single-Document Targeted Fetching (fetchDocument): Allows AI agents, MCP tools, and external BPM workflow engines to retrieve specific documents by UUID. Validates that the requested document is strictly within the connector's configured root folder scope, is not excluded by path filters, and satisfies caller visibility rules.
  • Search Query Ingestion: In addition to folder traversal, the connector can crawl documents matching an HXP query expression (query property), bypassing folder boundaries while preserving metadata and ACLs.
  • Virtual Threads Reactive Scanning: Emits non-blocking Project Reactor Flux<RepositoryDocument> streams orchestrated across lightweight virtual threads with automatic HTTP 429 backoff and retries.

2. Output Connector (HylandCicOutputConnector)

  • Enterprise Archival & Replatforming: Ingests, versions, and archives crawled documents into designated HXP target folders or dossiers via multipart REST uploads.
  • Conflict Resolution Policies: Configurable via conflictResolution:
    • create-version (default): Appends a new version to an existing document with the same name.
    • overwrite: Overwrites the current document content and metadata in place.
    • skip: Skips upload if a document with identical naming already exists.
  • Tombstone & Lifecycle Actions: Deletes downstream records in HXP via DELETE /api/content/documents/{id} when DocumentAction.DELETE events are processed.

3. Zero-Trust Security Mapper (HylandCicSecurityMapper)

Translates HXP Access Control Entries (ACEs) into Open Ingestion Standard (OIS) SecurityConfig rules:

  • Tenant Qualification: When tenantId is configured, principal identities are qualified as {tenantId}/{identity} (e.g. tenant-enterprise-01/jane.doe@company.com) to enforce multi-tenant isolation across shared vector indexes.
  • Explicit Deny Precedence: ACEs with granted=false produce access=deny rules, ranked highest (rank=4) during conflict resolution to ensure that an explicit deny always overrides an inherited allow.
  • Permission Normalization: Maps HXP permissions (READ, WRITE, MODIFY, DELETE, ADMIN, FULL_CONTROL) into canonical OIS access levels (read, write, delete, admin).
  • Inheritance Flag: Distinguishes direct document-level permissions from inherited folder-level ACEs.

📋 Configuration Properties Reference

All connector settings can be configured via application.yml, Spring Boot environment variables, or directly through the OpenCrawling Admin UI:

Property (YAML / Map) Spring Environment Variable Default Description
base-url SPRING_OPENCRAWLING_CONNECTOR_HYLAND_CIC_BASE_URL https://api.content.hylandcloud.com Base REST API URL of Hyland Content Innovation Cloud
idp-token-url SPRING_OPENCRAWLING_CONNECTOR_HYLAND_CIC_IDP_TOKEN_URL https://identity.hylandcloud.com/connect/token Hyland Identity Provider (IdP) OAuth2 token endpoint
client-id SPRING_OPENCRAWLING_CONNECTOR_HYLAND_CIC_CLIENT_ID "" OAuth2 Client ID provisioned in Hyland IdP
client-secret SPRING_OPENCRAWLING_CONNECTOR_HYLAND_CIC_CLIENT_SECRET "" OAuth2 Client Secret provisioned in Hyland IdP
scopes SPRING_OPENCRAWLING_CONNECTOR_HYLAND_CIC_SCOPES hxp hxp.integrations hxp.content Space-delimited OAuth2 scopes requested during token exchange
bearer-token SPRING_OPENCRAWLING_CONNECTOR_HYLAND_CIC_BEARER_TOKEN "" Optional pre-shared bearer token (bypasses IdP exchange)
tenant-id SPRING_OPENCRAWLING_CONNECTOR_HYLAND_CIC_TENANT_ID "" HXP Tenant ID (sent in X-Tenant-Id header and used for ACL qualification)
root-folder-path SPRING_OPENCRAWLING_CONNECTOR_HYLAND_CIC_ROOT_FOLDER_PATH / Root folder path in HXP to crawl
root-folder-id SPRING_OPENCRAWLING_CONNECTOR_HYLAND_CIC_ROOT_FOLDER_ID "" Root folder UUID (overrides root-folder-path if set)
content-types SPRING_OPENCRAWLING_CONNECTOR_HYLAND_CIC_CONTENT_TYPES [] Comma-separated list restricting crawl to specific content types
include-subfolders SPRING_OPENCRAWLING_CONNECTOR_HYLAND_CIC_INCLUDE_SUBFOLDERS true Whether to recursively descend into child folder hierarchies
excluded-folder-paths SPRING_OPENCRAWLING_CONNECTOR_HYLAND_CIC_EXCLUDED_FOLDER_PATHS [] Comma-separated paths excluded from traversal
query SPRING_OPENCRAWLING_CONNECTOR_HYLAND_CIC_QUERY "" Optional search query expression (overrides folder hierarchy traversal)
batch-size SPRING_OPENCRAWLING_CONNECTOR_HYLAND_CIC_BATCH_SIZE 100 Paged fetch batch size for folder item queries
include-acls SPRING_OPENCRAWLING_CONNECTOR_HYLAND_CIC_INCLUDE_ACLS true Extract Zero-Trust ACLs and permission rules
include-content-stream SPRING_OPENCRAWLING_CONNECTOR_HYLAND_CIC_INCLUDE_CONTENT_STREAM true Download and stream document binaries
max-content-size-bytes SPRING_OPENCRAWLING_CONNECTOR_HYLAND_CIC_MAX_CONTENT_SIZE_BYTES 52428800 (50MB) Max binary size permitted for download
connection-timeout-seconds SPRING_OPENCRAWLING_CONNECTOR_HYLAND_CIC_CONNECTION_TIMEOUT_SECONDS 30 HTTP connection and socket timeout
max-retries SPRING_OPENCRAWLING_CONNECTOR_HYLAND_CIC_MAX_RETRIES 5 Maximum retry attempts for HTTP 429 and 5xx responses
target-folder-path SPRING_OPENCRAWLING_CONNECTOR_HYLAND_CIC_TARGET_FOLDER_PATH /Enterprise_Ingestion/Automated_Archive Output connector destination folder path
target-folder-id SPRING_OPENCRAWLING_CONNECTOR_HYLAND_CIC_TARGET_FOLDER_ID "" Output connector destination folder UUID (overrides path)
default-content-type SPRING_OPENCRAWLING_CONNECTOR_HYLAND_CIC_DEFAULT_CONTENT_TYPE ArchivedDocument Content type applied during multipart document upload
conflict-resolution SPRING_OPENCRAWLING_CONNECTOR_HYLAND_CIC_CONFLICT_RESOLUTION create-version Conflict policy: create-version, overwrite, skip

🚀 Setting Environment Variables

Depending on your deployment topology, configure the environment variables as follows:

1. Local Terminal / Shell Session

Export the Spring Boot relaxed environment variables directly in your bash or zsh shell before launching OpenCrawling:

# Core Endpoint & OAuth2 Authentication
export SPRING_OPENCRAWLING_CONNECTOR_HYLAND_CIC_BASE_URL="https://api.content.hylandcloud.com"
export SPRING_OPENCRAWLING_CONNECTOR_HYLAND_CIC_IDP_TOKEN_URL="https://identity.hylandcloud.com/connect/token"
export SPRING_OPENCRAWLING_CONNECTOR_HYLAND_CIC_CLIENT_ID="opencrawling-service-client"
export SPRING_OPENCRAWLING_CONNECTOR_HYLAND_CIC_CLIENT_SECRET="sec_0192837465abcdef"
export SPRING_OPENCRAWLING_CONNECTOR_HYLAND_CIC_TENANT_ID="tenant-enterprise-01"

# Crawl Scope & Behavior
export SPRING_OPENCRAWLING_CONNECTOR_HYLAND_CIC_ROOT_FOLDER_PATH="/Finance/Invoices/2026"
export SPRING_OPENCRAWLING_CONNECTOR_HYLAND_CIC_INCLUDE_SUBFOLDERS="true"
export SPRING_OPENCRAWLING_CONNECTOR_HYLAND_CIC_BATCH_SIZE="100"

# Output Connector Target
export SPRING_OPENCRAWLING_CONNECTOR_HYLAND_CIC_TARGET_FOLDER_PATH="/Archive/AI_Processed"
export SPRING_OPENCRAWLING_CONNECTOR_HYLAND_CIC_CONFLICT_RESOLUTION="create-version"

2. Decoupled Integration Test Script (test-hyland-cic-decoupled.sh)

The automated integration test script accepts short-form HYLAND_CIC_* environment variables. You can pass them inline to test against a live HXP tenant:

HYLAND_CIC_BASE_URL="https://content.my-tenant.hyland.cloud" \
HYLAND_CIC_TOKEN_URL="https://identity.my-tenant.hyland.cloud/oauth/token" \
HYLAND_CIC_CLIENT_ID="opencrawling-service-client" \
HYLAND_CIC_CLIENT_SECRET="sec_0192837465abcdef" \
HYLAND_CIC_TENANT_ID="tenant-enterprise-01" \
./scripts/test-hyland-cic-decoupled.sh

Note

If no live cloud credentials are provided, test-hyland-cic-decoupled.sh automatically falls back to its hermetic embedded MockWebServer test harness, verifying 100% of connector capabilities offline.

3. Docker Compose & .env File

For multi-container deployments using docker-compose-decoupled-with-hyland-cic.yml:

# .env file
HYLAND_CIC_BASE_URL=https://content.my-tenant.hyland.cloud
HYLAND_CIC_TOKEN_URL=https://identity.my-tenant.hyland.cloud/oauth/token
HYLAND_CIC_CLIENT_ID=opencrawling-service-client
HYLAND_CIC_CLIENT_SECRET=sec_0192837465abcdef
HYLAND_CIC_TENANT_ID=tenant-enterprise-01

Start the decoupled stack:

docker compose -f oc-hyland-cic-connector/docker/docker-compose-decoupled-with-hyland-cic.yml up -d

4. Kubernetes Deployment (ConfigMap & Secret)

In production Kubernetes environments, store non-sensitive configuration in a ConfigMap and secrets in a Secret:

apiVersion: v1
kind: Secret
metadata:
  name: hyland-cic-credentials
type: Opaque
stringData:
  SPRING_OPENCRAWLING_CONNECTOR_HYLAND_CIC_CLIENT_SECRET: "sec_0192837465abcdef"
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: hyland-cic-config
data:
  SPRING_OPENCRAWLING_CONNECTOR_HYLAND_CIC_BASE_URL: "https://content.my-tenant.hyland.cloud"
  SPRING_OPENCRAWLING_CONNECTOR_HYLAND_CIC_IDP_TOKEN_URL: "https://identity.my-tenant.hyland.cloud/oauth/token"
  SPRING_OPENCRAWLING_CONNECTOR_HYLAND_CIC_CLIENT_ID: "opencrawling-service-client"
  SPRING_OPENCRAWLING_CONNECTOR_HYLAND_CIC_TENANT_ID: "tenant-enterprise-01"
  SPRING_OPENCRAWLING_CONNECTOR_HYLAND_CIC_ROOT_FOLDER_PATH: "/Enterprise_Content"
  SPRING_OPENCRAWLING_CONNECTOR_HYLAND_CIC_BATCH_SIZE: "100"

Reference them in your OpenCrawling Pod or Deployment specification:

spec:
  containers:
    - name: opencrawling-runtime
      image: ghcr.io/opencrawling/oc-runtime:latest
      envFrom:
        - configMapRef:
            name: hyland-cic-config
        - secretRef:
            name: hyland-cic-credentials

5. application.yml Example

Configure defaults directly in application.yml using property placeholders with environment variable fallbacks:

spring:
  opencrawling:
    connector:
      hyland-cic:
        base-url: ${HYLAND_CIC_BASE_URL:https://api.content.hylandcloud.com}
        idp-token-url: ${HYLAND_CIC_TOKEN_URL:https://identity.hylandcloud.com/connect/token}
        client-id: ${HYLAND_CIC_CLIENT_ID:}
        client-secret: ${HYLAND_CIC_CLIENT_SECRET:}
        tenant-id: ${HYLAND_CIC_TENANT_ID:}
        root-folder-path: ${HYLAND_CIC_ROOT_FOLDER_PATH:/}
        include-subfolders: true
        batch-size: 100
        include-acls: true
        include-content-stream: true
        target-folder-path: /Enterprise_Ingestion/Automated_Archive
        conflict-resolution: create-version

🌐 HXP REST Endpoints Reference

The table below lists the native Hyland Content Innovation Cloud REST endpoints invoked by HylandCicClient:

Operation Method & Endpoint Connector Purpose
Health Check GET /api/content/health Connector connect() lifecycle and health readiness check
IdP Token Exchange POST {idpTokenUrl} OAuth2 Client Credentials authentication with token caching
Folder by Path GET /api/content/folders?path={path} Resolve starting crawl folders and target archival destinations
Folder by ID GET /api/content/folders/{id} Inspect folder metadata, child counts, and inherited permissions
Folder Children GET /api/content/folders/{id}/items?offset&limit Tree exploration, pre-crawl discovery, and hierarchical folder crawl
Content Query POST /api/content/query Arbitrary search query crawling and dry-run volume estimation
Document Metadata GET /api/content/documents/{id} Single-document fetching (fetchDocument) and metadata extraction
Download Content GET /api/content/documents/{id}/content Binary payload streaming with size limits
Upload Document POST /api/content/documents Output connector multipart document ingestion and versioning
Delete Document DELETE /api/content/documents/{id} Downstream tombstone / deletion processing (DocumentAction.DELETE)

🖥️ Admin UI Integration

The OpenCrawling Admin UI (oc-admin-ui) provides first-class configuration forms for both repository and output roles:

  • Repository Connector Configuration:

    • Live API endpoint validation with test connectivity button (Test HXP Connection).
    • Pre-crawl discovery tree visualization showing live folder hierarchies.
    • Interactive Dry-Run estimator previewing document volume and estimated crawl durations.
    • Granular toggles for subfolder recursion, ACL extraction, and binary streaming.
  • Output Connector Configuration:

    • Target folder path and folder UUID resolution.
    • Content type specification (ArchivedDocument).
    • Conflict resolution selection (create-version, overwrite, skip).
    • Dedicated connectivity probe (Test HXP Output Connectivity).

🧪 Testing & Verification

Run Automated Unit & Mock Tests (Offline)

Execute the complete Maven test suite (36 tests) using the isolated MockWebServer fixtures:

mvn test -pl oc-core,oc-hyland-cic-connector -am

The test suite covers:

  • HylandCicPropertiesTest: Configuration parsing, environment binding, and defaults.
  • HylandCicSecurityMapperTest: ACE translation, permission normalization, deny-rule precedence, and tenant qualification.
  • HylandCicClientTest: MockWebServer-based tests for OAuth2 token exchange, token caching, expiration refresh, folder paging, query execution, content download, multipart upload, and automatic backoff on HTTP 429 rate limits.
  • HylandCicRepositoryConnectorTest: Discovery tree exploration, dry run tallying, scope targeting, anti-probing validation, single document fetch, and virtual-thread reactive scan.
  • HylandCicOutputConnectorTest: Upsert multipart uploads, deletion handling, and target folder resolution.

Run Decoupled Integration Test Script

To run the full end-to-end multi-service integration test suite (validating Kafka, Redis, PostgreSQL/pgvector, Ollama, Apache Ozone, and the MCP Server):

chmod +x scripts/test-hyland-cic-decoupled.sh
./scripts/test-hyland-cic-decoupled.sh

🔗 Related Documentation

🗺️ Documentation


🎓 Tutorials


🧪 Testing


🤝 Community

Clone this wiki locally