Repository navigation
Hyland Content Innovation Cloud CIC Connector
Issue / Pull Request: #129 / #255
Module: oc-hyland-cic-connector (bundled in oc-runtime, oc-crawler, and multi-service Docker images)
Repository Connector Class: org.opencrawling.hyland.cic.repository.HylandCicRepositoryConnector (Hyland_CIC_Repository)
Output Connector Class: org.opencrawling.hyland.cic.output.HylandCicOutputConnector (Hyland_CIC_Output)
Client Class: org.opencrawling.hyland.cic.client.HylandCicClient
Security Mapper: org.opencrawling.hyland.cic.security.HylandCicSecurityMapper
Properties Class: org.opencrawling.hyland.cic.config.HylandCicProperties
Spring Bean: @Component (activated when spring.opencrawling.connector.type=hyland-cic for repository, or spring.opencrawling.output.type=hyland-cic for output)
The Hyland Content Innovation Cloud (CIC) Platform Connector (oc-hyland-cic-connector) delivers cloud-native, bi-directional integration between OpenCrawling and the Hyland Experience Platform (HXP) content microservices. It connects enterprise repositories hosted on HXP to downstream vector databases, search engines, and modern AI/RAG architectures, while also offering a high-performance output connector for automated document archival, versioning, and migration directly back into HXP dossiers and folders.
The connector fully implements the Open Ingestion Standard (OIS) SPIs:
-
Pre-Crawl Discovery & Dry-Run Estimation: Implements
DiscoverableRepositoryConnectorto navigate folder trees, discover document hierarchies, and preview crawl volume and duration without downloading heavy binary streams. -
Single-Document Fetching & Scope Guarding: Implements
DocumentFetchingRepositoryConnectorto allow autonomous AI agents and BPM event-driven workflows to fetch individual documents by UUID with strict anti-probing boundary enforcement. -
Java 25 Virtual Threads Reactive Streaming: Non-blocking Project Reactor
Flux<RepositoryDocument>streams powered by lightweight virtual threads (StructuredTaskScope) with claim-check offloading. -
Zero-Trust Access Control (ACL) Synchronization: Translates HXP Access Control Entries (ACEs) into canonical OIS
SecurityConfigrules with tenant-aware principal qualification and explicit deny precedence. -
Bi-Directional Archival & Tombstone Lifecycle: High-throughput multipart document uploads, configurable conflict resolution (
create-version,overwrite,skip), and automated deletion handling (DocumentAction.DELETEcallingDELETE /api/content/documents/{id}).
Important
Enterprise Notice: This connector interacts with Hyland Content Innovation Cloud (HXP) REST services over TLS 1.3. OpenCrawling does not bundle proprietary Hyland software, runtime components, or tenant credentials. Connectivity requires an active Hyland Cloud subscription, valid tenant authorization, and OAuth2 credentials provisioned via the Hyland Identity Provider (IdP).
flowchart TD
subgraph HXP["Hyland Content Innovation Cloud (HXP Microservices)"]
IdP["Hyland IdP (OAuth2 Token Service /connect/token)"]
Folders["Folder Hierarchy API (/api/content/folders)"]
Docs["Document & Dossier API (/api/content/documents)"]
Query["Content Query Service (/api/content/query)"]
Download["Binary Streaming (/api/content/documents/{id}/content)"]
end
subgraph OC["OpenCrawling Runtime (Java 25 & Virtual Threads)"]
Client["HylandCicClient (Token Cache, Rate Limit Backoff, HTTP 429 Retry)"]
RepoConn["HylandCicRepositoryConnector (Discoverable & DocumentFetching)"]
OutConn["HylandCicOutputConnector (Multipart Upload & Tombstones)"]
SecMap["HylandCicSecurityMapper (Tenant Qualification & Deny Precedence)"]
ClaimCheck["Claim-Check Store (Redis / Apache Ozone / AWS S3)"]
end
subgraph Pipeline["Decoupled Kafka Pipeline"]
Kafka["Apache Kafka Topics (opencrawling-documents)"]
Ingest["oc-ingestion-consumer (Apache Tika 4 + Chunking)"]
Embed["oc-embedding-consumer (Ollama / OpenAI Embeddings)"]
Vector["Output Sinks (pgvector / Vespa / Solr / Qdrant / Milvus / OpenSearch / Elasticsearch)"]
end
IdP <-->|"OAuth2 client_credentials"| Client
Folders <-->|"GET /api/content/folders"| Client
Docs <-->|"GET / POST / DELETE"| Client
Query <-->|"POST /api/content/query"| Client
Download -->|"TLS 1.3 Streaming"| Client
Client --> RepoConn
RepoConn -->|"OIS RepositoryDocument"| ClaimCheck
RepoConn -->|"Publish Envelope"| Kafka
Kafka --> Ingest --> Embed --> Vector
OutConn <-->|"Multipart Upload & DELETE"| Client
-
Pre-Crawl Discovery (
explore): Recursively navigates HXP folder structures and document hierarchies up to configurable depth and node limits without downloading binary payloads. Returns rich metadata including item counts, sizes, mime types, and child counts. -
Dry-Run Estimation (
dryRun): Previews ingestion volume, calculates estimated storage and execution durations, and returns representative sample nodes for both folder hierarchy traversals and arbitrary content queries. -
Scope Targeting & Boundary Validation (
crawlStart): Validates starting folder paths or folder IDs against connector boundaries and live caller permissions, strictly enforcing anti-probing defense. -
Single-Document Targeted Fetching (
fetchDocument): Allows AI agents, MCP tools, and external BPM workflow engines to retrieve specific documents by UUID. Validates that the requested document is strictly within the connector's configured root folder scope, is not excluded by path filters, and satisfies caller visibility rules. -
Search Query Ingestion: In addition to folder traversal, the connector can crawl documents matching an HXP query expression (
queryproperty), bypassing folder boundaries while preserving metadata and ACLs. -
Virtual Threads Reactive Scanning: Emits non-blocking Project Reactor
Flux<RepositoryDocument>streams orchestrated across lightweight virtual threads with automatic HTTP 429 backoff and retries.
- Enterprise Archival & Replatforming: Ingests, versions, and archives crawled documents into designated HXP target folders or dossiers via multipart REST uploads.
-
Conflict Resolution Policies: Configurable via
conflictResolution:-
create-version(default): Appends a new version to an existing document with the same name. -
overwrite: Overwrites the current document content and metadata in place. -
skip: Skips upload if a document with identical naming already exists.
-
-
Tombstone & Lifecycle Actions: Deletes downstream records in HXP via
DELETE /api/content/documents/{id}whenDocumentAction.DELETEevents are processed.
Translates HXP Access Control Entries (ACEs) into Open Ingestion Standard (OIS) SecurityConfig rules:
-
Tenant Qualification: When
tenantIdis configured, principal identities are qualified as{tenantId}/{identity}(e.g.tenant-enterprise-01/jane.doe@company.com) to enforce multi-tenant isolation across shared vector indexes. -
Explicit Deny Precedence: ACEs with
granted=falseproduceaccess=denyrules, ranked highest (rank=4) during conflict resolution to ensure that an explicit deny always overrides an inherited allow. -
Permission Normalization: Maps HXP permissions (
READ,WRITE,MODIFY,DELETE,ADMIN,FULL_CONTROL) into canonical OIS access levels (read,write,delete,admin). - Inheritance Flag: Distinguishes direct document-level permissions from inherited folder-level ACEs.
All connector settings can be configured via application.yml, Spring Boot environment variables, or directly through the OpenCrawling Admin UI:
| Property (YAML / Map) | Spring Environment Variable | Default | Description |
|---|---|---|---|
base-url |
SPRING_OPENCRAWLING_CONNECTOR_HYLAND_CIC_BASE_URL |
https://api.content.hylandcloud.com |
Base REST API URL of Hyland Content Innovation Cloud |
idp-token-url |
SPRING_OPENCRAWLING_CONNECTOR_HYLAND_CIC_IDP_TOKEN_URL |
https://identity.hylandcloud.com/connect/token |
Hyland Identity Provider (IdP) OAuth2 token endpoint |
client-id |
SPRING_OPENCRAWLING_CONNECTOR_HYLAND_CIC_CLIENT_ID |
"" |
OAuth2 Client ID provisioned in Hyland IdP |
client-secret |
SPRING_OPENCRAWLING_CONNECTOR_HYLAND_CIC_CLIENT_SECRET |
"" |
OAuth2 Client Secret provisioned in Hyland IdP |
scopes |
SPRING_OPENCRAWLING_CONNECTOR_HYLAND_CIC_SCOPES |
hxp hxp.integrations hxp.content |
Space-delimited OAuth2 scopes requested during token exchange |
bearer-token |
SPRING_OPENCRAWLING_CONNECTOR_HYLAND_CIC_BEARER_TOKEN |
"" |
Optional pre-shared bearer token (bypasses IdP exchange) |
tenant-id |
SPRING_OPENCRAWLING_CONNECTOR_HYLAND_CIC_TENANT_ID |
"" |
HXP Tenant ID (sent in X-Tenant-Id header and used for ACL qualification) |
root-folder-path |
SPRING_OPENCRAWLING_CONNECTOR_HYLAND_CIC_ROOT_FOLDER_PATH |
/ |
Root folder path in HXP to crawl |
root-folder-id |
SPRING_OPENCRAWLING_CONNECTOR_HYLAND_CIC_ROOT_FOLDER_ID |
"" |
Root folder UUID (overrides root-folder-path if set) |
content-types |
SPRING_OPENCRAWLING_CONNECTOR_HYLAND_CIC_CONTENT_TYPES |
[] |
Comma-separated list restricting crawl to specific content types |
include-subfolders |
SPRING_OPENCRAWLING_CONNECTOR_HYLAND_CIC_INCLUDE_SUBFOLDERS |
true |
Whether to recursively descend into child folder hierarchies |
excluded-folder-paths |
SPRING_OPENCRAWLING_CONNECTOR_HYLAND_CIC_EXCLUDED_FOLDER_PATHS |
[] |
Comma-separated paths excluded from traversal |
query |
SPRING_OPENCRAWLING_CONNECTOR_HYLAND_CIC_QUERY |
"" |
Optional search query expression (overrides folder hierarchy traversal) |
batch-size |
SPRING_OPENCRAWLING_CONNECTOR_HYLAND_CIC_BATCH_SIZE |
100 |
Paged fetch batch size for folder item queries |
include-acls |
SPRING_OPENCRAWLING_CONNECTOR_HYLAND_CIC_INCLUDE_ACLS |
true |
Extract Zero-Trust ACLs and permission rules |
include-content-stream |
SPRING_OPENCRAWLING_CONNECTOR_HYLAND_CIC_INCLUDE_CONTENT_STREAM |
true |
Download and stream document binaries |
max-content-size-bytes |
SPRING_OPENCRAWLING_CONNECTOR_HYLAND_CIC_MAX_CONTENT_SIZE_BYTES |
52428800 (50MB) |
Max binary size permitted for download |
connection-timeout-seconds |
SPRING_OPENCRAWLING_CONNECTOR_HYLAND_CIC_CONNECTION_TIMEOUT_SECONDS |
30 |
HTTP connection and socket timeout |
max-retries |
SPRING_OPENCRAWLING_CONNECTOR_HYLAND_CIC_MAX_RETRIES |
5 |
Maximum retry attempts for HTTP 429 and 5xx responses |
target-folder-path |
SPRING_OPENCRAWLING_CONNECTOR_HYLAND_CIC_TARGET_FOLDER_PATH |
/Enterprise_Ingestion/Automated_Archive |
Output connector destination folder path |
target-folder-id |
SPRING_OPENCRAWLING_CONNECTOR_HYLAND_CIC_TARGET_FOLDER_ID |
"" |
Output connector destination folder UUID (overrides path) |
default-content-type |
SPRING_OPENCRAWLING_CONNECTOR_HYLAND_CIC_DEFAULT_CONTENT_TYPE |
ArchivedDocument |
Content type applied during multipart document upload |
conflict-resolution |
SPRING_OPENCRAWLING_CONNECTOR_HYLAND_CIC_CONFLICT_RESOLUTION |
create-version |
Conflict policy: create-version, overwrite, skip
|
Depending on your deployment topology, configure the environment variables as follows:
Export the Spring Boot relaxed environment variables directly in your bash or zsh shell before launching OpenCrawling:
# Core Endpoint & OAuth2 Authentication
export SPRING_OPENCRAWLING_CONNECTOR_HYLAND_CIC_BASE_URL="https://api.content.hylandcloud.com"
export SPRING_OPENCRAWLING_CONNECTOR_HYLAND_CIC_IDP_TOKEN_URL="https://identity.hylandcloud.com/connect/token"
export SPRING_OPENCRAWLING_CONNECTOR_HYLAND_CIC_CLIENT_ID="opencrawling-service-client"
export SPRING_OPENCRAWLING_CONNECTOR_HYLAND_CIC_CLIENT_SECRET="sec_0192837465abcdef"
export SPRING_OPENCRAWLING_CONNECTOR_HYLAND_CIC_TENANT_ID="tenant-enterprise-01"
# Crawl Scope & Behavior
export SPRING_OPENCRAWLING_CONNECTOR_HYLAND_CIC_ROOT_FOLDER_PATH="/Finance/Invoices/2026"
export SPRING_OPENCRAWLING_CONNECTOR_HYLAND_CIC_INCLUDE_SUBFOLDERS="true"
export SPRING_OPENCRAWLING_CONNECTOR_HYLAND_CIC_BATCH_SIZE="100"
# Output Connector Target
export SPRING_OPENCRAWLING_CONNECTOR_HYLAND_CIC_TARGET_FOLDER_PATH="/Archive/AI_Processed"
export SPRING_OPENCRAWLING_CONNECTOR_HYLAND_CIC_CONFLICT_RESOLUTION="create-version"The automated integration test script accepts short-form HYLAND_CIC_* environment variables. You can pass them inline to test against a live HXP tenant:
HYLAND_CIC_BASE_URL="https://content.my-tenant.hyland.cloud" \
HYLAND_CIC_TOKEN_URL="https://identity.my-tenant.hyland.cloud/oauth/token" \
HYLAND_CIC_CLIENT_ID="opencrawling-service-client" \
HYLAND_CIC_CLIENT_SECRET="sec_0192837465abcdef" \
HYLAND_CIC_TENANT_ID="tenant-enterprise-01" \
./scripts/test-hyland-cic-decoupled.shNote
If no live cloud credentials are provided, test-hyland-cic-decoupled.sh automatically falls back to its hermetic embedded MockWebServer test harness, verifying 100% of connector capabilities offline.
For multi-container deployments using docker-compose-decoupled-with-hyland-cic.yml:
# .env file
HYLAND_CIC_BASE_URL=https://content.my-tenant.hyland.cloud
HYLAND_CIC_TOKEN_URL=https://identity.my-tenant.hyland.cloud/oauth/token
HYLAND_CIC_CLIENT_ID=opencrawling-service-client
HYLAND_CIC_CLIENT_SECRET=sec_0192837465abcdef
HYLAND_CIC_TENANT_ID=tenant-enterprise-01Start the decoupled stack:
docker compose -f oc-hyland-cic-connector/docker/docker-compose-decoupled-with-hyland-cic.yml up -dIn production Kubernetes environments, store non-sensitive configuration in a ConfigMap and secrets in a Secret:
apiVersion: v1
kind: Secret
metadata:
name: hyland-cic-credentials
type: Opaque
stringData:
SPRING_OPENCRAWLING_CONNECTOR_HYLAND_CIC_CLIENT_SECRET: "sec_0192837465abcdef"
---
apiVersion: v1
kind: ConfigMap
metadata:
name: hyland-cic-config
data:
SPRING_OPENCRAWLING_CONNECTOR_HYLAND_CIC_BASE_URL: "https://content.my-tenant.hyland.cloud"
SPRING_OPENCRAWLING_CONNECTOR_HYLAND_CIC_IDP_TOKEN_URL: "https://identity.my-tenant.hyland.cloud/oauth/token"
SPRING_OPENCRAWLING_CONNECTOR_HYLAND_CIC_CLIENT_ID: "opencrawling-service-client"
SPRING_OPENCRAWLING_CONNECTOR_HYLAND_CIC_TENANT_ID: "tenant-enterprise-01"
SPRING_OPENCRAWLING_CONNECTOR_HYLAND_CIC_ROOT_FOLDER_PATH: "/Enterprise_Content"
SPRING_OPENCRAWLING_CONNECTOR_HYLAND_CIC_BATCH_SIZE: "100"Reference them in your OpenCrawling Pod or Deployment specification:
spec:
containers:
- name: opencrawling-runtime
image: ghcr.io/opencrawling/oc-runtime:latest
envFrom:
- configMapRef:
name: hyland-cic-config
- secretRef:
name: hyland-cic-credentialsConfigure defaults directly in application.yml using property placeholders with environment variable fallbacks:
spring:
opencrawling:
connector:
hyland-cic:
base-url: ${HYLAND_CIC_BASE_URL:https://api.content.hylandcloud.com}
idp-token-url: ${HYLAND_CIC_TOKEN_URL:https://identity.hylandcloud.com/connect/token}
client-id: ${HYLAND_CIC_CLIENT_ID:}
client-secret: ${HYLAND_CIC_CLIENT_SECRET:}
tenant-id: ${HYLAND_CIC_TENANT_ID:}
root-folder-path: ${HYLAND_CIC_ROOT_FOLDER_PATH:/}
include-subfolders: true
batch-size: 100
include-acls: true
include-content-stream: true
target-folder-path: /Enterprise_Ingestion/Automated_Archive
conflict-resolution: create-versionThe table below lists the native Hyland Content Innovation Cloud REST endpoints invoked by HylandCicClient:
| Operation | Method & Endpoint | Connector Purpose |
|---|---|---|
| Health Check | GET /api/content/health |
Connector connect() lifecycle and health readiness check |
| IdP Token Exchange | POST {idpTokenUrl} |
OAuth2 Client Credentials authentication with token caching |
| Folder by Path | GET /api/content/folders?path={path} |
Resolve starting crawl folders and target archival destinations |
| Folder by ID | GET /api/content/folders/{id} |
Inspect folder metadata, child counts, and inherited permissions |
| Folder Children | GET /api/content/folders/{id}/items?offset&limit |
Tree exploration, pre-crawl discovery, and hierarchical folder crawl |
| Content Query | POST /api/content/query |
Arbitrary search query crawling and dry-run volume estimation |
| Document Metadata | GET /api/content/documents/{id} |
Single-document fetching (fetchDocument) and metadata extraction |
| Download Content | GET /api/content/documents/{id}/content |
Binary payload streaming with size limits |
| Upload Document | POST /api/content/documents |
Output connector multipart document ingestion and versioning |
| Delete Document | DELETE /api/content/documents/{id} |
Downstream tombstone / deletion processing (DocumentAction.DELETE) |
The OpenCrawling Admin UI (oc-admin-ui) provides first-class configuration forms for both repository and output roles:
-
Repository Connector Configuration:
- Live API endpoint validation with test connectivity button (
Test HXP Connection). - Pre-crawl discovery tree visualization showing live folder hierarchies.
- Interactive Dry-Run estimator previewing document volume and estimated crawl durations.
- Granular toggles for subfolder recursion, ACL extraction, and binary streaming.
- Live API endpoint validation with test connectivity button (
-
Output Connector Configuration:
- Target folder path and folder UUID resolution.
- Content type specification (
ArchivedDocument). - Conflict resolution selection (
create-version,overwrite,skip). - Dedicated connectivity probe (
Test HXP Output Connectivity).
Execute the complete Maven test suite (36 tests) using the isolated MockWebServer fixtures:
mvn test -pl oc-core,oc-hyland-cic-connector -amThe test suite covers:
-
HylandCicPropertiesTest: Configuration parsing, environment binding, and defaults. -
HylandCicSecurityMapperTest: ACE translation, permission normalization, deny-rule precedence, and tenant qualification. -
HylandCicClientTest: MockWebServer-based tests for OAuth2 token exchange, token caching, expiration refresh, folder paging, query execution, content download, multipart upload, and automatic backoff on HTTP 429 rate limits. -
HylandCicRepositoryConnectorTest: Discovery tree exploration, dry run tallying, scope targeting, anti-probing validation, single document fetch, and virtual-thread reactive scan. -
HylandCicOutputConnectorTest: Upsert multipart uploads, deletion handling, and target folder resolution.
To run the full end-to-end multi-service integration test suite (validating Kafka, Redis, PostgreSQL/pgvector, Ollama, Apache Ozone, and the MCP Server):
chmod +x scripts/test-hyland-cic-decoupled.sh
./scripts/test-hyland-cic-decoupled.sh- Home
- User & Installation Guide
- Quick Start
- VM & Bare-Metal Distribution Installation
- Local Development & Fast Docker Builds
- User Guide
- Architecture
- Zero-Trust Security & ACL Management
- Security and Identity Management
- Document Authorization & OpenFGA
- Enterprise Multi-Agent Integration
- Zero-Trust Data Provider Agent & Active Ingestion
- Document Lifecycle Actions & Tombstones
- Apache Tika 4 Text Extraction
- Configuration Guide
- Native Job Scheduler
- Platform API (
/api/v1) - Platform API: Traceability
- Connectors Reference
- Alfresco Content Services (ACS) Connector
- Alfresco Process Services (APS) Connector
- Flowable Repository Connector
- Camunda Repository Connector
- OASIS CMIS Repository Connector
- Doxis Repository Connector
- M-Files Repository Connector
- SharePoint Online Repository Connector
- Google Drive Repository Connector
- Atlassian Confluence Repository Connector
- IBM FileNet P8 Repository Connector
- OpenText Documentum Repository Connector
- Zammad Repository Connector
- Hyland OpenArch Integration (Repository, Output & MCP)
- Hyland Content Innovation Cloud (CIC) Connector
- Relational Database (JDBC) Connector
- Web Repository Connector
- Apache StormCrawler Repository Connector
- gRPC Push Repository Connector
- Elasticsearch Output Connector
- Apache Solr Output Connector
- Luxir Output Connector
- Apache SeaTunnel Output Connector
- Apache Ozone Output Connector (Migration Mode)
- AWS S3 Output Connector (Migration Mode)
- Alfresco Content Services Output Connector (Migration Mode)
- Apache Iceberg Output Connector
- Doxis Output Connector
- Connector Execution Modes
- Model Context Protocol
- Apache Ozone Claim Check Store
- Lakehouse Audit Reporting & Ledger
- AI-Powered Observability (AIOps)
- Auto-Narrativization Copilot
- Maven Connector Archetypes
- Java Client SDK
- Python Client SDK
- OpenCrawling CLI
- OpenAPI Specification & Swagger UI
- gRPC Internal Transport
- Migration Guide (ManifoldCF)
- Integration Test — Decoupled Pipeline
- M-Files Integration Test
- SharePoint Online Integration Test
- Google Drive Integration Test
- Confluence Integration Test
- IBM FileNet Integration Test
- OpenText Documentum Integration Test
- Zammad Integration Test
- Hyland OpenArch Integration Test
- Hyland CIC Integration Test
- Web Crawler Test
- Elasticsearch Integration Test
- Local Docker Build Test
- Root CLI Runner Scripts Test
- Secure MCP Server Integration Test
- Narrativization Integration Test
- Reporting Integration Test
- AWS S3 Migration Integration Test
- Alfresco Migration Integration Test
- OpenFGA Authorization Integration Test
- CLI Decoupled Integration Test
- Java Client SDK Decoupled Test
- Apache Iceberg Decoupled Test
- Doxis Output Integration Test
- Platform API v1 Test Suite