Skip to content

Hyland OpenArch Integration

Jump to bottom
Piergiorgio Lucidi edited this page Oct 26, 2026 · 1 revision

Hyland OpenArch Integration

Hyland OpenArch Java Spring Boot Apache Parquet OIS Compliance MCP

Issue / Pull Request: #170 / #175
Modules: oc-openarch-repository-connector, oc-openarch-output-connector (both bundled in oc-runtime)
Repository Connector Class: org.opencrawling.openarch.repository.OpenArchRepositoryConnector (OpenArch_Repository)
Output Connector Class: org.opencrawling.openarch.output.OpenArchOutputConnector (OpenArch_Output)
Decoupled Writer: org.opencrawling.openarch.output.messaging.OpenArchStoreWriterConsumer
Parquet Vector Bridge: org.opencrawling.openarch.output.parquet.OpenArchParquetEmbeddingsWriter
Security Mapper: org.opencrawling.openarch.output.security.OpenArchSecurityMapper
MCP Backend: org.opencrawling.runtime.mcp.backend.OpenArchMcpSearchBackend

📌 Overview

OpenCrawling integrates with Hyland OpenArch in both directions. OpenArch is a cloud-native content and AI repository built on Spring Boot, MongoDB and OpenSearch. The integration has three parts:

  1. Repository connector (oc-openarch-repository-connector): Harvests documents, metadata, binaries and ACLs from OpenArch with paged HXQL queries (POST /api/query) and binary downloads (GET /api/download/{id}). It supports incremental filtering on sys_modified and fans out per-document work on Java 25 Virtual Threads (StructuredTaskScope).
  2. Output connector and Parquet vector bridge (oc-openarch-output-connector): Uploads binaries (POST /api/upload) and upserts SysFile documents (POST /api/documents/{parentId}) with mapped Zero-Trust ACLs. Embeddings are written as Apache Parquet files that follow OpenArch's Avro schema (embedding.avsc).
  3. Secure MCP backend (OpenArchMcpSearchBackend): Plugs OpenArch into McpVectorServer as a search backend. It adds the openarch_search and openarch_fetch_document MCP tools and serves the standard secureVectorSearch, getDocumentContent and listAccessibleSources tools.

The integration also adds Admin UI forms, catalogue defaults in ConnectorController, a connection check in ConnectorCheckerService, a new oc connector test alias, a decoupled Docker Compose stack and two integration test suites.

🏗️ Architecture & Flow

flowchart TD
    subgraph OA["Hyland OpenArch"]
        REST["REST API (/api/query, /api/query/embedding, /api/documents, /api/upload, /api/download)"]
        Mongo[("MongoDB (SysFile, SysFolder, sys_acl)")]
        OSearch[("OpenSearch (full-text and k-NN embeddings)")]
        REST --> Mongo
        REST --> OSearch
    end

    subgraph OC["OpenCrawling Runtime (Java 25)"]
        RepoConn["OpenArchRepositoryConnector (paged HXQL + Virtual Threads)"]
        OIS["OIS RepositoryDocument (UPSERT)"]
        Kafka["Kafka topics (raw documents / opencrawling-embedded)"]
        Ingest["oc-ingestion-consumer (Tika + chunking)"]
        Embed["oc-embedding-consumer (Ollama mxbai-embed-large)"]
        Writer["OpenArchStoreWriterConsumer (Parquet batching)"]
        OutConn["OpenArchOutputConnector (upload + upsert + Parquet)"]
        SecMap["OpenArchSecurityMapper (OIS to sys_acl)"]
        Parquet["OpenArchParquetEmbeddingsWriter (embedding.avsc, SNAPPY)"]
        Mcp["McpVectorServer + OpenArchMcpSearchBackend"]
    end

    Agent["AI Agent / LLM Client"]

    REST -->|"POST /api/query, GET /api/download/{id}"| RepoConn
    RepoConn --> OIS
    OIS --> Kafka
    Kafka --> Ingest
    Ingest --> Embed
    Embed -->|"opencrawling-embedded"| Writer
    Writer --> Parquet
    OutConn --> SecMap
    OutConn --> Parquet
    Parquet -->|"POST /api/upload (application/vnd.apache.parquet)"| REST
    OutConn -->|"POST /api/upload, POST /api/documents/{parentId}"| REST
    Agent <-->|"openarch_search / openarch_fetch_document"| Mcp
    Mcp -->|"POST /api/query/embedding, GET /api/documents/{id}"| REST
Loading

OpenArch REST endpoints used

All requests send Accept: application/json, the X-HXCS-Repository: <repositoryId> header and, when a username is configured, HTTP Basic authentication.

Endpoint Used by Purpose
POST /api/query Repository connector, MCP listAccessibleSources, connection check / ping Paged HXQL query (limit, offset, searchAfter)
POST /api/query/embedding MCP backend Vector + chunk full-text search over OpenArch embeddings
GET /api/documents/{id} MCP getDocumentContent / openarch_fetch_document Fetch document metadata
POST /api/documents/{parentId}?upsert={true|false} Output connector Create or update a SysFile document under the target folder
POST /api/upload?fileName=…&mimeType=… Output connector, decoupled writer Upload binaries and Parquet embedding files
GET /api/download/{id} Repository connector Stream the binary content of a document

📥 Repository Connector (OpenArchRepositoryConnector)

Key Features

  • Paged HXQL harvesting: Runs hxqlQuery (default SELECT * FROM SysFile) against POST /api/query in pages of batchSize documents. It follows the server's nextSearchAfter cursor if one is returned and falls back to offset paging otherwise. The crawl ends when a page is empty or smaller than batchSize.
  • Java 25 Structured Concurrency: scan() runs on a dedicated virtual thread (openarch-crawler-lane). Each page is mapped in parallel inside StructuredTaskScope.open(), and every task is wrapped in ObservabilityTask.observed(...) so OpenTelemetry trace context propagates. The connector reports supportsConcurrentProcessing() = true.
  • Binary streaming: When includeContentStream=true, the connector downloads each non-folder object (sys_primaryType != SysFolder) from GET /api/download/{id} and attaches it as the document content stream, which then follows the normal Claim Check flow. Failed downloads are logged at DEBUG and the document is still emitted with its metadata.
  • Incremental filtering: When incremental=true and lastCrawlTimestamp > 0 (epoch milliseconds), the connector appends sys_modified >= '<ISO-8601>' to the query, using AND if the query already has a WHERE clause and WHERE otherwise.
  • Zero-Trust ACL preservation: Maps each sys_acl entry to an OIS PermissionRule. See ACL mapping.
  • Schema introspection: getSchema() exposes the OpenArch system fields (sys_id, sys_name, sys_title, sys_description, sys_primaryType, sys_mixinTypes, sys_path, sys_created, sys_modified, sys_acl, sys_effectivePermissions) for the Auto-Narrativization Copilot.

Warning

Current limitations (as of PR #175)

  • crawlMode, rootFolderId and includeSubfolders are accepted by the configuration, the Admin UI and the connector catalogue, but scan() always runs the HXQL query path. Folder traversal (FOLDER mode) is not implemented yet. To limit a crawl to a folder, add a filter to hxqlQuery.
  • The scan(basePath) argument (SCAN_PATH) is ignored. The query comes only from hxqlQuery.
  • The connector emits only UPSERT documents. Deletions in OpenArch do not produce OIS DELETE tombstones.
  • If a document has no sys_acl array, or includeAcls=false, it is emitted with a public SecurityConfig.

Configuration Properties

Spring Property (spring.opencrawling.connector.openarch.*) connectors.json / Admin UI Key Default Description
endpoint-url endpointUrl http://localhost:8080 OpenArch REST API base URL
repository-id repositoryId default Repository identifier sent as X-HXCS-Repository
username username Administrator Basic authentication username
password password Administrator Basic authentication password
crawl-mode crawlMode QUERY QUERY or FOLDER (see limitations above)
hxql-query hxqlQuery SELECT * FROM SysFile HXQL query to harvest
root-folder-id rootFolderId 00000000-0000-0000-0000-000000000000 Root folder UUID for FOLDER mode (reserved)
include-subfolders includeSubfolders true Recurse into child folders in FOLDER mode (reserved)
batch-size batchSize 100 Page size for HXQL pagination
include-acls includeAcls true Map sys_acl to OIS permissions
include-content-stream includeContentStream true Download binaries through /api/download/{id}
incremental incremental false Add a sys_modified filter to the query
last-crawl-timestamp — 0 Epoch milliseconds used as the lower bound when incremental=true

Example application.yml 

spring:
  opencrawling:
    repository-connector-type: openarch
    crawl-on-startup: true
    connector:
      openarch:
        endpoint-url: "https://openarch.enterprise.local"
        repository-id: "default"
        username: "crawler_service"
        password: "${OPENARCH_PASSWORD}"
        hxql-query: "SELECT * FROM SysFile WHERE sys_title LIKE '%.pdf'"
        batch-size: 100
        include-acls: true
        include-content-stream: true
        incremental: true
        last-crawl-timestamp: 1767225600000   # 2026-01-01T00:00:00Z

With repository-connector-type: openarch, the runtime picks OpenArchRepositoryConnector because its class name contains openarch. If scan-path is empty, crawl-on-startup uses 00000000-0000-0000-0000-000000000000 as the default target and does not skip the crawl.

Docker Compose / Environment Variables

oc-crawler-openarch:
  image: ghcr.io/opencrawling/oc-crawler:latest
  environment:
    - SPRING_KAFKA_BOOTSTRAP_SERVERS=kafka:9094
    - SPRING_OPENCRAWLING_CRAWL_ON_STARTUP=true
    - SPRING_OPENCRAWLING_REPOSITORY_CONNECTOR_TYPE=openarch
    - SPRING_OPENCRAWLING_CONNECTOR_OPENARCH_ENDPOINT_URL=http://openarch:8080
    - SPRING_OPENCRAWLING_CONNECTOR_OPENARCH_REPOSITORY_ID=default
    - SPRING_OPENCRAWLING_CONNECTOR_OPENARCH_USERNAME=Administrator
    - SPRING_OPENCRAWLING_CONNECTOR_OPENARCH_PASSWORD=Administrator
    - SPRING_OPENCRAWLING_CONNECTOR_OPENARCH_HXQL_QUERY=SELECT * FROM SysFile
    - SPRING_OPENCRAWLING_CONNECTOR_OPENARCH_BATCH_SIZE=100
    - SPRING_OPENCRAWLING_CONNECTOR_OPENARCH_INCLUDE_ACLS=true

Metadata extracted per document

Metadata key Value
every top-level field Each non-null field of the OpenArch document is copied as-is (arrays become multi-valued lists), e.g. sys_name, sys_primaryType, sys_path, sys_modified, custom properties
title sys_title, falling back to sys_name, then to the document ID
description sys_description (if present)
  • Document ID: id, falling back to sys_id, then to a random UUID.
  • URI: openarch://<repositoryId>/<docId>.
  • ACL reference: openarch:acl:<docId>.
  • Last modified: sys_modified parsed as an ISO-8601 instant (or the current time if missing or unparsable).

OIS Document Example (action: "UPSERT")

{
  "id": "7c1e2f8a-4b3d-4e5f-9a0b-1c2d3e4f5a6b",
  "uri": "openarch://default/7c1e2f8a-4b3d-4e5f-9a0b-1c2d3e4f5a6b",
  "action": "UPSERT",
  "metadata": {
    "sys_name": ["contract-2026-0042.pdf"],
    "sys_primaryType": ["SysFile"],
    "sys_mixinTypes": ["SysHasEmbeddings"],
    "sys_modified": ["2026-10-20T09:15:00Z"],
    "title": ["Master Services Agreement 2026"],
    "description": ["Signed MSA with ACME Corp"]
  },
  "security": {
    "inheritanceEnabled": true,
    "permissions": [
      { "identity": "group:legal", "identityType": "group", "access": "read" },
      { "identity": "elena.weber", "identityType": "user", "access": "write" },
      { "identity": "restricted_user", "identityType": "user", "access": "deny" }
    ]
  }
}

📤 Output Connector & Parquet Vector Bridge (OpenArchOutputConnector)

The output connector is active when spring.opencrawling.output.type=openarch. OpenArchClientConfig then registers OpenArchOutputProperties and a shared OpenArchClient bean. On connect() the connector pings /api/query and logs a warning, without failing startup, if the ping does not succeed.

Processing flow for each UPSERT

  1. Binary upload: If uploadBinaries=true and the document has content, the content is uploaded to POST /api/upload with the mimeType / Content-Type metadata (default application/octet-stream). If the upload fails, the connector logs a warning and continues with metadata only.
  2. SysFile document: Builds the OpenArch document:
    • sys_primaryType: "SysFile", sys_name: <OIS id>, sys_title (from title or the document ID) and sys_description (from description).
    • sys_modified and sys_created from the OIS lastModified.
    • sys_mixinTypes: ["SysHasEmbeddings"].
    • file:content: { "data": <blobId>, "name": <OIS id> } when the binary upload succeeded.
    • sys_acl mapped by OpenArchSecurityMapper when syncAcls=true.
    • All other OIS metadata keys that do not start with sys_ (single values become scalars, multiple values stay arrays).
  3. Upsert: POST /api/documents/{rootFolderId}?upsert={upsert}. The id returned by OpenArch becomes the target document ID.
  4. Parquet vector bridge: If an ollamaEmbeddingModel bean is available and the document has content, the connector extracts the text with the Tika text extraction service, splits it with TokenTextSplitter and embeds each chunk. It then writes all chunks of the document to one SNAPPY-compressed Parquet file and uploads it as embeddings-<targetDocId>.parquet (application/vnd.apache.parquet). Chunk IDs are deterministic name-based UUIDs of <targetDocId>#<chunkIndex>, and blobref points to the uploaded binary.

Note

DELETE tombstones: The output connector and the decoupled writer log DELETE actions ("OpenArch tombstone recorded") but do not yet delete anything in OpenArch. See Document Lifecycle Actions & Tombstones.

Configuration Properties

Spring Property (spring.opencrawling.output.openarch.*) connectors.json / Admin UI Key Default Description
endpoint-url endpointUrl http://localhost:8080 OpenArch REST API base URL
repository-id repositoryId default Repository identifier (X-HXCS-Repository)
username username Administrator Basic authentication username
password password Administrator Basic authentication password
root-folder-id rootFolderId 00000000-0000-0000-0000-000000000000 Parent folder of ingested documents
parquet-batch-threshold parquetBatchThreshold 50 Number of chunks buffered by the decoupled writer before a Parquet file is flushed
embeddings-type embeddingsType mxbai-embed-large Value written to the Parquet type column and used for MCP embedding queries
dimensions dimensions 1024 Vector dimensionality (used by the MCP backend)
upsert upsert true Pass upsert=true to /api/documents/{parentId}
sync-acls syncAcls true Map OIS security to sys_acl
upload-binaries uploadBinaries true Upload binaries to /api/upload

Example application.yml 

spring:
  opencrawling:
    output:
      type: openarch
      openarch:
        endpoint-url: "https://openarch.enterprise.local"
        repository-id: "default"
        username: "ingest_service"
        password: "${OPENARCH_PASSWORD}"
        root-folder-id: "00000000-0000-0000-0000-000000000000"
        parquet-batch-threshold: 50
        embeddings-type: "mxbai-embed-large"
        dimensions: 1024
        upsert: true
        sync-acls: true
        upload-binaries: true

Parquet schema (embedding.avsc)

OpenArchParquetEmbeddingsWriter loads /schemas/embedding.avsc (record com.hyland.openarch.embeddings.avro.Embedding) from the classpath and writes Parquet files with AvroParquetWriter and SNAPPY compression.

Field Type Written value
id string Chunk ID (random UUID if missing)
type string embeddingsType (e.g. mxbai-embed-large)
vector array<double> Embedding vector (float vectors are widened to double)
text string Chunk text
blobref ["null", "string"] Uploaded binary ID (in-process connector) or source document ID (decoupled writer)
location record Location text → TextLocation { page, paragraph } when page or paragraph is known; position { left, top, right, bottom }, timestamp { start, end } and spreadsheet { sheet, row, column } are written as null

Decoupled writer (OpenArchStoreWriterConsumer)

In the decoupled Kafka pipeline, oc-writer-consumer writes the embeddings. OpenArchStoreWriterConsumer is active when both spring.opencrawling.output.type=openarch and opencrawling.consumer.writer.enabled=true are set. It:

  • listens on the opencrawling-embedded topic (DocumentEmbeddedMessage),
  • turns each chunk into an EmbeddingChunk (id = chunkId, type = embeddingsType, blobref = documentId),
  • buffers chunks until parquetBatchThreshold is reached, then writes one Parquet file and uploads it as embeddings-batch-<epochMillis>.parquet,
  • logs and skips DELETE tombstones.

Tip

The decoupled compose stack sets SPRING_OPENCRAWLING_OUTPUT_OPENARCH_PARQUET_BATCH_THRESHOLD=1 so that every chunk is flushed immediately during tests. In production, raise the threshold so you upload fewer, larger Parquet files.

🔐 Zero-Trust ACL Mapping

OpenArchSecurityMapper (output) and OpenArchRepositoryConnector (source) translate between OIS SecurityConfig and OpenArch sys_acl entries (ACEs).

OIS → OpenArch (toOpenArchAcl)

OIS access OpenArch ACE
read { "permission": "Read", "granted": true }
write { "permission": "Write", "granted": true }
delete { "permission": "Delete", "granted": true }
admin { "permission": "ManageSecurity", "granted": true }
deny { "permission": "Read", "granted": false }
anything else { "permission": "Read", "granted": true }
  • An identity is treated as a group when identityType=group or the identity starts with group: or GROUP_. Groups become "group": { "groupname": … } with the group: prefix removed. All other identities become "user": { "username": … } with any user: prefix removed.
  • Every ACE is stamped with "creator": "opencrawling".

OpenArch → OIS (fromOpenArchAcl / repository connector)

OpenArch permission OIS access
Read read
Write, ReadWrite write
Delete, DeleteChild delete
ManageSecurity, ManageLock, All admin
any ACE with granted: false deny
  • user.username (or user.id) becomes a user identity. group.groupname (or group.id) becomes a group:<name> identity. A flat username field is also accepted.
  • Mapped documents get inheritanceEnabled: true.

🤖 Secure MCP Server Integration

When spring.opencrawling.output.type=openarch (and opencrawling.mcp.server.enabled is not false), OpenArchMcpSearchBackend (backendType() = "openarch") becomes the active McpVectorSearchBackend of McpVectorServer.

Backend operation OpenArch call Behaviour
secureVectorSearch POST /api/query/embedding Embeds the query with the ollamaEmbeddingModel (or openAiEmbeddingModel as fallback). Sends the vector, embeddingsType and the raw query as chunk full-text, then drops results below minScore
getDocumentContent GET /api/documents/{id} Strips the openarch://<repo>/ prefix and returns sys_title + sys_description as content, with all fields as metadata
listAccessibleSources POST /api/query (SELECT * FROM SysFile, limit 50) Returns id, name (sys_name) and uri (openarch://<repo>/<id>)

OpenArch-specific MCP tools

MCP Tool Parameters Behaviour
openarch_search query (string, required)
userPrincipal (string, required)
userRoles (string, optional, comma-separated)
maxResults (int, optional)
minScore (double, optional)		 Delegates to secureVectorSearch on the active backend
openarch_fetch_document documentId (string, required): OpenArch ID or openarch:// URI
userPrincipal (string, required)
userRoles (string, optional)		 Bare IDs are converted to openarch://default/<id>, then the call is delegated to getDocumentContent

Example JSON-RPC call:

{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "tools/call",
  "params": {
    "name": "openarch_search",
    "arguments": {
      "query": "termination clauses in master services agreements",
      "userPrincipal": "elena.weber",
      "userRoles": "legal",
      "maxResults": 5,
      "minScore": 0.5
    }
  }
}

Important

  • openarch_search and openarch_fetch_document are always registered and run against the active backend, so they reach OpenArch only when spring.opencrawling.output.type=openarch.
  • The MCP backend currently calls OpenArch with the configured service account (spring.opencrawling.output.openarch.username), and every result is tagged openarch:acl:read. userPrincipal and userRoles are logged but are not forwarded to OpenArch or checked by McpAclEvaluator in this backend. Results are therefore limited to what the service account can see. Restrict that account accordingly until per-caller ACL filtering is added.
  • openarch_fetch_document assumes the default repository for bare IDs. For other repositories, pass a full openarch://<repositoryId>/<id> URI.

🖥️ Admin UI, REST Catalogue & CLI

Admin UI (oc-admin-ui)

  1. Navigate to Connectors → New Connector.
  2. Choose one of:
    • Source: Hyland OpenArch Federated Repository (org.opencrawling.openarch.repository.OpenArchRepositoryConnector). Fields: OpenArch Endpoint URL, Repository ID, Crawl Mode (QUERY shows HXQL Query, FOLDER shows Root Folder ID and Include Subfolders), Username, Password, Batch Size (Pages), and toggles for ACLs, content streams and incremental crawling.
    • Output: Hyland OpenArch Federated Ingestion & Parquet Vector Bridge (org.opencrawling.openarch.output.OpenArchOutputConnector). Fields: OpenArch Endpoint URL, Repository ID, Target Root Folder ID, Username, Password, Parquet Vector Batch Threshold, Embeddings Model Name / Type, Embedding Vector Dimensions, and toggles for upsert, ACL sync and binary upload.
  3. Click Test OpenArch Connectivity to run the server-side connection check.

Default connector catalogue (ConnectorController)

The built-in connector catalogue now includes two entries with the defaults listed above:

Name Description Type Class
OpenArch_Repository Hyland OpenArch Federated Repository repository org.opencrawling.openarch.repository.OpenArchRepositoryConnector
OpenArch_Output Hyland OpenArch Federated Ingestion & Parquet Vector Bridge output org.opencrawling.openarch.output.OpenArchOutputConnector

Connection check (ConnectorCheckerService)

For any class name containing OpenArch / openarch, the checker sends POST <endpointUrl>/api/query with {"repositoryId": …, "query": "SELECT * FROM SysContent", "limit": 1}, the X-HXCS-Repository header and Basic auth, with a 5-second timeout.

Response Result
2xx ✅ "Successfully connected to Hyland OpenArch endpoint …"
404 ✅ "Hyland OpenArch server reachable … endpoint returned 404."
Other status / exception ❌ Failure with the HTTP status or exception message

endpointUrl falls back to url, then to http://localhost:8080, and a trailing / is removed.

OpenCrawling CLI (oc-cli)

PR #175 adds a test alias to oc connector check and lets you pass the connector name as a positional argument (see OpenCrawling CLI):

# Positional name + the new "test" alias
oc connector test OpenArch_Repository --type repository

# Classic form
oc connector check --name OpenArch_Output --type output

# Inline overrides of endpoint / credentials during the check
oc connector test OpenArch_Repository --type repository \
  --config endpointUrl=http://openarch.internal:8080 \
  --config username=crawler --config password=secret

Note

--type still defaults to output, so add --type repository when you check OpenArch_Repository.

🐳 Decoupled Docker Compose Stack

The decoupled stack is defined in oc-openarch-output-connector/docker/docker-compose-decoupled-with-openarch.yml. An identical copy lives in oc-openarch-repository-connector/docker/.

Service Container Host ports Role
openarch openarch-decoupled ${OPENARCH_PORT:-8081} → 8080, 5005 (debug) ghcr.io/hyland/openarch-app-jvm:${OPENARCH_VERSION:-latest}
mongodb / mongodb-init mongodb-openarch-decoupled 27017 MongoDB 7 replica set rs0 (hostname mongo), initialized by a one-shot container
opensearch opensearch-openarch-decoupled 9200, 9600 OpenSearch 2.19 for OpenArch full-text and vector search
redis redis-stack-decoupled-openarch 6379, 8001 Redis Stack
ollama / ollama-model-puller ollama-decoupled-openarch 11434 Ollama 0.40 with mxbai-embed-large
kafka kafka-decoupled-openarch 9092 Apache Kafka 3.7
ozone-scm, ozone-om, ozone-datanode, ozone-s3g ozone-*-openarch 9878 (S3G) Apache Ozone 2.2.0 claim-check store
oc-crawler oc-crawler-service-openarch — Scans /data and publishes to Kafka (output.type=openarch)
oc-ingestion-consumer oc-ingestion-consumer-service-openarch — Tika (fork mode) parsing and chunking
oc-embedding-consumer oc-embedding-consumer-service-openarch 8082 Ollama embedding generation
oc-writer-consumer oc-writer-service-openarch — OpenArchStoreWriterConsumer Parquet vector bridge
oc-mcp-server oc-mcp-server-service-openarch ${MCP_PORT:-8080}, 9095 Secure MCP Server backed by OpenArchMcpSearchBackend 
docker compose -f oc-openarch-output-connector/docker/docker-compose-decoupled-with-openarch.yml up -d

All CI and local Dockerfiles (docker/Dockerfile.* and docker/local/Dockerfile.*) now copy both OpenArch modules. See Local Development & Fast Docker Builds.

🧪 Testing & Verification

1. Unit & MockWebServer tests

mvn test -pl oc-openarch-output-connector,oc-openarch-repository-connector,oc-runtime -am
Test class Module Coverage
OpenArchOutputConnectorTest output Binary upload, SysFile upsert, ACL sync, Parquet upload
OpenArchOutputConnectorActivationTest output Conditional bean activation on output.type=openarch
OpenArchParquetEmbeddingsWriterTest output Avro record creation and Parquet round-trip against embedding.avsc
OpenArchSecurityMapperTest output OIS ↔ sys_acl mapping, deny handling, user/group detection
OpenArchStoreWriterConsumerTest output Kafka chunk buffering, threshold flush, DELETE tombstone handling
OpenArchRepositoryConnectorTest repository Paged HXQL, binary streaming, metadata and ACL mapping
OpenArchMcpSearchBackendTest runtime MCP search, fetch and source listing
ConnectorCheckerServiceOpenArchTest runtime Connection check outcomes
OpenCrawlingCliTest#testConnectorTestCommandHelpAndAlias cli oc connector test --help alias

2. OpenArch integration test suite

./scripts/test-openarch-integration.sh

Checks for curl, jq and mvn and probes ${OPENARCH_URL:-http://localhost:8080}/api/status for a live instance. It then runs the Maven test suites of the three OpenArch-related modules.

3. OpenArch decoupled pipeline test suite

./scripts/test-openarch-decoupled.sh

An 11-step suite: prerequisites → cleanup of ports 8080 8081 9200 9092 6379 11434 27017 9878 → image build → infrastructure startup (MongoDB, OpenSearch, Redis, Ollama, OpenArch) → direct /api/upload + /api/documents ingestion with ACLs → oc-crawler scan and Kafka publication → OpenArchStoreWriterConsumer verification → paged HXQL harvesting → MCP readiness (openarch_search, openarch_fetch_document) → OIS DELETE tombstone → full *OpenArch* Maven suite.

If Docker is not available, the script builds with Maven and runs the MockWebServer / Parquet Avro test harness instead.

Variable Default Description
OPENARCH_HOST / OPENARCH_PORT localhost / 8081 Host and port of the OpenArch container
OPENARCH_URL http://${OPENARCH_HOST}:${OPENARCH_PORT} OpenArch base URL
OPENARCH_USERNAME / OPENARCH_PASSWORD Administrator / Administrator Basic auth credentials
OPENARCH_REPOSITORY_ID default Repository ID
OPENARCH_ROOT_FOLDER_ID 00000000-0000-0000-0000-000000000000 Target folder
MCP_PORT 8080 MCP server host port
TIMEOUT 300 Startup timeout (seconds)
KEEP_CONTAINERS false Skip docker compose down on exit

4. CI matrix

Both suites run in .github/workflows/integration-tests.yml as "Hyland OpenArch Integration Test Suite" (test-openarch-integration.sh) and "Hyland OpenArch Decoupled Pipeline Test Suite" (test-openarch-decoupled.sh). See Integration Test — Decoupled Pipeline.

🗺️ Documentation

🧪 Testing

🤝 Community

Clone this wiki locally