Skip to content

Alfresco Content Services Output Connector

Jump to bottom
Piergiorgio Lucidi edited this page Nov 2, 2026 · 1 revision

Alfresco Content Services Output Connector (Migration Mode)

Alfresco Content Services Open Ingestion Standard Zero-Trust Security Java 25

The oc-alfresco-output-connector module migrates enterprise content bit-for-bit from legacy ECM repositories or cloud stores into Alfresco Content Services (ACS) (Community Edition 26.1.0+ or Enterprise). It operates strictly in Migration Mode (opencrawling.pipeline.mode=migration).

While AI and vector output connectors (pgvector, OpenSearch, Solr, Qdrant, Vespa, Luxir) chunk, narrativize, and embed document text for semantic search and LLM context windows, the Alfresco output connector streams raw binary content directly from the Claim Check Store (MinIO / S3 / Local filesystem), reproduces hierarchical folder structures (cm:folder), synchronizes Zero-Trust Access Control Lists (ACLs) with target authorities, updates metadata properties and aspects (cm:versionable, cm:titled), and handles Open Ingestion Standard (OIS) document lifecycle tombstones (such as MOVE_TO_TRASH into the Alfresco Trashcan).

🧭 Pipeline Modes: RAG vs Migration

Dimension rag Mode (Default) migration Mode
Primary Purpose Semantic search, RAG, knowledge retrieval As-is content migration, repository replatforming, ECM consolidation
Document Processing Tika parsing → chunking → embedding generation None: original binary is streamed bit-for-bit via Virtual Threads
Output Target Vector databases / search engines Alfresco Content Services REST API v1 (/nodes)
Storage Payload Text chunks + dense vectors Raw binary stream (cm:content) + metadata properties + aspects
Security & ACLs Vector metadata payload filters Target Alfresco permissions (locallySet ACEs on /permissions)
Search Engine (Solr) Active indexing required Bypassed / noindex by default (direct DB repository writes)

Mode Resolution & Guardrails

  • Global Default: Configured via opencrawling.pipeline.mode (rag or migration, default rag).
  • Per-Job Override: JobDTO.pipelineMode allows setting the mode per individual crawl/migration job.
  • Fail-Fast Guardrail: Attempting to run a job targeting the Alfresco output connector in rag mode is rejected immediately with HTTP 409 Conflict: 
    IllegalStateException: AlfrescoOutputConnector only supports MIGRATION mode (opencrawling.pipeline.mode=migration).
Current mode is RAG. AI narrativization and vector embeddings are not supported by this connector.

Setting Migration Mode

Client / Interface Configuration
CLI oc job start --name <name> --connector <conn> --path <path> --mode migration
oc job create --file job.json --mode migration
Admin UI Select Migration in the Pipeline Mode radio group. Choosing the Alfresco output connector automatically configures Migration Mode.
Java Client SDK JobRequest.builder().pipelineMode("migration").build()
REST API Include "pipelineMode": "migration" in the job payload (POST /api/jobs).

🏗️ Architecture & Data Flow

flowchart LR
    R["Repository Connector<br/>(FileNet, SharePoint, CMIS, FS)"] --> C["oc-crawler-service<br/>(parallel lanes)"]
    C -- "1. upload binary" --> CC[("Claim Check Store<br/>(S3 / MinIO / Local)")]
    C -- "2. publish reference" --> K[["Kafka Topic<br/>opencrawling-documents"]]
    K -- "3. consume<br/>opencrawling-alfresco-migration-group" --> W["AlfrescoMigrationWriterConsumer"]
    W -- "4. stream binary" --> CC
    W -- "5. REST v1: folder + binary + ACLs" --> ACS[("Target Alfresco ACS<br/>(REST API v1)")]
    W -- "6. log missing authorities" --> GAP[("Authority Gap Ledger<br/>(JSON Audit Report)")]
Loading
Component Class / Identification Description
Output Connector org.opencrawling.alfresco.output.AlfrescoOutputConnector Core connector managing node resolution, binary streaming, versioning, and tombstone lifecycle actions.
Decoupled Writer org.opencrawling.alfresco.output.messaging.AlfrescoMigrationWriterConsumer Decoupled Kafka listener on opencrawling-documents under consumer group opencrawling-alfresco-migration-group.
REST Client org.opencrawling.alfresco.output.client.AlfrescoRestClient JDK 25 HTTP client communicating with Alfresco REST API v1 (/nodes, /people, /groups, /deleted-nodes, /probes).
Authority Validator org.opencrawling.alfresco.output.security.AlfrescoAuthorityValidator Validates target users and groups using Java 25 Structured Concurrency (StructuredTaskScope), maintaining an in-memory cache and exporting audit ledgers.
Configuration org.opencrawling.alfresco.output.config.AlfrescoOutputProperties Strongly-typed Spring @ConfigurationProperties(prefix = "spring.opencrawling.output.alfresco").

📁 Target Node Layout & Path Strategies

When documents are ingested into target Alfresco, paths are mapped according to pathStrategy:

Company Home (or targetParentNodeId, e.g. -root-)
└── <targetPathPrefix> (e.g. MigratedContent/v1)
    ├── <subfolder-1>/                             <-- cm:folder
    │   └── <subfolder-2>/                         <-- cm:folder
    │       └── document.pdf                       <-- cm:content
    └── root-document.docx                         <-- cm:content

Path Strategies

  1. HIERARCHICAL (Default):
    • Reads relativePath or relative_path metadata emitted by the source repository connector (e.g. FileSystemRepositoryConnector, FileNetRepositoryConnector, SharePointRepositoryConnector).
    • Automatically provisions missing parent folders (cm:folder) along the path using POST /nodes/{parentId}/children.
    • Caches resolved folder node UUIDs to eliminate redundant REST round-trips.
  2. FLAT:
    • Ingests all documents directly into the base target folder defined by targetParentNodeId and targetPathPrefix.

Conflict Handling Strategies

When a document with the same name already exists in the target folder:

Strategy Behavior
OVERWRITE (Default) Updates the existing node in-place via PUT /nodes/{nodeId}/content and updates properties via PUT /nodes/{nodeId}. Alfresco increments the version (cm:versionable).
AUTO_RENAME Appends a timestamp suffix to the filename before creating a new node (e.g. contract_20261102_153000.pdf).
FAIL Aborts ingestion of the duplicate document and logs a conflict warning in the migration ledger.

🛡️ Zero-Trust Security & Pre-Flight Authority Auditing

The connector translates Open Ingestion Standard (OIS) permission rules into native Alfresco Access Control Entries (locallySet on POST /nodes/{nodeId}/permissions).

Role Mapping

OIS Access Level Target Alfresco Role
read Consumer
write Contributor (or Editor)
delete Collaborator
admin / full_control Coordinator
(unmapped) Configurable via defaultPermissionRole (default Consumer)

Built-in Authorities

The validator automatically recognizes and preserves standard Alfresco built-in authorities without hitting REST endpoints:

  • GROUP_EVERYONE, EVERYONE
  • GUEST
  • ROLE_ANONYMOUS, ROLE_AUTHENTICATED, ROLE_ADMINISTRATOR, ADMIN

Pre-Flight vs In-Flight Authority Auditing

In enterprise migrations, source repositories frequently contain orphaned users or deprecated groups that do not exist in the target Alfresco LDAP/Active Directory sync. If unhandled, Alfresco rejects permission payloads with HTTP 400 Bad Request or 404 Not Found.

The connector provides configurable authority validation modes:

Validation Mode (authorityValidationMode) Behavior
IN_FLIGHT (Default) Checks authorities during document ingestion. Results are cached in a thread-safe ConcurrentHashMap. Valid authorities are applied; missing authorities are skipped (or failed based on missingAuthorityAction).
PRE_FLIGHT Executes a parallel pre-crawl audit across all distinct authorities using Java 25 StructuredTaskScope. Produces an early gap report before content ingestion begins.
BOTH Executes pre-flight batch audit and verifies dynamic in-flight additions.
DISABLED Directly submits permissions without pre-validation.

Missing Authority Action & Gap Ledger Report

When an authority is missing from target Alfresco:

  • If missingAuthorityAction=SKIP: The missing authority is omitted from the document's permission list so valid permissions are successfully applied, and the gap is recorded.
  • If missingAuthorityAction=FAIL: Ingestion of the affected document is aborted with an error.

All omitted or missing authorities are exported to a structured Gap Ledger Report:

{
  "timestamp": "2026-11-02T22:00:00.000Z",
  "targetUrl": "http://localhost:8081/alfresco",
  "missingAuthoritiesCount": 2,
  "missingAuthorities": [
    {
      "authority": "retired_contractor",
      "identityType": "user",
      "firstEncounteredNode": "doc-legal-001",
      "timestamp": "2026-11-02T22:00:01.120Z"
    },
    {
      "authority": "GROUP_LEGACY_FINANCE",
      "identityType": "group",
      "firstEncounteredNode": "doc-fin-042",
      "timestamp": "2026-11-02T22:00:02.450Z"
    }
  ]
}

🗑️ Document Lifecycle Actions & Tombstones

The connector natively supports Open Ingestion Standard (OIS) lifecycle actions:

  • UPSERT: Downloads binary from Claim Check store, provisions folders, creates/updates cm:content node, sets properties and Zero-Trust ACLs.
  • DELETE: Resolves target node UUID and executes lifecycle action configured by tombstoneAction:
    • MOVE_TO_TRASH (Default): Calls DELETE /nodes/{nodeId}?permanent=false. Moves the item to the Alfresco Trashcan, preserving audit trails and enabling accidental deletion recovery.
    • PERMANENT_DELETE: Calls DELETE /nodes/{nodeId}?permanent=true. Purges the node from PostgreSQL and content store immediately.

⚡ Docker Compose & Search Subsystem (Solr6 vs noindex)

The migration stack is defined in oc-alfresco-output-connector/docker/docker-compose-decoupled-with-alfresco.yml.

Search Engine Disabled by Default (noindex)

In Migration Mode, OpenCrawling exclusively uses direct repository REST endpoints against Alfresco's database (PostgreSQL). No search queries are executed during ingestion.

  • -Dindex.subsystem.name=${ALFRESCO_INDEX_SUBSYSTEM:-noindex} defaults to noindex.
  • Zero Log Spam: Eliminates UnknownHostException: solr6 and connection retry warnings.
  • Fast Startup: Alfresco container boots in ~25–30s instead of ~90s.
  • Resource Efficient: Saves ~2 GB RAM and reduces CPU load during bulk migration.

Enabling Solr 6 Search Services on Demand

If search indexing and AFTS queries are needed on the target Alfresco instance, a preconfigured solr6 container is available under Docker Compose profiles ["search", "solr"]:

# Set index subsystem to solr6
export ALFRESCO_INDEX_SUBSYSTEM=solr6

# Start stack with search profile enabled
docker compose -f oc-alfresco-output-connector/docker/docker-compose-decoupled-with-alfresco.yml --profile search up -d

⚙️ Configuration Reference

Application Configuration (application.yml)

spring:
  opencrawling:
    output:
      type: alfresco
      alfresco:
        enabled: true
        url: http://localhost:8081/alfresco
        username: admin
        password: admin
        auth-type: basic                      # basic | bearer
        bearer-token: ""
        target-parent-node-id: -root-         # -root- | -my- | -shared- | <node-uuid>
        target-path-prefix: MigratedContent/v1
        path-strategy: HIERARCHICAL           # HIERARCHICAL | FLAT
        node-type: cm:content
        aspects:
          - cm:versionable
          - cm:titled
        conflict-strategy: OVERWRITE          # OVERWRITE | AUTO_RENAME | FAIL
        sync-permissions: true
        default-permission-role: Consumer     # Consumer | Contributor | Editor | Collaborator | Coordinator
        inheritance-enabled: true
        tombstone-action: MOVE_TO_TRASH       # MOVE_TO_TRASH | PERMANENT_DELETE
        validate-authorities: true
        authority-validation-mode: IN_FLIGHT  # IN_FLIGHT | PRE_FLIGHT | BOTH | DISABLED
        missing-authority-action: SKIP        # SKIP | FAIL
        authority-gap-report-path: reports/alfresco-authority-gap.json
        timeout-seconds: 60
        parallelism: 4

opencrawling:
  pipeline:
    mode: migration                           # Required for Alfresco output connector

Full Properties Table

Property Default Description
spring.opencrawling.output.alfresco.enabled true Enables the Alfresco output connector bean.
spring.opencrawling.output.alfresco.url http://localhost:8081/alfresco Base REST API URL of target Alfresco instance.
spring.opencrawling.output.alfresco.username admin Username for Basic authentication.
spring.opencrawling.output.alfresco.password admin Password for Basic authentication.
spring.opencrawling.output.alfresco.auth-type basic Authentication mechanism: basic or bearer.
spring.opencrawling.output.alfresco.bearer-token "" JWT / Bearer token if auth-type=bearer.
spring.opencrawling.output.alfresco.target-parent-node-id -root- Starting folder node UUID or alias (-root-, -my-, -shared-).
spring.opencrawling.output.alfresco.target-path-prefix "" Root folder path prefix under parent node (e.g. MigratedContent/v1).
spring.opencrawling.output.alfresco.path-strategy HIERARCHICAL Path layout: HIERARCHICAL (subfolders preserved) or FLAT.
spring.opencrawling.output.alfresco.node-type cm:content Alfresco content type for uploaded documents.
spring.opencrawling.output.alfresco.aspects ["cm:versionable", "cm:titled"] List of Alfresco aspects applied to created nodes.
spring.opencrawling.output.alfresco.conflict-strategy OVERWRITE Duplicate handling: OVERWRITE (version bump), AUTO_RENAME, or FAIL.
spring.opencrawling.output.alfresco.sync-permissions true Replicate Zero-Trust ACL permissions to target node.
spring.opencrawling.output.alfresco.default-permission-role Consumer Target role when source access level is unmapped.
spring.opencrawling.output.alfresco.inheritance-enabled true Sets isInheritanceEnabled flag on target node ACL.
spring.opencrawling.output.alfresco.tombstone-action MOVE_TO_TRASH Deletion lifecycle: MOVE_TO_TRASH (recoverable) or PERMANENT_DELETE.
spring.opencrawling.output.alfresco.validate-authorities true Toggles authority existence validation before applying ACEs.
spring.opencrawling.output.alfresco.authority-validation-mode IN_FLIGHT Validation timing: IN_FLIGHT, PRE_FLIGHT, BOTH, or DISABLED.
spring.opencrawling.output.alfresco.missing-authority-action SKIP Behavior when authority is absent: SKIP (omits ACE) or FAIL (aborts doc).
spring.opencrawling.output.alfresco.authority-gap-report-path reports/alfresco-authority-gap.json Path where missing authority gap ledger report is exported.
spring.opencrawling.output.alfresco.timeout-seconds 60 HTTP client request timeout in seconds.
spring.opencrawling.output.alfresco.parallelism 4 Concurrency worker threads for writing into Alfresco.

🖥️ Admin UI Integration

In the OpenCrawling Admin UI:

  • Output Connectors Catalog: Appears as Alfresco Content Services (Migration Mode only).
  • Connection Checker: Validates endpoint connectivity by probing /alfresco/api/-default-/public/alfresco/versions/1/probes/-ready-. Distinguishes between HTML responses (such as OpenCrawling Admin UI on port 8080) and true Alfresco REST JSON responses on port 8081, providing clear diagnostic guidance.
  • Automatic Migration Guardrail: Selecting Alfresco Output automatically configures the job for Migration Mode.

🧪 Testing & Verification

1. Maven Unit Suite

Execute the dedicated unit suite verifying node resolution, authority auditing, and Kafka consumer processing:

mvn test -pl oc-alfresco-output-connector

2. End-to-End Decoupled Integration Test

The repository includes a decoupled integration test script:

# Run end-to-end integration test
./scripts/test-alfresco-migration-decoupled.sh

# Skip Docker image build if already compiled:
SKIP_BUILD=true ./scripts/test-alfresco-migration-decoupled.sh

Verifications performed by the test suite:

  1. Migration Mode RAG Bypass: Asserts that vector_store tables in PostgreSQL contain 0 records (text parsing and embeddings were completely bypassed).
  2. Binary Content Integrity: Downloads the migrated binary from target Alfresco (GET /nodes/{id}/content) and verifies exact bit-for-bit content match.
  3. Hierarchical Folder Path Preservation: Confirms that nested subfolder documents (e.g. subfolder/subfolder-spec.txt) are created under cm:folder nodes matching source directory hierarchies.
  4. Metadata & Aspects Synchronization: Asserts cm:content node type and checks cm:versionable and cm:titled aspects.
  5. OIS Document Lifecycle Tombstone: Verifies that a DELETE tombstone moves the document into the Alfresco Trashcan (GET /deleted-nodes/{id} returns HTTP 200).

🗺️ Documentation

🧪 Testing

🤝 Community

Clone this wiki locally