AWS S3 Output Connector

AWS S3 Output Connector (Migration Mode)

The oc-s3-output-connector module migrates enterprise content bit-for-bit into Amazon Simple Storage Service (AWS S3) and S3-compatible cloud object stores (MinIO, Ceph, Cloudflare R2, Google Cloud Storage S3 API). It operates strictly in Migration Mode (opencrawling.pipeline.mode=migration).

While AI and vector output connectors (pgvector, OpenSearch, Solr, Qdrant, Vespa, Luxir) chunk, narrativize, and embed document text for LLM semantic search, the AWS S3 output connector preserves raw document binaries with cryptographic fidelity (SHA-256 on the fly) and produces companion Open Ingestion Standard (OIS) Zero-Trust JSON metadata sidecars (<key>.ois.json), completely bypassing compute-heavy text parsing and embedding generation.

Note

Migration Mode (this page) is distinct from the Migration Guide. The Migration Guide covers migrating legacy Apache ManifoldCF configurations to OpenCrawling.

🧭 Pipeline Modes: RAG vs Migration

Dimension	`rag` Mode (Default)	`migration` Mode
Primary Purpose	Semantic search, RAG, knowledge retrieval	As-is content migration, cloud data lakes, ECM data liberation
Document Processing	Tika parsing → chunking → embedding generation	None: original binary is streamed bit-for-bit
Output Target	Vector databases / search engines	AWS S3 or S3-compatible object stores
Storage Payload	Text chunks + dense vectors	Raw binary + OIS companion JSON sidecar + SHA-256 digest
Security & ACLs	Metadata filters in vector indices	Zero-Trust ACLs embedded in companion `.ois.json` sidecar

Mode Resolution & Guardrails

Global Default: Configured via opencrawling.pipeline.mode (rag or migration, default rag).
Per-Job Override: JobDTO.pipelineMode allows overriding the global mode per individual job.

Fail-Fast Guardrail: Attempting to run a job targeting the AWS S3 output connector in rag mode is rejected immediately with HTTP 409 Conflict:

IllegalStateException: S3OutputConnector only supports MIGRATION mode (opencrawling.pipeline.mode=migration).
Current mode is RAG. AI narrativization and vector embeddings are not supported by this connector.

Setting Migration Mode

Client / Interface	Configuration
CLI	`oc job start --name <name> --connector <conn> --path <path> --mode migration` `oc job create --file job.json --mode migration`
Admin UI	Select Migration in the Pipeline Mode radio group. Choosing AWS S3 output automatically enforces Migration Mode.
Java Client SDK	`JobRequest.builder().pipelineMode("migration").build()`
REST API	Include `"pipelineMode": "migration"` in the job payload (`POST /api/jobs`).

🏗️ Architecture & Data Flow

flowchart LR
    R["Repository Connector<br/>(FileNet, SharePoint, CMIS, FS)"] --> C["oc-crawler-service<br/>(parallel lanes)"]
    C -- "1. upload binary" --> CC[("Claim Check Store<br/>(S3 / MinIO)")]
    C -- "2. publish reference" --> K[["Kafka Topic<br/>opencrawling-documents"]]
    K -- "3. consume<br/>opencrawling-s3-migration-group" --> W["S3MigrationWriterConsumer"]
    W -- "4. stream binary" --> CC
    W -- "5. binary + .ois.json sidecar + index" --> S3[("Target AWS S3 Bucket")]

Component	Class / Identification	Description
Output Connector	`org.opencrawling.s3.S3OutputConnector`	Core connector handling binary streaming, SHA-256 calculation, sidecar generation, and lifecycle actions.
Decoupled Writer	`org.opencrawling.s3.messaging.S3MigrationWriterConsumer`	Decoupled Kafka listener on `opencrawling-documents` under consumer group `opencrawling-s3-migration-group`.
Client Factory	`org.opencrawling.s3.client.S3ClientFactory`	AWS SDK v2 client initialization supporting default credentials chain, static keys, and STS AssumeRole.
Multipart Uploader	`org.opencrawling.core.s3.S3MultipartUploader`	Parallel multipart upload engine for files exceeding `multipart-threshold`.
Sidecar Model	`org.opencrawling.s3.model.OisMigrationDocument`	Canonical Zero-Trust JSON companion model adhering to OIS v1 schema.

📁 Target S3 Object & Sidecar Layout

For every migrated document, S3OutputConnector writes objects into the destination S3 bucket using a clean layout:

s3://<bucket>/
├── <prefix>/<path>/<document-name>.<ext>           <-- Raw pristine binary
├── <prefix>/<path>/<document-name>.<ext>.ois.json <-- Companion OIS Zero-Trust JSON sidecar
├── .opencrawling/
│   └── index/
│       └── <sha256(documentId)>                   <-- Fast lookup index pointer
└── .tombstones/
    └── <prefix>/<path>/<document-name>.<ext>.ois.json <-- Preserved tombstone (on DELETE)

Document Lifecycle Actions

UPSERT:
1. Streams binary directly from ClaimCheckStore to S3 computing SHA-256 on the fly.
2. For large files exceeding multipart-threshold, executes parallel multipart upload via S3MultipartUploader.
3. Uploads companion <key>.ois.json metadata sidecar with Zero-Trust permissions.
4. Stores fast reverse-lookup pointer at .opencrawling/index/<sha256(documentId)>.
DELETE:
- DELETE_OBJECTS: Permanently deletes the binary, companion sidecar, and index pointer from target S3.
- ARCHIVE_TOMBSTONE: Deletes the binary, marks the OIS sidecar as action: "DELETE" with tombstone timestamp, and archives it under .tombstones/<key>.ois.json.

📜 Companion OIS JSON Sidecar Schema

The companion .ois.json sidecar guarantees full data governance, traceability, and Zero-Trust access control:

{
  "$schema": "https://opencrawling.org/schemas/v1/ois-document.json",
  "id": "/contracts/2026/master_agreement.pdf",
  "action": "UPSERT",
  "uri": "s3://enterprise-migration-lake/contracts/2026/master_agreement.pdf",
  "lastModified": "2026-10-30T16:08:43.563Z",
  "source": {
    "type": "repository",
    "originalUri": "filenet://OS1/Document/{C1234-5678-ABCD}",
    "originalPath": "/Contracts/2026/master_agreement.pdf"
  },
  "contentRef": {
    "key": "contracts/2026/master_agreement.pdf",
    "bucket": "enterprise-migration-lake",
    "region": "us-east-1",
    "claimCheckUri": "s3://claims/contracts_2026_master_agreement.pdf",
    "filename": "master_agreement.pdf",
    "mimeType": "application/pdf",
    "contentLength": 16777216,
    "checksumSha256": "4319ce2aee714d6960661857472736ad486665bf1a5b09b1782bf25642a9be01",
    "storageClass": "INTELLIGENT_TIERING",
    "eTag": "\"43bc60be1ce1aa9225c2fb83f95afea3-4\""
  },
  "metadata": {
    "contract_id": ["CTR-2026-09"],
    "department": ["Legal", "Procurement"],
    "classification": ["Confidential"]
  },
  "security": {
    "inheritanceEnabled": false,
    "permissions": [
      {
        "identity": "legal-reviewers",
        "identityType": "group",
        "displayName": "Legal Reviewers",
        "access": "read"
      },
      {
        "identity": "admin@enterprise.internal",
        "identityType": "user",
        "displayName": "Global Admin",
        "access": "full_control"
      }
    ]
  },
  "acl": "legal-reviewers:read,admin@enterprise.internal:full_control"
}

⚙️ Configuration Reference

Application Configuration (`application.yml`)

spring:
  opencrawling:
    output:
      type: s3
      s3:
        bucket: enterprise-migration-lake
        region: us-east-1
        credentials-strategy: DEFAULT_CHAIN   # DEFAULT_CHAIN | STATIC | ASSUME_ROLE
        # For STATIC:
        access-key: AKIA...
        secret-key: wJalr...
        # For ASSUME_ROLE:
        role-arn: arn:aws:iam::123456789012:role/OpenCrawlingMigrationRole
        role-session-name: opencrawling-s3-writer
        external-id: enterprise-migration-token
        # S3 Features:
        storage-class: INTELLIGENT_TIERING     # STANDARD, GLACIER_IR, DEEP_ARCHIVE, etc.
        server-side-encryption: SSE-KMS        # NONE | SSE-S3 | SSE-KMS
        kms-key-id: arn:aws:kms:us-east-1:123456789012:key/abc-123
        bucket-key-enabled: true
        # Performance & Multipart:
        multipart-threshold: 64MB              # Threshold for parallel multipart uploads
        multipart-part-size: 16MB              # Chunk part size (min 5MB)
        multipart-concurrency: 4               # Concurrent upload threads per large object
        # Consumer & Key Layout:
        key-strategy: HIERARCHICAL             # HIERARCHICAL | FLAT
        sidecar-suffix: .ois.json
        tombstone-action: ARCHIVE_TOMBSTONE    # DELETE_OBJECTS | ARCHIVE_TOMBSTONE
        auto-create-bucket: false
        # Custom HTTP Headers / Vendor Extensions:
        custom-headers:
          x-emc-namespace: "finance-tenant"
          x-emc-metadata-search: "Department=${metadata.department},Classification=${metadata.classification | Public}"
          x-migration-source: "IBM-FileNet-P8"

opencrawling:
  pipeline:
    mode: migration                            # Required for AWS S3 output connector

Full Properties Table

Property	Default	Description
`spring.opencrawling.output.s3.bucket`	Required	Destination S3 bucket name.
`spring.opencrawling.output.s3.region`	`us-east-1`	AWS region where the bucket resides.
`spring.opencrawling.output.s3.endpoint-override`	null	Custom endpoint URL (e.g. `http://localhost:9000` for MinIO/LocalStack).
`spring.opencrawling.output.s3.path-style-access`	`false`	Enables path-style addressing (required for MinIO/Ceph).
`spring.opencrawling.output.s3.credentials-strategy`	`DEFAULT_CHAIN`	`DEFAULT_CHAIN` (EKS IRSA, EC2 IAM roles, env), `STATIC`, or `ASSUME_ROLE`.
`spring.opencrawling.output.s3.access-key`	null	AWS access key ID for `STATIC` credentials.
`spring.opencrawling.output.s3.secret-key`	null	AWS secret access key for `STATIC` credentials.
`spring.opencrawling.output.s3.role-arn`	null	IAM Role ARN to assume when `credentials-strategy=ASSUME_ROLE`.
`spring.opencrawling.output.s3.role-session-name`	`opencrawling-s3-writer`	Session identifier for STS `AssumeRole`.
`spring.opencrawling.output.s3.external-id`	null	Optional external ID for cross-account STS `AssumeRole`.
`spring.opencrawling.output.s3.storage-class`	`STANDARD`	S3 Storage Class: `STANDARD`, `INTELLIGENT_TIERING`, `GLACIER_IR`, `GLACIER`, `DEEP_ARCHIVE`, `ONEZONE_IA`.
`spring.opencrawling.output.s3.server-side-encryption`	`NONE`	Server-side encryption: `NONE`, `SSE-S3` (`AES256`), or `SSE-KMS` (`aws:kms`).
`spring.opencrawling.output.s3.kms-key-id`	null	AWS KMS Key ARN/ID used when encryption is `SSE-KMS`.
`spring.opencrawling.output.s3.bucket-key-enabled`	`false`	Enables S3 Bucket Keys to reduce KMS encryption costs by up to 99%.
`spring.opencrawling.output.s3.checksum-algorithm`	`SHA256`	Checksum algorithm verified during transfer.
`spring.opencrawling.output.s3.key-strategy`	`HIERARCHICAL`	`HIERARCHICAL` (mirrors folder hierarchy) or `FLAT` (sanitized document ID).
`spring.opencrawling.output.s3.prefix`	`""`	Optional key prefix applied to all uploaded objects.
`spring.opencrawling.output.s3.sidecar-suffix`	`.ois.json`	Suffix for the companion OIS JSON sidecar file.
`spring.opencrawling.output.s3.tombstone-action`	`DELETE_OBJECTS`	Lifecycle action on document delete: `DELETE_OBJECTS` or `ARCHIVE_TOMBSTONE`.
`spring.opencrawling.output.s3.custom-headers`	`{}`	Map of custom HTTP request headers with static values or dynamic interpolation templates.
`spring.opencrawling.output.s3.auto-create-bucket`	`false`	Auto-creates the target bucket if missing (useful for testing).
`spring.opencrawling.output.s3.multipart-threshold`	`64MB`	Files exceeding this size are uploaded via parallel multipart uploads.
`spring.opencrawling.output.s3.multipart-part-size`	`16MB`	Multipart part chunk size (minimum `5MB`).
`spring.opencrawling.output.s3.multipart-concurrency`	`4`	Number of concurrent threads uploading parts per large object.
`spring.opencrawling.output.s3.consumer-group`	`opencrawling-s3-migration-group`	Kafka consumer group for the decoupled migration writer.
`spring.opencrawling.output.s3.consumer-concurrency`	`3`	Number of concurrent Kafka consumer threads in the writer service.

🖥️ Admin UI Integration & Multi-Vendor Storage Presets

In the OpenCrawling Admin UI, the connector appears in the Output Connectors catalog as AWS S3 (Migration Mode only) marked with an orange Cloud icon:

Automatically checks and suggests switching the job to Migration Mode.
Provides input fields for Bucket, Region, Endpoint Override, and Path-Style Access.
Dropdown for Credentials Strategy (DEFAULT_CHAIN, STATIC, ASSUME_ROLE) showing dynamic fields (Access Key, Secret Key, Role ARN, Session Name, External ID).
Selectors for Storage Class (STANDARD, INTELLIGENT_TIERING, GLACIER_IR, etc.) and Server-Side Encryption (SSE-S3, SSE-KMS, KMS Key ID, Bucket Key).
Selectors for Key Strategy (HIERARCHICAL, FLAT) and Tombstone Action (DELETE_OBJECTS, ARCHIVE_TOMBSTONE).
Custom HTTP Request Headers Editor: Interactive dynamic key-value table allowing operators to configure custom request headers.
Vendor Storage Quick Presets: Built-in presets for instant configuration:

Vendor / Storage Target	Preset Name	Injected Request Headers	Purpose
Dell EMC ECS	Metadata Search & Multi-Tenancy	`x-emc-metadata-search` `x-emc-namespace` `x-emc-retention-period`	Injects document metadata for ECS real-time indexing, tenant routing, and retention hold.
Dell EMC ECS	Tenant Namespace only	`x-emc-namespace`	Routes objects to a designated tenant namespace without multi-bucket sprawl.
IBM Cloud Object Storage	Key Protect Customer Root Key	`ibm-sse-kp-customer-root-key-crn` `ibm-sse-kp-encryption-algorithm`	Enforces envelope encryption with IBM Key Protect customer root key (CRN).
Hitachi Content Platform (HCP)	HCP Retention & Shredding	`x-hcp-retention` `x-hcp-shred`	Enforces WORM compliance retention hold and cryptographic shredding on disposal.
NetApp StorageGRID	Compliance Retention	`x-ntap-sg-compliance-retention`	Sets NetApp StorageGRID regulatory compliance retention lock timestamp.
Cloudflare R2	Jurisdiction Placement	`cf-placement`	Sets regional data sovereignty and localization hints (e.g. `eu`, `fedramp`).
Standard S3 / MinIO	Custom User Metadata	`x-amz-meta-source-id` `x-amz-meta-department`	Attaches custom user metadata using document and metadata variables.

Header Template Interpolation Syntax

Headers support dynamic template placeholders resolved during request execution:

${metadata.<name>}: Injects metadata from the document (multiple values joined with commas).
${metadata.<name> | <fallback>}: Injects metadata with a default fallback if empty or missing.
${document.id}: Unique document identifier.
${document.uri}: Original document source URI.
${document.action}: Lifecycle action (UPSERT or DELETE).
${job.id}: Active crawling job identifier.

🧪 Testing & Verification

1. Maven Unit Suite

Execute the dedicated unit suite verifying connector fail-fast behavior, client factory credential strategies, and Kafka consumer processing:

mvn test -pl oc-s3-output-connector

2. End-to-End Decoupled Integration Test

The repository includes an end-to-end integration test exercising the full decoupled microservices pipeline against MinIO S3:

# Run complete integration suite against MinIO S3
./scripts/test-s3-migration-decoupled.sh

# Skip Docker image build if already compiled:
SKIP_BUILD=true ./scripts/test-s3-migration-decoupled.sh

Verifications performed by the test:

Migration Mode RAG Bypass: Asserts that vector_store tables in PostgreSQL contain 0 records (narrativization and embeddings were skipped).
Binary Integrity: Retrieves the migrated document from target S3 and verifies exact SHA-256 match with the source.
Companion OIS JSON Sidecar: Inspects <key>.ois.json, validating schema compliance, action: "UPSERT", content references, and Zero-Trust ACLs.
Decoupled Kafka Consumer: Verifies S3MigrationWriterConsumer consumed events from opencrawling-documents and committed offsets.
OIS Tombstone Deletion & Header Template Resolver: Verifies ARCHIVE_TOMBSTONE creates .tombstones/<key>.ois.json, purges the binary object, and asserts dynamic header interpolation and fallback handling via HeaderTemplateResolverTest.
Parallel Multipart Upload: Tests a 16 MB binary upload with low threshold (8 MB), verifying bit-for-bit SHA-256 integrity and multipart part consolidation.

🗺️ Documentation

Home
User & Installation Guide
Quick Start
VM & Bare-Metal Distribution Installation
Local Development & Fast Docker Builds
User Guide
Architecture
Zero-Trust Security & ACL Management
Document Lifecycle Actions & Tombstones
Apache Tika 4 Text Extraction
Configuration Guide
Connectors Reference
Alfresco Content Services (ACS) Connector
Alfresco Process Services (APS) Connector
Flowable Repository Connector
Camunda Repository Connector
OASIS CMIS Repository Connector
Doxis Repository Connector
M-Files Repository Connector
SharePoint Online Repository Connector
IBM FileNet P8 Repository Connector
OpenText Documentum Repository Connector
Hyland OpenArch Integration (Repository, Output & MCP)
Relational Database (JDBC) Connector
Apache StormCrawler Repository Connector
gRPC Push Repository Connector
Apache Solr Output Connector
Luxir Output Connector
Apache SeaTunnel Output Connector
Apache Ozone Output Connector (Migration Mode)
AWS S3 Output Connector (Migration Mode)
Connector Execution Modes
Model Context Protocol
Apache Ozone Claim Check Store
Lakehouse Audit Reporting & Ledger
AI-Powered Observability (AIOps)
Auto-Narrativization Copilot
Maven Connector Archetypes
Java Client SDK
OpenCrawling CLI
OpenAPI Specification & Swagger UI
gRPC Internal Transport
Migration Guide (ManifoldCF)

🧪 Testing

Integration Test — Decoupled Pipeline
M-Files Integration Test
SharePoint Online Integration Test
IBM FileNet Integration Test
OpenText Documentum Integration Test
Hyland OpenArch Integration Test
Local Docker Build Test
Secure MCP Server Integration Test
Narrativization Integration Test
Reporting Integration Test
AWS S3 Migration Integration Test

AWS S3 Output Connector

AWS S3 Output Connector (Migration Mode)

🧭 Pipeline Modes: RAG vs Migration

Mode Resolution & Guardrails

Setting Migration Mode

🏗️ Architecture & Data Flow

📁 Target S3 Object & Sidecar Layout

Document Lifecycle Actions

📜 Companion OIS JSON Sidecar Schema

⚙️ Configuration Reference

Application Configuration (application.yml)

Full Properties Table

🖥️ Admin UI Integration & Multi-Vendor Storage Presets

Header Template Interpolation Syntax

🧪 Testing & Verification

1. Maven Unit Suite

2. End-to-End Decoupled Integration Test

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

🗺️ Documentation

🧪 Testing

🤝 Community

Clone this wiki locally

Application Configuration (`application.yml`)