Skip to content

Apache Iceberg Output Connector

Piergiorgio Lucidi edited this page Nov 13, 2026 · 3 revisions

Apache Iceberg Output Connector

Apache Iceberg Apache Parquet Java Spring Boot OIS Compliance

Module: oc-iceberg-output-connector
Direct Connector Class: org.opencrawling.iceberg.output.IcebergOutputConnector
Decoupled Kafka Writer: org.opencrawling.iceberg.output.messaging.IcebergStoreWriterConsumer
Catalog Factory: org.opencrawling.iceberg.output.catalog.IcebergCatalogFactory
Table & Schema Manager: org.opencrawling.iceberg.output.schema.IcebergTableManager
Record & Parquet Writer: org.opencrawling.iceberg.output.writer.IcebergRecordWriter
Spring Bean: @Component (activated when spring.opencrawling.output.type=iceberg)


📌 Overview

The Apache Iceberg Output Connector integrates Apache Iceberg (v1.11.0) into OpenCrawling as an enterprise-grade lakehouse table destination. It allows structured, semi-structured, and unstructured documents crawled from enterprise repositories (SharePoint, Alfresco, Documentum, FileNet, M-Files, Doxis, JDBC, Web, gRPC Push) to be streamed into high-performance columnar Apache Parquet tables managed under ACID transactions with snapshot isolation, time travel, and partition pruning.

The connector supports both direct synchronous batching (IcebergOutputConnector.commitBatch(...)) and asynchronous decoupled streaming via Kafka (IcebergStoreWriterConsumer). Every ingested document is written with the canonical OpenCrawling lakehouse schema, complete with Zero-Trust Open Ingestion Standard (OIS) Access Control Lists (ACLs), cryptographic checksums, extracted text, and dynamic entity evolution.

flowchart TD
    subgraph Ingestion["OpenCrawling Ingestion Pipeline"]
        Doc["Crawled RepositoryDocument"]
        Direct["IcebergOutputConnector<br/>(Direct / Standalone Batching)"]
        Crawler["oc-crawler / Ingestion"]
        Kafka[("Kafka Topic<br/>(opencrawling-documents)")]
        Writer["IcebergStoreWriterConsumer<br/>(Decoupled Kafka Stage)"]
        ClaimCheck[("Claim Check Store<br/>MinIO / S3 / Ozone")]
    end

    subgraph CoreEngine["Iceberg Lakehouse Core Engine"]
        Factory["IcebergCatalogFactory<br/>(REST, Hive, Hadoop, Glue, JDBC, In-Memory)"]
        TableMgr["IcebergTableManager<br/>(Namespace Provisioning & PartitionSpec)"]
        SchemaEvol["Dynamic Schema Evolution<br/>(Custom Entity Columns)"]
        RecWriter["IcebergRecordWriter<br/>(GenericRecord + Parquet FileAppender)"]
    end

    subgraph Storage["Lakehouse Storage Tier"]
        Catalog["Iceberg REST Catalog / Metastore"]
        S3IO["org.apache.iceberg.aws.s3.S3FileIO<br/>(AWS S3 / MinIO)"]
        ParquetData[("Columnar Parquet Data Files<br/>(s3://warehouse/...)")]
        Snapshots[("Iceberg Metadata & Snapshots<br/>(ACID Appends & Time Travel)")]
    end

    subgraph Consumption["Downstream Analytical Consumption"]
        DuckDB["DuckDB OLAP Engine"]
        Spark["Apache Spark"]
        Trino["Trino / Starburst"]
        Snowflake["Snowflake / BigQuery"]
    end

    Doc -->|Direct Mode| Direct
    Doc -->|Decoupled Mode| Crawler
    Crawler -->|Upload Binary| ClaimCheck
    Crawler -->|Publish IngestionMessage| Kafka
    Kafka --> Writer
    Writer -->|Stream Binary| ClaimCheck
    Direct & Writer --> TableMgr
    Factory --> Catalog
    TableMgr --> Factory
    TableMgr --> SchemaEvol
    SchemaEvol --> RecWriter
    RecWriter --> S3IO
    S3IO --> ParquetData
    RecWriter --> Snapshots
    Catalog --> Snapshots
    ParquetData & Snapshots <--> DuckDB & Spark & Trino & Snowflake
Loading

🚀 Key Features

  • Multi-Catalog Architecture: Pluggable support for enterprise catalog technologies through IcebergCatalogFactory:
    • REST Catalog (catalog-type: rest): Standard REST catalog protocol compatible with Apache Iceberg REST fixture, Tabular, Apache Polaris, and Lakekeeper.
    • Hive Metastore (catalog-type: hive): Enterprise metadata integration via iceberg-hive-metastore.
    • Hadoop Catalog (catalog-type: hadoop): Direct filesystem or HDFS/S3 directory-backed catalog.
    • In-Memory Catalog (catalog-type: in-memory): Ephemeral catalog for testing and isolated validation.
    • AWS Glue Catalog (catalog-type: glue): Native AWS lakehouse integration via AWS SDK Glue clients.
    • JDBC Catalog (catalog-type: jdbc): Relational catalog backend via JDBC.
  • Canonical Enterprise Lakehouse Schema: Every document is mapped to a strongly typed schema including document identification, content typing, cryptographic SHA-256 validation, extracted text, and claim check URIs.
  • Zero-Trust OIS ACL Security Enforcement: Preserves document access control lists directly within Iceberg table schemas:
    • security_owner (string): Primary document owner / author.
    • security_inheritance (boolean): Whether permission inheritance is active.
    • allowed_principals (list<string>): Authorized users, groups, and roles mapped via EffectiveAcl.
    • denied_principals (list<string>): Explicitly denied identities.
  • Dynamic Runtime Schema Evolution: Custom extracted metadata and entities (e.g. entity.vendor, entity.amount, entity.contract_id, invoice_date, iban) are detected at runtime. When auto-schema-evolution=true, IcebergTableManager automatically calls table.updateSchema().addColumn(...) to add new strongly-typed columns without table locking, downtime, or rewriting existing Parquet data files.
  • Flexible Table Routing Strategies:
    • UNIFIED (default): Routes all documents into a centralized enterprise lakehouse table (enterprise_documents).
    • PER_DOCUMENT_TYPE: Dynamically creates and routes documents into dedicated tables partitioned by document category (e.g., invoices, contracts, claims, cmis_document).
  • Optimized Partitioning Transforms: Configurable partition strategies through partition-by:
    • YEAR_MONTH (default): Automatic month transform on crawled_at (builder.month("crawled_at")), preventing small-file proliferation while maintaining fast range queries.
    • SOURCE_AND_DATE: Partitioned by source repository (source_type) and month of ingestion (crawled_at).
    • NONE: Unpartitioned tables for lightweight collections.
  • Columnar Parquet Storage & Compression: Writes native Parquet files using GenericParquetWriter with configurable compression codecs (zstd by default for superior compression and decompression throughput, or snappy, gzip, uncompressed).
  • S3FileIO Storage Layer: Bundled with iceberg-aws and AWS SDK v2, enabling direct streaming to Amazon S3, MinIO, Ceph, and Apache Ozone S3 Gateway with path-style access, custom endpoints, and explicit region credentials.
  • ACID Transactions & Snapshot Isolation: Writes are committed using Iceberg's atomic append API (table.newAppend().appendFile(dataFile).commit()), ensuring writers never block readers and analytical engines always query clean snapshots.
  • OIS Document Lifecycle & Tombstone Handling: Processes DocumentAction.DELETE events by recording explicit tombstone rows (action: DELETE) with snapshot audit trails, enabling downstream CDC and soft-deletion queries.

📋 Canonical Table Schema

The default schema provisioned by IcebergTableManager (CANONICAL_SCHEMA) defines the following fields:

Field ID Column Name Type Required Description
1 id string Yes Unique record UUID for this ingestion event
2 document_id string Yes Canonical repository document identifier
3 source_type string No Source repository connector type (e.g., sharepoint, alfresco, jdbc, web)
4 document_uri string No Original repository URI, file path, or locator
5 title string No Document title or filename
6 mime_type string No MIME content type (e.g., application/pdf, text/plain)
7 content_length long No Binary length in bytes
8 sha256_checksum string No SHA-256 cryptographic checksum of raw binary
9 claim_check_uri string No Object storage URI in Claim-Check store (s3://..., o3://...)
10 extracted_text string No Full-text extracted content (parsed via Apache Tika)
11 extracted_entities map<string, string> No Key-value dictionary of all extracted entities
12 custom_metadata map<string, string> No Key-value dictionary of repository-specific metadata
13 security_owner string No Primary document author or security owner
14 security_inheritance boolean No Whether folder/space permission inheritance is enabled
15 allowed_principals list<string> No Normalized allowed identities (u:<user>, g:<group>, role:<role>)
16 denied_principals list<string> No Normalized denied identities (explicit deny precedence)
17 action string Yes Open Ingestion Standard action (UPSERT or DELETE)
18 crawled_at timestamp Yes Crawl timestamp in UTC (used for partition transforms)
dyn * (evolved) string No Dynamically evolved entity columns (e.g., vendor, revenue, contract_id)

⚙️ Configuration Properties

All properties are configured under spring.opencrawling.output.iceberg.* (IcebergOutputProperties):

Property Environment Variable Default Description
spring.opencrawling.output.iceberg.catalog-type SPRING_OPENCRAWLING_OUTPUT_ICEBERG_CATALOG_TYPE in-memory Iceberg catalog type: rest, hive, hadoop, glue, jdbc, or in-memory
spring.opencrawling.output.iceberg.catalog-uri SPRING_OPENCRAWLING_OUTPUT_ICEBERG_CATALOG_URI "" URI for remote catalog service (e.g., http://localhost:8181 for REST, thrift://... for Hive)
spring.opencrawling.output.iceberg.catalog-name SPRING_OPENCRAWLING_OUTPUT_ICEBERG_CATALOG_NAME enterprise_lakehouse Logical catalog name identifier
spring.opencrawling.output.iceberg.warehouse SPRING_OPENCRAWLING_OUTPUT_ICEBERG_WAREHOUSE tmp/iceberg-warehouse Lakehouse root warehouse location URI (e.g., s3://warehouse, hdfs:///lakehouse)
spring.opencrawling.output.iceberg.namespace SPRING_OPENCRAWLING_OUTPUT_ICEBERG_NAMESPACE opencrawling_ecm Target Iceberg namespace / database name
spring.opencrawling.output.iceberg.table-name SPRING_OPENCRAWLING_OUTPUT_ICEBERG_TABLE_NAME enterprise_documents Target Iceberg table name (under UNIFIED table strategy)
spring.opencrawling.output.iceberg.table-strategy SPRING_OPENCRAWLING_OUTPUT_ICEBERG_TABLE_STRATEGY UNIFIED Table routing strategy: UNIFIED or PER_DOCUMENT_TYPE
spring.opencrawling.output.iceberg.auto-create-table SPRING_OPENCRAWLING_OUTPUT_ICEBERG_AUTO_CREATE_TABLE true Automatically create namespace and table on startup if missing
spring.opencrawling.output.iceberg.auto-schema-evolution SPRING_OPENCRAWLING_OUTPUT_ICEBERG_AUTO_SCHEMA_EVOLUTION true Automatically evolve table schema when new metadata entity fields are discovered
spring.opencrawling.output.iceberg.partition-by SPRING_OPENCRAWLING_OUTPUT_ICEBERG_PARTITION_BY YEAR_MONTH Partition strategy: YEAR_MONTH, SOURCE_AND_DATE, or NONE
spring.opencrawling.output.iceberg.write-format SPRING_OPENCRAWLING_OUTPUT_ICEBERG_WRITE_FORMAT parquet Data file format: parquet
spring.opencrawling.output.iceberg.compression-codec SPRING_OPENCRAWLING_OUTPUT_ICEBERG_COMPRESSION_CODEC zstd Parquet compression codec: zstd, snappy, gzip, or uncompressed
spring.opencrawling.output.iceberg.batch-size SPRING_OPENCRAWLING_OUTPUT_ICEBERG_BATCH_SIZE 500 Document batch threshold before flushing a Parquet data file
spring.opencrawling.output.iceberg.flush-interval-ms SPRING_OPENCRAWLING_OUTPUT_ICEBERG_FLUSH_INTERVAL_MS 5000 Flush interval in milliseconds for buffered documents
spring.opencrawling.output.iceberg.consumer-group SPRING_OPENCRAWLING_OUTPUT_ICEBERG_CONSUMER_GROUP opencrawling-iceberg-group Kafka consumer group name for IcebergStoreWriterConsumer
spring.opencrawling.output.iceberg.consumer-concurrency SPRING_OPENCRAWLING_OUTPUT_ICEBERG_CONSUMER_CONCURRENCY 3 Number of concurrent Kafka listener threads
spring.opencrawling.output.iceberg.io-impl SPRING_OPENCRAWLING_OUTPUT_ICEBERG_IO_IMPL "" Custom FileIO class (automatically defaults to org.apache.iceberg.aws.s3.S3FileIO when warehouse starts with s3://)
spring.opencrawling.output.iceberg.s3.endpoint SPRING_OPENCRAWLING_OUTPUT_ICEBERG_S3_ENDPOINT "" S3-compatible service endpoint URL (e.g., http://minio:9000)
spring.opencrawling.output.iceberg.s3.path-style-access SPRING_OPENCRAWLING_OUTPUT_ICEBERG_S3_PATH_STYLE_ACCESS true Enable path-style access for MinIO and custom S3 implementations
spring.opencrawling.output.iceberg.s3.access-key-id SPRING_OPENCRAWLING_OUTPUT_ICEBERG_S3_ACCESS_KEY_ID "" AWS / S3 access key ID
spring.opencrawling.output.iceberg.s3.secret-access-key SPRING_OPENCRAWLING_OUTPUT_ICEBERG_S3_SECRET_ACCESS_KEY "" AWS / S3 secret access key
spring.opencrawling.output.iceberg.s3.region SPRING_OPENCRAWLING_OUTPUT_ICEBERG_S3_REGION us-east-1 AWS / S3 region identifier
spring.opencrawling.output.iceberg.entity-extraction.enabled SPRING_OPENCRAWLING_OUTPUT_ICEBERG_ENTITY_EXTRACTION_ENABLED true Populate extracted_entities map and evaluate dynamic column additions
spring.opencrawling.output.iceberg.entity-extraction.include-custom-metadata SPRING_OPENCRAWLING_OUTPUT_ICEBERG_ENTITY_EXTRACTION_INCLUDE_CUSTOM_METADATA true Populate custom_metadata map with non-system repository attributes
spring.opencrawling.output.iceberg.entity-extraction.include-ois-security SPRING_OPENCRAWLING_OUTPUT_ICEBERG_ENTITY_EXTRACTION_INCLUDE_OIS_SECURITY true Populate Zero-Trust ACL columns (security_owner, allowed_principals, denied_principals)

🐳 Decoupled Docker Compose Architecture

OpenCrawling provides a turnkey decoupled Docker Compose configuration specifically tailored for Iceberg lakehouse streaming under oc-iceberg-output-connector/docker/docker-compose-decoupled-with-iceberg.yml:

services:
  # S3 Object Storage Backend
  minio:
    image: milvusdb/minio:RELEASE.2024-12-18T13-15-44Z
    container_name: minio-decoupled-iceberg
    environment:
      MINIO_ROOT_USER: admin
      MINIO_ROOT_PASSWORD: password
    ports:
      - "9000:9000"
      - "9001:9001"

  # Apache Iceberg REST Catalog Fixture
  iceberg-rest:
    image: apache/iceberg-rest-fixture:latest
    container_name: iceberg-rest-decoupled
    ports:
      - "8181:8181"
    environment:
      - AWS_ACCESS_KEY_ID=admin
      - AWS_SECRET_ACCESS_KEY=password
      - AWS_REGION=us-east-1
      - CATALOG_WAREHOUSE=s3://warehouse/
      - CATALOG_IO__IMPL=org.apache.iceberg.aws.s3.S3FileIO
      - CATALOG_S3_ENDPOINT=http://minio:9000
      - CATALOG_S3_PATH__STYLE__ACCESS=true
    depends_on:
      minio:
        condition: service_healthy

  # Decoupled Writer Consumer writing to Apache Iceberg
  oc-writer-consumer:
    image: ghcr.io/opencrawling/oc-writer-consumer:latest
    container_name: oc-writer-service-iceberg
    environment:
      - SPRING_KAFKA_BOOTSTRAP_SERVERS=kafka:9094
      - OPENCRAWLING_CONSUMER_WRITER_ENABLED=true
      - SPRING_OPENCRAWLING_OUTPUT_TYPE=iceberg
      - SPRING_OPENCRAWLING_OUTPUT_ICEBERG_CATALOG_TYPE=rest
      - SPRING_OPENCRAWLING_OUTPUT_ICEBERG_CATALOG_NAME=opencrawling_lakehouse
      - SPRING_OPENCRAWLING_OUTPUT_ICEBERG_CATALOG_URI=http://iceberg-rest:8181
      - SPRING_OPENCRAWLING_OUTPUT_ICEBERG_WAREHOUSE=s3://warehouse
      - SPRING_OPENCRAWLING_OUTPUT_ICEBERG_IO_IMPL=org.apache.iceberg.aws.s3.S3FileIO
      - SPRING_OPENCRAWLING_OUTPUT_ICEBERG_NAMESPACE=enterprise_data
      - SPRING_OPENCRAWLING_OUTPUT_ICEBERG_TABLE_NAME=documents
      - SPRING_OPENCRAWLING_OUTPUT_ICEBERG_S3_ENDPOINT=http://minio:9000
      - SPRING_OPENCRAWLING_OUTPUT_ICEBERG_S3_ACCESS_KEY_ID=admin
      - SPRING_OPENCRAWLING_OUTPUT_ICEBERG_S3_SECRET_ACCESS_KEY=password
      - SPRING_OPENCRAWLING_OUTPUT_ICEBERG_AUTO_SCHEMA_EVOLUTION=true

🖥️ Web Admin UI Configuration

The Apache Iceberg Output Connector is fully integrated into the OpenCrawling Web Admin UI (http://localhost:3000):

  1. Navigate to Connectors → Create Connector.
  2. Select Output Connector and choose "Apache Iceberg Lakehouse" (org.opencrawling.iceberg.output.IcebergOutputConnector) from the dropdown.
  3. Configure the lakehouse parameters directly via dedicated form controls:
    • Catalog Type: REST, HIVE, HADOOP, GLUE, JDBC, or IN_MEMORY.
    • Catalog URI: Endpoint of the catalog (e.g. http://iceberg-rest:8181).
    • Catalog Name: Target catalog name (e.g. opencrawling_lakehouse).
    • Warehouse Location: Storage URI (e.g. s3://warehouse).
    • Target Namespace & Table: E.g. enterprise_data.documents.
    • Table Strategy: CREATE_IF_NOT_EXISTS, FAIL_IF_NOT_EXISTS, or RECREATE.
    • Partition Strategy: DAY, MONTH, YEAR, MIME_TYPE, or NONE.
    • Write Format & Compression: Parquet with Snappy, Gzip, or ZSTD compression.
    • Schema Evolution: Toggle automated column addition for dynamic document entities.
    • S3 / Object Store Credentials: S3 endpoint, region, access key, secret key, and path-style access.
  4. Save and test connector connectivity.

🧪 Testing & Verification

1. Standalone Connector & REST Catalog Test

OpenCrawling includes an automated integration test script (scripts/test-iceberg-output-connector.sh) that verifies catalog connectivity, table creation, Zero-Trust OIS ACL schema fields, and dynamic schema evolution:

# Make executable and run
chmod +x scripts/test-iceberg-output-connector.sh
./scripts/test-iceberg-output-connector.sh

What the script validates:

  1. Checks for curl, jq, and docker utilities.
  2. Checks Iceberg REST Catalog (http://localhost:8181/v1/config) and spins up the fixture if offline.
  3. Automatically ensures the MinIO warehouse bucket exists.
  4. Ensures the target namespace exists (POST /v1/namespaces).
  5. Provisions an Iceberg table with canonical fields and verifies Zero-Trust ACL columns (security_allowed_read, security_denied_read).
  6. Tests dynamic schema evolution by posting schema updates that add new structured entity fields (entity_vendor, entity_amount) and confirms the table upgrades to schema-id: 1.

2. End-to-End Decoupled Pipeline Integration Test

To test the entire multi-service ingestion pipeline (crawler → Kafka → ingestion consumer → writer consumer → Iceberg Parquet files on MinIO):

chmod +x scripts/test-iceberg-output-decoupled.sh

# Standard execution
./scripts/test-iceberg-output-decoupled.sh

# Local BuildKit acceleration with shared ~/.m2 cache mount
./scripts/test-iceberg-output-decoupled.sh --local

# Fast iteration skipping Docker builds and preserving containers on failure
./scripts/test-iceberg-output-decoupled.sh -s -k

What the decoupled test validates:

  1. Builds and starts all microservices (oc-crawler, oc-ingestion-consumer, oc-writer-consumer, iceberg-rest, minio, kafka, redis).
  2. Creates a human-readable enterprise contract document containing metadata, entities, and Zero-Trust ACLs (role:lakehouse-analytics, user:piergiorgio@apache.org, group:data-engineering, and explicit deny role:external-guest).
  3. Executes the crawl, streaming the document across Kafka to the Iceberg writer consumer.
  4. Polls the Iceberg REST Catalog until snapshot commits are confirmed.
  5. Verifies the generated Parquet files in MinIO storage (mc ls local/warehouse/).
  6. Confirms Zero-Trust OIS security columns are present in the table schema.

3. Apache Iceberg MCP Server & OIS Sidecar Integration Test (#302)

Verify Apache Iceberg snapshot lineage, point-in-time getIcebergDocumentAsOf time-travel (with Zero-Trust ACL revocation enforcement), queryIcebergEntities, getIcebergIngestionStats, IcebergSidecarStore (listSidecarStores, readOisSidecars), and IcebergMcpSearchBackend over MCP Streamable HTTP:

chmod +x scripts/test-mcp-iceberg.sh
./scripts/test-mcp-iceberg.sh --local-build

📊 Querying Ingested Data

Once persisted into Apache Iceberg, data can be queried across various modern analytical engines without data movement:

DuckDB (In-Process OLAP)

INSTALL iceberg;
LOAD iceberg;

SELECT document_id, title, mime_type, security_owner, allowed_principals, entity_vendor
FROM iceberg_scan('s3://warehouse/enterprise_data/documents',
                  endpoint='http://localhost:9000',
                  access_key_id='admin',
                  secret_access_key='password')
WHERE 'role:lakehouse-analytics' = ANY(allowed_principals);

Trino / Starburst

SELECT document_id, title, action, crawled_at
FROM iceberg.enterprise_data.documents
WHERE contains(allowed_principals, 'user:piergiorgio@apache.org')
  AND NOT contains(denied_principals, 'role:external-guest');

Apache Spark SQL

SELECT document_id, title, extracted_entities['vendor'] AS vendor, crawled_at
FROM enterprise_lakehouse.enterprise_data.documents
WHERE crawled_at >= current_date() - INTERVAL 30 DAYS;

🗺️ Documentation


🎓 Tutorials


🧪 Testing


🤝 Community

Clone this wiki locally