Interoperability in OLake Go

[siddharth-chevella]

Proposal: Improving Apache Iceberg Interoperability in OLake Go

Hey Community👋

We're exploring a new approach in OLake Go to address one of the biggest challenges in the Iceberg ecosystem today: interoperability across catalogs and query engines.

Different catalogs and query engines support different Iceberg features and delete mechanisms. As a result, users often have to design their ingestion architecture around the limitations of their downstream systems rather than their actual requirements.

Our goal is simple:

Allow users to ingest data into Iceberg while choosing the delete format that best fits the catalogs and query engines they plan to use.

Current Thinking

We're considering an architecture where all upsert workloads are first written to an internal changelog table and then materialized into the main Iceberg table.

The changelog table would be internally managed by OLake Go and remain invisible to users.

The main table could then be written using one of the following delete mechanisms:

• Equality Deletes
• Position Deletes
• Deletion Vectors

High-level flow:

Source
   ↓
Changelog Table
   ↓
Main Table

Append-only workloads would continue to write directly to the main table.

Why We're Considering This

This approach gives us a common ingestion path while allowing OLake Go to generate the appropriate delete format based on user requirements and downstream compatibility constraints.

Potential examples:

• A catalog may support Position Deletes but not Equality Deletes (Example: Snowflake's Horizon Catalog).
• A query engine may support Equality Deletes but not Deletion Vectors (Example: Clickhouse).
• Users may prefer one delete type over another due to performance or compatibility considerations.

Things We're Evaluating

Performance

Generating Position Deletes or Deletion Vectors introduces additional processing overhead compared to direct writes.

Questions we're exploring:

• What performance trade-offs are acceptable?
• Are there optimization techniques we should consider?
• Have others implemented similar conversion pipelines?

Resource Consumption

Converting changelog records into Position Deletes or Deletion Vectors can be resource intensive.

Current thinking is to expose a concurrency setting that controls how many tables can perform this conversion simultaneously.

We're interested in hearing how others have approached resource management for similar workloads.

Scalability

The initial implementation would run within a standalone OLake Go deployment.

This means the approach is primarily vertically scalable today.

Longer term, we may explore distributed execution models (Kubernetes workers, Spark-based execution, etc.) if the community believes that is necessary for large-scale deployments.

Questions for the Community

We would appreciate feedback on the following:

1. Does the changelog → materialization approach seem reasonable?
2. Have you implemented a similar interoperability layer for Iceberg?
3. What challenges should we expect when supporting Equality Deletes, Position Deletes, and Deletion Vectors from a single ingestion workflow?
4. Are there existing projects or designs we should review before moving forward?
5. Would you prioritize catalog compatibility, query engine compatibility, or both?
6. Are there scalability concerns with this approach that we may be overlooking?
7. If you were building this today, would you take a different approach?

We're still in the design phase and would greatly appreciate feedback, concerns, alternative approaches, and lessons learned from anyone operating Iceberg at scale.