Skip to content

[codex] Implement local Iceberg accepted sink MVP#162

Merged
malon64 merged 2 commits intomainfrom
feature/iceberg-feature
Feb 24, 2026
Merged

[codex] Implement local Iceberg accepted sink MVP#162
malon64 merged 2 commits intomainfrom
feature/iceberg-feature

Conversation

@malon64
Copy link
Copy Markdown
Owner

@malon64 malon64 commented Feb 24, 2026

Refs #70

What Was Implemented

This PR implements the local filesystem MVP for the Iceberg accepted sink in Floe.

Core changes:

  • Added sink.accepted.format: iceberg support to the accepted sink adapter registry and config validation.
  • Implemented a local Iceberg writer adapter in floe-core that writes a valid Iceberg table layout under the configured local table root (metadata/, data/).
  • Added append and overwrite handling through the existing sink.write_mode flow and accepted sink adapter architecture.
  • Added scalar-type mapping from Floe/Polars output columns to a supported Iceberg scalar subset.
  • Added report fields for accepted output metadata (write_mode, files_written, table_root_uri, snapshot_id when available) while keeping existing report fields backward-compatible.
  • Updated docs/support matrix and Iceberg sink docs for the local MVP scope.

Behavior notes:

  • append loads/registers the existing local Iceberg table metadata and commits a new snapshot.
  • overwrite recreates table metadata at the same table root and writes new logical contents (no cleanup/GC of old files/metadata).
  • Empty accepted dataframe overwrite/create initializes table metadata without writing data files or creating a snapshot.

Supported Scope / Explicit Non-Goals (Issue #70)

Supported in this PR:

  • Accepted sink format: iceberg
  • Local storage only
  • write_mode: append, overwrite
  • Scalar columns only (strings, booleans, signed ints, floats, date/time/datetime subset)
  • Report metadata for Iceberg writes

Explicit non-goals in this PR:

  • S3 / GCS / Glue catalogs
  • Schema evolution
  • Merge/upsert semantics
  • Cleanup / garbage collection of orphaned files/metadata
  • Advanced Iceberg table features beyond local unpartitioned MVP behavior

Test Coverage Summary

Added / updated tests:

  • Local run() integration test that writes an Iceberg table on a temp dir and validates table layout + report fields.
  • Unit tests for Iceberg writer append snapshot/version progression.
  • Unit tests for overwrite logical replacement behavior.
  • Unit tests for unsupported dtype errors (clear message).
  • Unit tests for empty dataframe behavior.
  • Config validation tests for local Iceberg acceptance and non-local Iceberg rejection.
  • Updated existing report/CLI fixture tests for new accepted-output report fields.

Validation run in this branch:

  • cargo fmt --all
  • cargo clippy --all-targets --all-features -- -D warnings
  • cargo test --all

Follow-up Notes (#71 / #69 / #157)

@malon64 malon64 changed the title Implement local Iceberg accepted sink MVP [codex] Implement local Iceberg accepted sink MVP Feb 24, 2026
@malon64 malon64 marked this pull request as ready for review February 24, 2026 13:57
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9dc6d6a7a8

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread crates/floe-core/src/io/write/iceberg.rs
Comment thread crates/floe-core/src/io/write/iceberg.rs Outdated
@malon64 malon64 force-pushed the feature/iceberg-feature branch from 9dc6d6a to da0d83b Compare February 24, 2026 14:13
@malon64 malon64 merged commit a743d91 into main Feb 24, 2026
3 checks passed
@malon64 malon64 deleted the feature/iceberg-feature branch February 24, 2026 14:23
@malon64 malon64 mentioned this pull request Feb 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant