Repository navigation
Replies: 1 comment 3 replies
|
Hey @kyo-tom, having the ability to read like this does make sense to me. In terms of API, it makes sense, but I would personally keep it under the Also, it would be helpful is you could share the expected output column datatypes. |
3 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Feature summary
I would like to add a lazy
daft.read_iceberg_changes()API for reading row-level changes between two snapshots of an Apache Iceberg copy-on-write table. The proposed semantics follow Iceberg Spark'screate_changelog_view, including carry-over removal, optional update pairing, and net changes. This would allow Daft users to build incremental pipelines from Iceberg snapshot history without requiring Spark.Motivation
daft.read_iceberg()reads the state of a table at a snapshot, but it does not expose the rows that changed between snapshots.For copy-on-write tables, the manifests contain enough information to construct a changelog:
INSERTevents;DELETEevents;This is useful for incremental feature pipelines, downstream index updates, audit streams, and synchronizing Iceberg tables into external systems.
Proposed API
The returned DataFrame contains the table columns plus:
Snapshot IDs and UTC millisecond timestamps can be used as range boundaries. The start boundary is exclusive and the end boundary is inclusive, matching Iceberg's changelog procedure.
Compute update images
Matching
DELETE/INSERTevents are exposed asUPDATE_BEFORE/UPDATE_AFTER. Ifidentifier_columnsis omitted, the identifier fields from the end snapshot's schema are used.Net changes
This removes intermediate changes across the complete range and returns only the remaining net
INSERT/DELETEevents.net_changesandcompute_updatesare mutually exclusive, matching Iceberg's API.Copy-on-write behavior
A copy-on-write update may report an entire replaced file as deleted and its replacement as inserted. Rows whose values did not change are carry-over rows rather than real table changes.
The default mode removes these carry-over pairs. For example, if a file contains Alice, Bob, and Carol and only Bob is updated, the unchanged Alice and Carol delete/insert pairs are removed.
compute_updates=Truecan then pair Bob's remainingDELETEandINSERTas update images.net_changes=Trueinstead performs net cancellation across the whole requested range.Schema evolution
The output schema is the schema associated with the end snapshot.
Historical Parquet files are reconciled by Iceberg field ID rather than by column name, including nested
struct,list, andmapfields. This supports compatible changes such as:Dropping a field and later adding a new field with the same name does not reconnect the old data: the new field has a different field ID, so old rows correctly read it as null.
Initial scope and fail-closed behavior
The initial implementation is intentionally conservative:
initial_defaultare rejected because Daft does not currently materialize that default value;read_iceberg_changes()requires PyIceberg 0.11 or newer for the schema compatibility API it uses.The operation remains lazy with respect to manifest and Parquet-footer I/O: those are planned when scan tasks are materialized, not while the DataFrame is constructed.
Prototype and validation
I have a working implementation and have validated it using:
SqlCatalogtables and real Parquet files;struct,list, andmap;tests/io/iceberg/regression tests.I can provide the detailed design and test matrix if the overall direction is accepted.
Questions for maintainers
daft.read_iceberg_changes()look like the right API name and location, or would you prefer a different interface?compute_updates,net_changes, and schema evolution?Thanks — I would appreciate feedback on the API and initial scope before preparing the contribution for review.
All reactions