Support Newer Delta Protocol Versions in Delta to Iceberg Migration Using Delta Kernel #15420
Description
Feature Request / Improvement
The current Delta-to-Iceberg migration implementation in iceberg-delta-lake relies on Delta Standalone, which only supports older Delta protocol versions (minReaderVersion = 1, minWriterVersion = 2) and does not support newer protocol versions (for example, 3,7).
Running the migration on newer Delta tables fails with the following error:
Exception in thread "main" io.delta.standalone.internal.exception.DeltaErrors$InvalidProtocolVersionException:
Delta protocol version (3,7) is too new for this version of Delta
Standalone Reader/Writer (1,2). Please upgrade to a newer release.
Since Delta Standalone is deprecated and not up to date with the latest protocol changes (delta-io/delta#2243 (comment)), Delta Kernel APIs should be used for connector integrations instead.
Query engine
None
Willingness to contribute
- I can contribute this improvement/feature independently
- I would be willing to contribute this improvement/feature with guidance from the Iceberg community
- I cannot contribute this improvement/feature at this time