[Feature Request] Reading and Preserving Row Tracking information

[johanl-db]

Feature request

Which Delta project/connector is this regarding?

• Spark
• Standalone
• Flink
• Kernel
• Other (fill in here)

Overview

This feature request tracks the remaining effort to fully implement Row Tracking.
The feature required changes in Spark that are being released with Spark 3.5. With Spark 3.5 being released, fully implementing the feature is now unblocked.

Further details

Row tracking specification was added with 8dfa534 and #1747. See Row ID design document for the initial design.

Support for writing fresh row IDs / row commit versions was introduced in the following PRs and didn't require any changes in Spark:

• Assign fresh Row IDs during commit #1723
• Add Default Row Commit Version to AddFile and RemoveFile #1781
• Assign materialized Row ID and Row commit version column names #1896

Reading and preserving row IDs / row commit versions on the other does require changes that are shipping with Spark 3.5 (see e.g. apache/spark#40677 and apache/spark#40545). With Spark 3.5 we can now implement reading and preserving row tracking information

Willingness to contribute

The Delta Lake Community encourages new feature contributions. Would you or another member of your organization be willing to contribute an implementation of this feature?

• Yes. I can contribute this feature independently.
• Yes. I would be willing to contribute this feature with guidance from the Delta Lake community.
• No. I cannot contribute this feature at this time.

[johanl-db]

All the changes that allow reading and preserving row tracking information are in 3.2 thanks to @longvu-db!
Some of the more relevant PRs part of that effort:
Read support:

• [Spark] Add read support for RowId #2856
• [Spark] Add Read support for RowCommitVersion and Row Tracking Preservation in Read/Write #2878

Preserving row tracking information:

• [Spark] Add Preserving Row Tracking in Delete #2925
• [Spark] Add Preserving Row Tracking in Update #2927
• [Spark] Add Preserving Row Tracking in Merge #2936