[SUPPORT] Change Data Capture(CDC) for hudi

[melin]

https://docs.google.com/document/d/1bN6rdLNcYOHnT3xVBfB33BoiPO06aKBo56SZmuU9pnY/edit#heading=h.czd6l3c9xj87

[danny0405]

Thanks for the reporting @melin , actually hoodie already implemented this feature already since release 0.9, we add a metadata field named __hoodie_operation to record the per-record change flag. And hoodie can consume CDC and be read a CDC source both through Flink engine already. The spark engine should follow in this feature soon.

But like you said, we should have a more general solution, let's say hoodie would support CDC logs not only for CDC source, but appending logs too, the RFC is coming soon ~

[melin]

Spark PR： https://issues.apache.org/jira/browse/HUDI-2941
See databricks delta support for Spark SQL：https://docs.databricks.com/delta/delta-change-data-feed.html

-- version as ints or longs e.g. changes from version 0 to 10
SELECT * FROM table_changes('tableName', 0, 10)

-- timestamp as string formatted timestamps
SELECT * FROM table_changes('tableName', '2021-04-21 05:45:46', '2021-05-21 12:00:00')

-- providing only the startingVersion/timestamp
SELECT * FROM table_changes('tableName', 0)

-- database/schema names inside the string for table name, with backticks for escaping dots and special characters
SELECT * FROM table_changes('dbName.`dotted.tableName`', '2021-04-21 06:45:46' , '2021-05-21 12:00:00')

-- path based tables
SELECT * FROM table_changes_by_path('\path', '2021-04-21 05:45:46')

[nsivabalan]

@xushiyan to also follow up.

[nsivabalan]

Hey @YannByron, can you follow up on this. Feel free to file a jira ticket on the ask and take it from there.

[YannByron]

Hey @YannByron, can you follow up on this. Feel free to file a jira ticket on the ask and take it from there.

sure. https://issues.apache.org/jira/browse/HUDI-3478 to this issue.