DatahubSyncTool does not correctly re-ingest soft-deleted entities

[hudi-bot]

Datahub has a notion of soft-deletes (the entity still exists in the database with a status=removed:true). Such entities could get re-ingested with new properties at a later time, such that the older one gets overwritten. The current implementation in DatahubSyncTool does not handle this scenario. It fails to update the status flag to removed:false during ingest, which means the entity won't surface in the Datahub UI at all.

Ref: See sections on Soft Delete and Hard Delete in the Datahub docs: [https://datahubproject.io/docs/how/delete-metadata/#soft-delete-the-default]

JIRA info

• Link: https://issues.apache.org/jira/browse/HUDI-4994
• Type: Task

[hudi-bot]

Linked PR(s)

• [HUDI-4994] Fix bug that prevents re-ingestion of soft-deleted Datahub entities #6886