[HUDI-5953][DNM] Handle duplicate records in HoodieCreateHandle

[codope]

Change Logs

Handle potential duplicates in HoodieCreateHandle. It is already handled for compaction.
DO NOT MERGE. Yet to cleanup tests.

Impact

No public API but critical change. Will affect inserts.

Risk level (write none, low medium or high below)

high

Documentation Update

Describe any necessary documentation update if there is any new feature, config, or user-facing change

• The config description must be updated if new configs are added or the default value of the configs are changed
• Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
  ticket number here and follow the instruction to make
  changes to the website.

Contributor's checklist

• Read through contributor's guide
• Change Logs and Impact were stated clearly
• Adequate tests were added if applicable
• CI passed

[codope]

@hudi-bot run azure

[danny0405]

-1, don't think we should go in this direction, for HoodieCreateHandle, a premise is that the input data set should already be decuplicated, there are two impacts I could think of here:

1. Introduces unnecessary overhead for many workflows, like bulk_insert, insert, compaction, clustering, etc.
2. the cache of the keys could results in OOM exception, it is more safe the engine handles the deduplication job

[hudi-bot]

CI report:

• 1e2111d Azure: FAILURE

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build

[codope]

Closing it as the fix landed via cabcb2b