[HUDI-6594] Support duplicate with HFile Reader/Writer

[codope]

Change Logs

Support duplicate with HFile Reader/Writer and MDT record index.

• HoodieAvroHFileReader#getRecordByKeyIteratorInternal to return an Iterator<IndexedRecord> for given keys.
• HoodieTableMetadata API changes - new method that returns a list of HoodieRecords:

/**
   * Fetch records for given keys. A key could have multiple records associated with it. This method returns all the records for given keys.
   *
   * @param keys          list of key for which interested records are looked up for.
   * @param partitionName partition name in metadata table where the records are looked up for.
   * @return Map of key to {@link List} of {@link HoodieRecord}s with records matching the passed in keys.
   */
  Map<String, List<HoodieRecord<HoodieMetadataPayload>>> getAllRecordsByKeys(List<String> keys, String partitionName);

• Added tests for record index and HFile reader.

Impact

MDT can have duplcates. Useful for record index to work as regular non-global index.

Risk level (write none, low medium or high below)

medium

Documentation Update

Describe any necessary documentation update if there is any new feature, config, or user-facing change

• The config description must be updated if new configs are added or the default value of the configs are changed
• Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
  ticket number here and follow the instruction to make
  changes to the website.

Contributor's checklist

• Read through contributor's guide
• Change Logs and Impact were stated clearly
• Adequate tests were added if applicable
• CI passed

[codope]

@nsivabalan I added a separate method (yet to fully implement) for separation as we will have duplicates only in case of RLI for now. But, should we chnage the return type existing getRecordsByKeys?

[codope]

this will be reverted after the new BaseTableMetadata#getAllRecordsByKeys is implemented.

[hudi-bot]

CI report:

• 94ad9c1 Azure: FAILURE

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build

[nsivabalan]

guess we don't need this anymore. feel free to re-open or create a new one if need be