[HUDI-7240] Clean delete logic

[linliu-code]

Change Logs

1. When we create HoodieRecord for a delete, we store the necessary information into the metadata field.
2. When we need to merge delete records, we extract orderingVal from metadata field of HoodieRecord.
3. Removed HoodieRecordTestPayload.

Impact

Simplifies the logic for handling delete records.

Risk level (write none, low medium or high below)

Low.

Contributor's checklist

• Read through contributor's guide
• Change Logs and Impact were stated clearly
• Adequate tests were added if applicable
• CI passed

[linliu-code]

@yihua @codope @danny0405

[danny0405]

The data is designed to be non-null, we even have some validation logic for it in HoodieRecord:

public T getData() {
    if (data == null) {
      throw new IllegalStateException("Payload already deflated for record.");
    }
    return data;
  }

[linliu-code]

We can check the metadata first in the validation.

[linliu-code]

Will clean the failures.

[hudi-bot]

CI report:

• 3d71d1c Azure: FAILURE

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build

[danny0405]

Thanks for raising this fix, I think it is a good chance we fix the event time sequence comparison of delete records with payloads, I can see 2 mistaks in our code that uses processing time sequence for deletes:

1. OverwriteWithLatestAvroPayload#preCombine:

  public OverwriteWithLatestAvroPayload preCombine(OverwriteWithLatestAvroPayload oldValue) {
    if (oldValue.recordBytes.length == 0) {
      // use natural order for delete record
      return this;
    }
    ...
  }

2. DefaultHoodieRecordPayload#combineAndGetUpdateValue

  public Option<IndexedRecord> combineAndGetUpdateValue(IndexedRecord currentValue, Schema schema, Properties properties) throws IOException {
    if (recordBytes.length == 0) {
      return Option.empty();
    }

    ...
  }

In any case, the orderingVal should be set up correctly and we should utilize it as much as possible.

[yihua]

Closing this PR as we should revisit the delete ordering overall.