[HUDI-5238] Fixing HoodieMergeHandle shutdown sequence

[alexeykudinkin]

Change Logs

This PR addresses the #7234 related to HoodieMergeHandle shutdown sequence:
in introduced at #4264 we changed the ordering in which we shut down the handle relative to the executor:

Before it was

1. Handle
2. Executor

After

1. Executor
2. Handle

The reason it was switched was to handle the case when during exception thrown executor might still be writing out records, and closing of the handle (before the executor) was leaving some of the produced Parquet files corrupted.

This PR, addresses this issue by making sure that in the successful path we close the Handle immediately as soon as writing has finished (before we shutdown the executor), which would make sure this will not result in any PipeBroken exceptions in GCS

Impact

No impact

Risk level (write none, low medium or high below)

Low

Documentation Update

N/A

Contributor's checklist

• Read through contributor's guide
• Change Logs and Impact were stated clearly
• Adequate tests were added if applicable
• CI passed

[hudi-bot]

CI report:

• 10b74fc Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build