Rollback inflight indexing does not clean table config inflight MDT partition

[hudi-bot]

steps:

latest hudi master branch (as of aug 24th)

create index on table inflight and writer crash

run cleaner table service. instants are rolled back, but inflight MDT partition in table config is not updated accordingly.

Steps to reproduce:

• Start async index building for a decent sized data table. (say rli). We can use HoodieIndexer for this purpose.
• Crash the index building process mid-way. Atleast we should have the requested and inllight in the timeline. or atelast the requested.
• hoodie.properties will have a reference to the "rli" as part of hoodie.properties (hoodie.table.metadata.partitions.inflight).
• Run HoodieIndexer to drop the RLI index which failed mid-way.
• hoodie.properties missed to clean up the entry for "hoodie.table.metadata.partitions.inflight".

Expected behavior:
once rollback or dropping of index command completes, we should not see the rli as part of "hoodie.table.metadata.partitions.inflight" in hoodie.properties

JIRA info

• Link: https://issues.apache.org/jira/browse/HUDI-9740
• Type: Bug
• Fix version(s):
  • 1.2.0

[rangareddy]

This issue was reviewed as part of the JIRA-migrated backlog triage (HUDI-9740).

Findings: needs a reproduction against a current build before it can be dispositioned.

The report is concrete: after an async index build (e.g. RLI via HoodieIndexer) is crashed mid-way and the cleaner rolls the instants back, hoodie.properties still lists the partition as an inflight MDT partition. Reproducing it looks tractable - the description gives an ordered recipe - start async index building on a decent-sized table, crash it once requested/inflight exist, run the cleaner, then inspect hoodie.properties.

I have not called this from the source. The surrounding code has been rewritten substantially since this was filed - table versions 8 and 9, the BufferedRecordMerger family replacing payload-based merging, the file-group-reader migration, the indexer abstraction in #18348 / #18372, and the hudi-common package moves in #19193 / #19195. Inferring either "still broken" or "silently fixed" through that much churn would be a guess.

There is also no commit anywhere in the tree referencing HUDI-9740, so if it was fixed it was incidental rather than deliberate - another reason to verify rather than assume.

The consequence is worth stating on the ticket: a stale inflight entry makes the table look mid-index forever, which can block a later index build or confuse readers. Mirror image of #17208 (HUDI-9802), where the inflight set is empty when a partition is expected - both are the inflight MDT partition list drifting from the timeline, and a single owner for that invariant would fix both.

Keeping this open.