Clustering w/ two multi-writers could lead to issues

[hudi-bot]

if two writers have enabled clustering, each could rollback the clustering that the other writer is currently executing and could lead to unrecoverable issues. 

 

 
{code:java}
t1 t2
--------------------------------------------------------➝ t
writer1 |-----------------------------|
writer2 |--------------| {code}
lets say writer1 starts a clustering at t1. 

and then writer2 starts clustering at time t2. at this time, it will rollback the clustering started at time t1. but writer 1 could still be continuing to execute the clustering. 

 

JIRA info

• Link: https://issues.apache.org/jira/browse/HUDI-5174
• Type: Bug

---

Comments

08/Dec/22 23:02;haox;Hi do we have any updates on this issue?;;;

[rangareddy]

This issue was reviewed as part of the JIRA-migrated backlog triage.

Findings: this describes a potential concurrent-clustering rollback race - a second writer rolling back the first writer's in-flight clustering. We could not reproduce it or pin a specific fix commit. The standard multi-writer guard (HoodieFailedWritesCleaningPolicy=LAZY + heartbeat-based expiry) is expected to prevent one writer from rolling back another's in-progress operation, but we have not confirmed it fully covers in-flight clustering.

If you can still hit this on a recent version (1.x), please share your multi-writer setup (lock provider, cleaning policy, whether async clustering) and repro steps so we can validate.