[to be discussed] Users should be able to configure HUDI to attempt lazy failed writes rollback and/or clean before the startCommit

[kbuci]

Task Description

What needs to be done:

Add a new policy for hoodie.clean.failed.writes.policy`

• LAZY_WITH_PREWRITE : Same as LAZY, except that rollbackFailedWrites will also be attempted in startCommit before creating the new instant

Add a new clean config that, if enabled, attempts a clean in startCommit before creating the new instant

Why this task is needed:

• We have had cases where write ingestion jobs repeatedly fail. Because we use LAZY policy, files from these failed writes were not implicitly rolled back between these ingestion jobs. This leads to a buildup of files in DFS partition directory and increase unbounded. And can cause them to hit file quota limits. To mitigate we had to manually run clean jobs or temporarily set policy to EAGER
• We want to handle potential future cases where write jobs make keep on repeatedly failing after completing the commit but before attempting post-commit clean, which can result in the same impact as above. For example, if there are infra issues like spark jobs failing/timing out when the HUDI write reaches the post-commit phase.

We have added this functionality to our internal 0.x HUDI build to address these issues, we can upstream once we achieve consensus.

Task Type

Code improvement/refactoring

Related Issues

Parent feature issue: (if applicable )
Related issues:
NOTE: Use Relationships button to add parent/blocking issues after issue is created.

[nsivabalan]

Should we try moving rollbacks out of clean path.
Essentially, rollbackFailedWrites warrants a separate table service in itself.

We could call it across many places.

Single writer:
a. just before starting a new commit.

Multi-writer:

1. just before starting a new commit optionally.
2. post commit -> clean, rollbackFailedWrites, archive.
3. Separate async table service.

Now coming to your proposal:
hoodie.clean.failed.writes.policy is meant to dictate how the clean code path should handle failed writes. I think we should stick to just 2 values in here (EAGER, LAZY), since there are only two modes, single writer or not.

may be we could introduce another writer config to force the rollback failed writes before starting a new commit
hoodie.write.rollback.failed.writes.pre.write.enable

We might have to fix

hudi/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java

Line 1014 in eb1d772

CleanerUtils.rollbackFailedWrites(config.getFailedWritesCleanPolicy(),

piece of code to accommodate this.

[nsivabalan]

Making rollbackFailedWrites as a separate table service is a good follow up. but we can de-couple that from immediate deliverable.