[Issue]: Hudi spill path not getting cleared when containers getting killed abruptly.

[Prasheel3001]

Tips before filing an issue

• Have you gone through our FAQs?

• Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.

• If you have triaged this as a bug, then file an issue directly.

Describe the problem you faced
When yarn kills the containers abruptly for any reason while hudi stage is in progress then the spill path created by hudi on the disk is not cleaned and as a result of which the nodes on the cluster start running out of space. We need to clear the spill path manually to free out disk space.

To Reproduce

Steps to reproduce the behavior:
Problem would be reproducible if hudi stage fails with containers getting killed abruptly.

Expected behavior

Hudi should clear the spill path even if the containers are killed abruptly by yarn for any reason.

Environment Description

• Hudi version : 0.7.0

• Spark version : 2.4.6

• Hive version :

• Hadoop version :

• Storage (HDFS/S3/GCS..) : HDFS

• Running on Docker? (yes/no) : no

[nsivabalan]

thanks for reporting. have filed a tracking jira. Let us know if you are interested to work on it.