[SUPPORT] hudi mor table on Spark upsert may have "small files" problems

[Doorwood]

Hi!~When i am reading hudi code about getting small files on hoodie spark upsert,one question bothers me a lot.

If I use hoodie mor table written by spark,as long as I update one record,it will create one log file for this fileslice to record this update operation.After this update operation,i insert a new record,it needs to find a small file to write,but the fileslice who has log files can not be treated as small file,so that hoodie may create a new filegroup to hold this record.What i want u understand,it may create a lot of small file with a lot of file groups.
I have tested the question i described,and actually it causes "small files" problems.

I am looking forward to someone can help me solve this problem!

[yihua]

@Doorwood For MOR table, the compaction is scheduled and executed every few commits to compact the base and log files in the file slice to a new base file, so that it can be considered for file sizing optimization like small file handling and clustering later on. Here's how you can run compaction for a MOR table. After compaction, you can also run clustering to auto-size the data files.

[Doorwood]

@yihua Thanks for your reply,I have tried your suggestions.This problem seems solved.