[SUPPORT]What is the functionality that hudi cleaner provides.

[bithw1]

Hi,
I would like to know the real functionality that hudi cleaner does.

In my opinion, there may be two choices a cleaner could provide per user's business usage.

1. Delete old commits and also the data, if a cleaner works in this way, then the historic data belonging to these commits will also be deleted. It could be useful if historic data is no use to end user's business and possibly speed up read/write since there are fewer commits/data there.

2. Merge the old commits into a new commit, also merge the data belonging to the old commits into new commit(like Spark's RDD checkpoint to cut off the long lineage). If a cleaner works in this way, then end user could keep the historic data, and since there fewer commits there, incremental read between commits will be speed up.

I want to know how hudi cleaner works ,thanks.

[bvaradar]

https://cwiki.apache.org/confluence/display/HUDI/FAQ#FAQ-WhatdoestheHudicleanerdo

Hudi Cleaner removes old file versions that are no longer required based on the retention configuration.

[bithw1]

Thanks @bvaradar.
In my case, I would keep all the data since the hoodie table is created. Should I set hoodie.cleaner.commits.retained to a large value, eg, Integer.MAX_VALUE, so that hudi will not delete any old files?

[bvaradar]

If you want to retain all older version, you can turn off cleaning by setting "hoodie.clean.automatic=false".

Note that not cleaning older file versions would result in unbounded increase in the number of physical files.

[bithw1]

Thanks @bvaradar .

What is the effect of cleaning older file versions ? Does this operation also delete data?

Task the following operations as an example(15 commits and each commit inserts one letter)

commit id: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
data to insert: A B C D E F G H I J K L M N O P Q

When hudi cleans the old commit from 1 to 5, will A to E also be deleted or they are kept(A~E can be queried out with select *）