Compact the iceberg table of V2 to delete two kinds of deleting file and merge the some small datafile

[AlexLY123]

[AlexLY123]

Now there are two kinds of deleting file that are position delete file an d enqulity delete file in the model of MOR. In our practical application, it produced a large number of deleted files. And the more serious problem is that we can't merge data files as there are some errors. These problems will result in very slow queries and adding cluster stress with a large number of small files. So I don't think the upsert function works very well without compaction. It is necessary to develop the compaction feature just like the minor compaction and major compaction.

[AlexLY123]

We need to make different compression mechanisms based on execution time and resource costs. For example， we can make the minor compaction and major compaction just like hbase. The minor compact will be executed frequently with very low resource consumption and short time consumption， for example, only merging postition delete files generated by the latest snapshot at a time. The major compact will not be executed frequently to delete the two kinds of delete files and merge the data files， for example, applying postition delete files and enqulity delete files to data files of the whole table and merging the small data file to generate the new data files without delete files.

[ajantha-bhat]

@AlexLY123 :

compaction already handles merging the delete files. but by default it is disabled. Need to configure the
delete-file-threshold to enable it.
https://github.com/apache/iceberg/blob/master/core/src/main/java/org/apache/iceberg/actions/BinPackStrategy.java#L87-L88

Also regarding major and minor compaction. Even though we don't have similar to hbase, we can run compaction based on the resource availability by selecting few files itself;
Currently possible by using the filter method (where clause) of rewrite_data_files

Also lot of control is there by configuring parallelism and enabling partial commit during compaction etc.
cc: @jackye1995 , @RussellSpitzer

[AlexLY123]

Now we user the iceberg with 0.12.0. In this release of 0.12.0, there isn't the compaction function to merge the delete files.
We think we should add the action to merge the delete file as major compact. After compacting we can merge the data files.

[github-actions]

This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.

[github-actions]

This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'