[Improvement]: Improve optimizer's compaction performance in large delete scenarios

[zhongqishang]

Search before asking

• I have searched in the issues and found no similar issues.

What would you like to be improved?

In the case of large data volume, we can enable rocksdb as overflow storage.

The following scenarios the merge performance needs to be improved:

• Write full data through flink cdc
• A large amount of data is refreshed in a short period of time, include insert update delete

How should we improve?

When eq delete total record count is greater than 1 million, we write the data file primary key into bloom filter.
When reading eq-delete files, we determine whether it exists in data bloom filter. If it does not exist, we can directly ignore it (currently all written to eq delete StructLikeMap), which can greatly reduce the storage operation of eq delete. In this way, the size of bloom filter is controllable, so the memory usage is also controllable, depending on the size of the file.

Overflow writing to rocksdb will cause a large number of disk operations, so reduce the storage amount of eq-delete will speeds up merge operations.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Subtasks

No response

Code of Conduct

• I agree to follow this project's Code of Conduct