Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Enhancement]: Remove all bf from datanode #34585

Open
1 task done
bigsheeper opened this issue Jul 10, 2024 · 0 comments
Open
1 task done

[Enhancement]: Remove all bf from datanode #34585

bigsheeper opened this issue Jul 10, 2024 · 0 comments
Assignees
Labels
kind/enhancement Issues or changes related to enhancement

Comments

@bigsheeper
Copy link
Contributor

Is there an existing issue for this?

  • I have searched the existing issues

What would you like to be added?

Currently, datanode keeps a full in-memory bloom filter for all segments. This is mainly to check if a delete message hits and to reduce delete data.

I suggest removing the bf from datanode for these reasons:

  1. Maintaining bf only optimizes in two cases:
  • a) User deletes lots of non-existent primary keys
  • b) User uses upsert as insert.
    We can rely on L0 compaction for the two cases. With batched bf, L0 compaction is fast enough to handle these cases (this can be tested).
  1. Maintaining a full bf in datanode is error-prone(see: enhance: [cherry-pick] Periodically synchronize segments to datanode watcher (#33420)  #34186), and any omissions could result in losing delete data(see: enhance: add the seal segment when dispatch delete msgs #34565).
  2. Removing bf will speed up delete processing in flowgraph and reduce datanode's complexity.

Why is this needed?

No response

Anything else?

No response

@bigsheeper bigsheeper added the kind/enhancement Issues or changes related to enhancement label Jul 10, 2024
@bigsheeper bigsheeper self-assigned this Jul 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/enhancement Issues or changes related to enhancement
Projects
None yet
Development

No branches or pull requests

1 participant