Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Garbage collection write amplification #429

Open
sanebay opened this issue May 21, 2024 · 0 comments
Open

Garbage collection write amplification #429

sanebay opened this issue May 21, 2024 · 0 comments
Assignees
Labels
Milestone

Comments

@sanebay
Copy link
Contributor

sanebay commented May 21, 2024

Because GC is a heavy operation and can cause unnecessary write amplification, ti should do following.

  1. Group shards of similar TTL or buckets across customers to be together. Even if TTL is some arbitrary number, we can group them, so that we dont have to move data and we can release chunk with a quick metadata update.
  2. DM can send some hints as part of create shard and chunk selector tries its best to group similar shards in same chunk.
  3. Keep track of how many times a shard has been moved in the shard header. This needs to be exported via metrics.
@sanebay sanebay added this to the MileStone4.3 milestone May 21, 2024
@xiaoxichen xiaoxichen added the GC label Aug 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants