Skip to content

[PERF] Concurrent compaction — run multiple compaction threads in parallel for different CFs #200

@ElioNeto

Description

@ElioNeto

Description

Currently ApexStore runs compaction in a single background thread (maybe_compact). Multiple column families compact sequentially. Competitive engines support concurrent compaction across column families.

Impact

  • With 8+ column families, compaction becomes a bottleneck
  • Write stalls while waiting for single-thread compaction to finish
  • CPU cores are underutilized during compaction I/O

Proposed Implementation

  1. Change compaction_thread from Mutex<Option<JoinHandle>> to Mutex<Vec<JoinHandle>>
  2. Allow up to N concurrent compactions (configurable via compaction.max_concurrent)
  3. Assign one compaction per column family
  4. Use a semaphore to limit total concurrent compactions
  5. Ensure close() waits for all compaction threads to finish

Labels

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions