quick compact: compact only evicted range when it reaches 32MB #11

Merged
merged 1 commit into from Aug 25, 2012

3 participants

@erikfrey

When darner pops an item, it writes a deletion marker to its journal. Occasionally, it should sweep through the journal and delete the old items and the deletion markers.

The old approach had two problems:

  • Old Darner compacted every 1024 pops, which meant the actual compaction range varied widely depending on item size. With really large items (hundreds of kilobytes to megabytes) going through the queue, compaction happened infrequently, caused unbounded memory bloat, and was expensive, freezing the world for tens of seconds, when it did.

  • Old Darner swept the whole journal instead of just the evicted range. Sweeping just the known evicted range is much more efficient than trying to compact the whole journal.

This new compaction approach is more carefully amortized into regular operation: compactions will be upper bounded by how quickly the machine can sweep through 32MB of ram: in tests this is typically hundreds of milliseconds.

Many other queues do compaction in a separate thread - I've amortized it into regular operation because I don't want to deal with the extra complexity. But I may consider that in the future.

@matpalm

nice!

@nova77

Looks fine. Just one question: is the compact operation be run in a multi-threaded environment? Leveldb has different levels of safety for concurrency and I don't think CompactRange is among these.

@erikfrey

@nova77 darner does all its work in a single thread.

@erikfrey erikfrey merged commit a989904 into master Aug 25, 2012
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment