Skip to content

Convert memtable rotation to a channel message #78

@marvin-j97

Description

@marvin-j97

Currently the memtable rotation happens in the write hot path when the memtable size threshold is reached:

fjall/src/partition/mod.rs

Lines 747 to 752 in 9395ba1

pub(crate) fn check_memtable_overflow(&self, size: u32) -> crate::Result<()> {
if size > self.config.max_memtable_size {
self.rotate_memtable()?;
self.check_journal_size();
self.check_write_halt();
}

The performance hit mostly comes from the journal rotation which needs to fsync 2x (?) to disk.
It would be better for performance to send a message via channel, announcing that the partition needs to be flushed, nullifying rotation costs inside the write hot path, which should give quite a performance boost for write heavy applications (currently it is slightly slower than Rocks).

The messages should be picked up by a worker that rotates the partition asynchronously, essentially performing the Partition::rotate_memtable function:

pub fn rotate_memtable(&self) -> crate::Result<bool> {

Metadata

Metadata

Assignees

Type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions