appendix

A small append-only key-value store in Rust. Bitcask-style: writes go to the end of a log, an in-memory hash index maps keys to value offsets, and reads seek straight to the offset. Built as a learning project.

Quick start

use appendix::Appendix;

let db = Appendix::open("data/my.db")?;

db.set("user", b"alice")?;
let value = db.get("user");          // Some(b"alice".to_vec())
db.delete("user")?;
db.compact()?;                       // optional; runs automatically too

API

Appendix::open(path) -> Result<Appendix>
db.get(key: &str)         -> Option<Vec<u8>>
db.set(key: &str, value: &[u8]) -> Result<()>
db.delete(key: &str)      -> Result<()>
db.compact()              -> Result<()>

Limits: keys up to 65535 bytes, values up to 4 GiB.

Examples

cargo run --example basic
cargo run --example persistence
cargo run --example concurrent
cargo run --example compaction

On-disk format

Each record is appended as:

[crc32c 4][type 1][key_len 2][value_len 4][key][value]

type is 0x01 for a set and 0x02 for a tombstone. The CRC covers everything after itself. sync_data is called after every append.

Design notes

Open. The file is replayed from the start. Each valid record updates the in-memory index; the first record that fails CRC, runs past EOF, or has an unknown type marks the end of the good prefix. The writer is truncated to that offset, so a torn tail from a previous crash is discarded.

Reads. JournalReader uses positional reads (pread on Unix), so multiple threads can read concurrently without a lock on the file handle. The index lives behind an RwLock, and the reader handle behind an ArcSwap so compaction can swap it without blocking readers.

Writes. A single Mutex<JournalWriter> serializes appends. Lock order is always writer → index, which matches compact() so the two can't deadlock.

Compaction. Walks the live index, copies each live value into <path>.compact, renames over the original, and fsyncs the parent directory so the rename is durable. Auto-triggered after a write when the file is at least 1 MiB and live bytes are less than 50% of file size.

Non-goals

No transactions, no range scans, no concurrent writers, no networking. The index is fully in memory, so total key size has to fit in RAM.