wedb_embed: An embedded Redis/Valkey-compatible LSM disk database engine in Rust

[x-at-01]

wedb_embed: An embedded Redis/Valkey-compatible LSM disk database engine in Rust

Hi Valkey community,

I wanted to share an open-source project we built for the Rust ecosystem: wedb_embed.

While Valkey provides a high-performance distributed key-value server, wedb_embed implements an embedded, in-process disk-backed engine in pure Rust built on top of the Fjall LSM storage engine.

Much like SQLite provides an in-process alternative to client-server SQL databases, wedb_embed provides an in-process, disk-backed alternative for applications that require Redis/Valkey data structures without running an external server process or keeping entire datasets in memory.

_{Benchmark Environment: CPU: AMD EPYC 7763 (4 cores) | Memory: 15.6 GB | Disk: Cloud Standard SSD | OS: Ubuntu 24.04.4 LTS | Rust: 1.98.0 | Redis: v8.10.1}

Architecture and data modeling

wedb_embed maps operations across two LSM keyspaces:

• data keyspace: stores raw string values, subkeys for composite collections, and blob references. Configured with 8KB block size, LZ4 compression, and KV separation (4KB threshold) to minimize write amplification during compaction.
• meta keyspace: stores composite key metadata, version counters, and tenant catalog entries. Configured with 4KB block size, in-memory hash indexing, and pinned index/filter blocks for low-latency point lookups.
• Zero-heap key encoding: uses Order-Preserving Prefix Varint (OPPV) with 64-byte stack buffers (SmallKey) to avoid allocator contention in hot paths.
• Data structures: 16 Redis-compatible data types, including String, Hash (with field-level TTL), List, Set, Sorted Set, Bitmap, JSON (JSONPath), Bloom/Cuckoo filters, TimeSeries, Geo, HyperLogLog, TDigest, Streams, Full-Text Search, and HNSW vector search.

Benchmarks on 5GB dataset (5,000,000 structured items)

• Physical disk usage: 1,179 MB vs 7,531 MB for Redis AOF (84% reduction via LZ4 compression and KV separation).
• Resident memory (RSS): 281 MB vs 4,800 MB for Redis (94% reduction). Memory footprint is strictly bounded by the LSM block cache limit.
• Latency: Running in-process eliminates network syscalls, socket buffers, and RESP encoding, bringing P95 latency down to the microsecond and sub-microsecond range.

We welcome thoughts from the Valkey community on storage layout, command compatibility, and embedded data design.

Links:

• Repository: https://github.com/webc-site/wedb_embed
• Examples: https://github.com/webc-site/wedb_embed/tree/main/examples
• Crates.io: https://crates.io/crates/wedb_embed
• Docs.rs: https://docs.rs/wedb_embed