An embeddable key-value store for the Clojure ecosystem — pure Clojure, zero native dependencies, and usable directly from Babashka with no pod and no native binary. Just add it to your classpath.
Built on an LSM tree with leveled compaction, IgelDB persists every write to disk and is designed to be crash-safe: durability isn't an afterthought but a tested property.
Add IgelDB to deps.edn:
com.igel-data/igeldb {:mvn/version "1.0.0"}- Drop-in for Babashka. Unlike stores that ship a native binary (and reintroduce the per-OS/per-arch distribution problem), IgelDB is pure Clojure. It loads on a Babashka classpath as-is — no pod process, no compiled artifact to build or match to your platform.
- Zero native dependencies. Runs on plain JVM bytecode, so distribution and debugging on multi-architecture environments (e.g. mixed-arch Kubernetes) stay simple.
- Crash-safe by design. Every write goes through a write-ahead log with
fsync-backed group commit; the memtable is only updated after the WAL is durably persisted. Compaction and the manifest are structured so a crash at any point recovers to a consistent state. These properties are covered by tests, including crash-recovery paths. - Lightweight footprint. Small defaults tuned for embedding in an application process. Everything (memtable size, compaction thresholds, level sizing) is configurable. For hundreds-of-GB workloads, reach for RocksDB — IgelDB targets the embedded niche.
- Concurrent reads. Readers work against an immutable snapshot with no locking, so reads proceed concurrently with writes and compaction.
- ACID transactions. Multi-key transactions provide snapshot isolation with first-committer-wins conflict detection. Snapshot isolation is not serializability: transactions whose write sets do not overlap can still exhibit write skew.
- LSM tree with a memtable (an immutable sorted map, swapped atomically), on-disk SSTables, and leveled compaction.
- Write path: WAL append →
fsync(group-committed) → memtable apply → return. A single commit worker applies each batch in WAL order, so WAL order and memtable order always match. - Manifest: an append-only log of version edits (added/deleted tables),
fsync-committed; startup rebuilds state by replaying it. A crash between steps recovers cleanly. - Bloom filters on SSTables (via blossom) skip disk reads on misses.
(require '[igeldb.core :as igel])
;; Generate a KVS with a config file
(def kvs (igel/gen-kvs "config.yaml"))
;; A key and a value as bytes
(def key1 (.getBytes "key1"))
(def val1 (.getBytes "val1"))
;; Write the key-value pair
(igel/write! kvs key1 val1)
;; Read the key-value pair
(igel/select kvs key1)
; -> #whidbey/bin "dmFsMQ"
;; with deserialization
(String. (igel/select kvs key1))
; -> "val1"
;; Write some pairs
(doseq [i (range 2 5)]
(igel/write! kvs (.getBytes (str "key" i)) (.getBytes (str "val" i))))
;; Scan pairs with from-key(inclusive) and to-key(not-inclusive)
(def from-key (.getBytes "key0"))
(def to-key (.getBytes "key3"))
(igel/scan kvs from-key to-key)
; -> ([#whidbey/bin "a2V5MQ" #whidbey/bin "dmFsMQ"]
; [#whidbey/bin "a2V5Mg" #whidbey/bin "dmFsMg"])
;; with deserialization
(map (fn [[k v]] [(String. k) (String. v)])
(igel/scan kvs from-key to-key))
; -> (["key1" "val1"] ["key2" "val2"])
;; Delete a key
(igel/delete! kvs (.getBytes "key2"))
;; The key2 was deleted
(map (fn [[k v]] [(String. k) (String. v)])
(igel/scan kvs from-key to-key))
; -> (["key1" "val1"])
;; Atomically read and update multiple keys. with-tx commits on success and
;; rolls back if its body throws.
(igel/with-tx [tx kvs]
(let [current (String. (igel/tx-get tx key1))]
(igel/tx-put tx key1 (.getBytes (str current "-updated")))
(igel/tx-put tx (.getBytes "key5") (.getBytes "val5"))))
;; Close the store when done: flushes and releases resources.
;; After close!, reads and writes are rejected.
(igel/close! kvs)Keys and values are byte[]. Serialize your own values to bytes before writing.
with-tx does not automatically retry write conflicts; retry the whole transaction
when conflict handling is appropriate for your application.
IgelDB runs under Babashka with no extra setup — add src and the runtime deps to your
classpath and require it as usual. See test/bb_smoke.clj for a
full write → flush → compaction → reopen (recovery) cycle running on bb.
Configuration is a YAML file (see gen-kvs). Key parameters:
| Key | Meaning |
|---|---|
sstable-dir / wal-dir |
Directories for SSTables and WAL files |
memtable-size |
Flush threshold; write buffer size before flushing to an SSTable |
sync-window-time |
Group-commit window (ms) for batching fsync |
l0-compaction-trigger |
L0 table count that triggers L0→L1 compaction |
l0-stall-threshold |
L0 table count at which writers stall (back-pressure safety valve) |
level-size-multiplier |
Per-level size growth factor |
manifest-rotation-bytes |
Manifest edit bytes between full snapshot rotations |
manifest-rotation-edits |
Manifest edit count between full snapshot rotations |
Manifest format v1 uses CURRENT to select a generational
MANIFEST-<generation> file. Earlier unversioned MANIFEST files are rejected;
they are not upgraded automatically.
IgelDB's crash tests use process termination (SIGKILL). They verify recovery
from WALs, SSTables, and rotated manifests, but do not simulate sudden power loss:
the operating system may still retain acknowledged filesystem writes in its page
cache after a process is killed.
An open IgelDB instance exclusively owns both configured storage directories.
Opening another instance that shares either directory fails until the first
instance's synchronous close! returns.
Copyright © 2023 Yuji Ito
Distributed under the Eclipse Public License 2.0 (or the secondary licenses it allows). See LICENSE for details.