Skip to content

Vanderhell/loxdb

BranchesTags

Repository files navigation

loxdb

loxdb

Embedded database for microcontrollers. Three engines. One malloc. Zero dependencies. Deterministic durable storage core for MCU/embedded systems.

CI Language: C99 License: MIT Platform: MCU | Linux | Windows | macOS Tests Release Wiki Contributing Security

What is loxdb?

loxdb is a compact embedded database written in C99 for firmware and small edge runtimes. It combines three storage models behind one API surface:

  • KV for configuration, caches, and TTL-backed state
  • Time-series for sensor samples and rolling telemetry
  • Relational for small indexed tables

The library allocates exactly once in lox_init(), runs without external dependencies, and can operate either in RAM-only mode or with a storage HAL for persistence and WAL recovery.

Free-tier additions (latest)

  • Runtime integrity API: lox_selfcheck(...)
  • WCET package:
    • compile-time bounds: include/lox_wcet.h
    • analysis guide: docs/WCET_ANALYSIS.md
  • TS logarithmic retention:
    • policy: LOX_TS_POLICY_LOG_RETAIN
    • extended registration: lox_ts_register_ex(...)

Product Contract

Project Governance

Why not SQLite?

SQLite is excellent, but it targets a different operating point. loxdb is intentionally narrower:

  • one malloc at init, no allocator churn during normal operation
  • fixed RAM budgeting across engines
  • tiny integration surface for MCUs and RTOS firmware
  • simpler persistence model for flash partitions and file-backed simulation

If you need SQL, dynamic schemas, concurrent access, or large secondary indexes, use SQLite. If you need predictable memory and embedded-first behavior, loxdb is the better fit.

Quick start

1. Add to your project:

add_subdirectory(loxdb)
target_link_libraries(your_app PRIVATE loxdb)

2. Configure and initialize:

#define LOX_RAM_KB 32
#include "lox.h"

static lox_t db;

lox_cfg_t cfg = {
    .storage = NULL,   // RAM-only; provide HAL for persistence
    .now     = NULL,   // provide timestamp fn for TTL support
};
lox_init(&db, &cfg);

C++ wrapper (incremental)

Header:

  • include/lox_cpp.hpp

Current wrapper surface:

  • lifecycle: init/deinit/flush
  • diagnostics: stats, db_stats, kv_stats, ts_stats, rel_stats, effective_capacity, pressure
  • KV: kv_set/kv_put/kv_get/kv_del/kv_exists/kv_iter/kv_clear/kv_purge_expired, admit_kv_set
  • TS: ts_register/ts_insert/ts_last/ts_query/ts_query_buf/ts_count/ts_clear, admit_ts_insert
  • REL: schema/table helpers + rel_insert/find/find_by/delete/iter/count/clear, admit_rel_insert
  • txn: txn_begin/txn_commit/txn_rollback

Minimal example:

#include "lox_cpp.hpp"

loxdb::cpp::Database db;
lox_cfg_t cfg{};
cfg.ram_kb = 32u;
if (db.init(cfg) != LOX_OK) { /* handle error */ }

uint8_t v = 7u, out = 0u;
db.kv_put("k", &v, 1u);
db.kv_get("k", &out, 1u);

db.txn_begin();
db.kv_put("k2", &v, 1u);
db.txn_commit();

db.deinit();

Optional wrappers and adapter modules

Core loxdb is intentionally lean. Extra wrappers/adapters are separate modules and can be toggled in CMake.

Build toggles:

  • LOX_BUILD_JSON_WRAPPER (default ON)
  • LOX_BUILD_IMPORT_EXPORT (default ON)
  • LOX_BUILD_OPTIONAL_BACKENDS (default ON)
  • LOX_BUILD_BACKEND_ALIGNED_STUB / LOX_BUILD_BACKEND_NAND_STUB / LOX_BUILD_BACKEND_EMMC_STUB / LOX_BUILD_BACKEND_SD_STUB / LOX_BUILD_BACKEND_FS_STUB / LOX_BUILD_BACKEND_BLOCK_STUB

Wrapper targets:

  • lox_json_wrapper
  • lox_import_export (links to lox_json_wrapper when available)
  • lox_backend_registry
  • lox_backend_compat
  • lox_backend_decision
  • lox_backend_aligned_adapter
  • lox_backend_managed_adapter
  • lox_backend_fs_adapter
  • lox_backend_open

Core contract:

  • optional modules are not linked into loxdb core by default.
  • loxdb must remain independent from optional wrapper/backend targets.

3. Use all three engines:

// Key-value
float temp = 23.5f;
lox_kv_put(&db, "temperature", &temp, sizeof(temp));

// Time-series
lox_ts_register(&db, "sensor", LOX_TS_F32, 0);
lox_ts_insert(&db, "sensor", time_now(), &temp);

// Relational
lox_schema_t schema;
lox_schema_init(&schema, "devices", 32);
lox_schema_add(&schema, "id",   LOX_COL_U16, 2, true);
lox_schema_add(&schema, "name", LOX_COL_STR, 16, false);
lox_schema_seal(&schema);
lox_table_create(&db, &schema);

Configuration

Configuration is compile-time first, with a small runtime override surface in lox_cfg_t.

  • LOX_RAM_KB sets the total heap budget
  • LOX_RAM_KV_PCT, LOX_RAM_TS_PCT, LOX_RAM_REL_PCT set default engine slices
  • cfg.ram_kb overrides the total budget per instance
  • cfg.kv_pct, cfg.ts_pct, cfg.rel_pct override the slice split per instance
  • LOX_ENABLE_WAL toggles WAL persistence when a storage HAL is present
  • cfg.wal_sync_mode selects WAL durability/latency mode:
    • LOX_WAL_SYNC_ALWAYS (default): sync on each append, strongest per-op durability
    • LOX_WAL_SYNC_FLUSH_ONLY: sync on explicit lox_flush(), lower write latency
    • see measured ESP32 tradeoffs in bench/lox_esp32_s3_bench/README.md ("WAL Sync Mode Decision Table")
  • LOX_LOG(level, fmt, ...) enables internal diagnostic logging
  • smallest-size variant is available as CMake target lox_tiny (KV-only, TS/REL/WAL disabled, weaker power-fail durability)
  • strict smallest durable profile is available as LOX_PROFILE_FOOTPRINT_MIN (KV + WAL + reopen/recovery contract)

Storage budget (separate from RAM budget):

  • storage capacity comes from lox_storage_t.capacity (bytes)
  • geometry comes from lox_storage_t.erase_size and lox_storage_t.write_size
  • current fail-fast storage contract requires erase_size > 0 and write_size == 1
  • use tools/lox_capacity_estimator.html for storage/layout planning (2/4/8/16/32 MiB profiles)

KV engine

The KV engine stores short keys with binary values and optional TTL.

  • fixed-size hash table with overwrite or reject overflow policy
  • LRU eviction for LOX_KV_POLICY_OVERWRITE
  • TTL expiration checked on access
  • WAL-backed persistence for set, delete, and clear

Time-series engine

The time-series engine stores named streams of F32, I32, U32, or raw samples.

  • one ring buffer per registered stream
  • range queries by timestamp
  • overflow policies: drop oldest, reject, downsample, or logarithmic retain
  • per-stream extended registration via lox_ts_register_ex(...)
  • WAL-backed persistence for inserts and stream metadata

Relational engine

The relational engine stores small fixed schemas with packed rows.

  • one indexed column per table
  • binary-search index on the indexed field
  • linear scans for non-index lookups
  • insertion-order iteration
  • WAL-backed persistence for inserts, deletes, and table metadata

Storage HAL

loxdb supports three storage modes:

  • RAM-only: cfg.storage = NULL
  • POSIX file HAL for tests and simulation
  • ESP32 partition HAL via esp_partition_*
  • RTOS skeleton templates: examples/freertos_port/, examples/zephyr_port/
  • SD + FatFS glue skeleton: examples/sd_fatfs_port/

Supported Platforms

Verified on hardware

Platform Flash RAM Status
ESP32-S3 N16R8 16MB external NOR 8MB PSRAM Verified (run_real PASS, benchmarked)

Compatible (direct byte-write port)

Platform Flash path Typical kv_put (directional) Practical minimum RAM
ESP32 / ESP32-C3 / ESP32-S2 SPI NOR via esp_partition ~70-90us 16KB
STM32H7 QSPI NOR ~30us 16KB
STM32F4 External SPI NOR ~80us 16KB
RP2040 QSPI XIP flash ~50us 16KB
nRF52840 Internal flash ~90us 16KB
SAMD51 External SPI NOR ~100us 16KB

Compatible via aligned adapter (write_size > 1)

Platform Flash path Notes
STM32F103 Internal flash (write_size=2) Requires lox_backend_aligned_adapter
STM32L4 Internal flash (write_size=8) Requires lox_backend_aligned_adapter
nRF5340 Internal flash (write_size=4) Requires lox_backend_aligned_adapter

Not supported without larger changes

Platform family Limitation
AVR (ATmega class) No practical malloc model for current core + different EEPROM persistence model
MSP430 (small variants) Constrained address/model assumptions for current storage offset contract

Notes:

  • Latency values are board/flash/vendor dependent; treat them as directional and re-measure on target hardware.
  • kv_get is RAM-only in normal path; write costs are typically dominated by flash backend latency.

Persistent layout starts with a WAL region and then separate KV, TS, and REL regions aligned to the storage erase size.

Core storage positioning: loxdb core today natively supports byte-write durable backends; aligned/block/NAND media require a translation layer.

Optional backend adapter modules (modular, not linked into core by default):

  • lox_backend_registry (adapter registration layer)
  • lox_backend_compat (open-time direct / via_adapter / unsupported classification)
  • lox_backend_aligned_adapter (RMW byte-write shim for aligned-write media)
  • lox_backend_managed_adapter (managed-media adapter skeleton for eMMC/SD/NAND via managed interface)
  • lox_backend_open (decision + adapter wiring helper for optional backend flow)
  • stub modules for managed media (nand/emmc/sd) used for integration/testing flow

Important:

  • nand/emmc/sd stubs are capability descriptors, not hardware drivers.
  • SD/eMMC integration still requires a real platform stack (for example FatFS/LittleFS over SDMMC/SPI, or vendor managed block API) mapped into lox_storage_t.
  • Raw NAND is not a direct loxdb target: you need a managed layer that handles ECC, bad blocks, and wear leveling, then place loxdb above that layer.
  • See examples/sd_fatfs_port/main.c for a practical SD+FatFS glue skeleton.

Managed adapter contract (fail-fast):

  • validates storage hooks and non-zero capacity/erase geometry
  • default expectations require byte-write contract and a successful sync probe at mount time
  • exposes an explicit expectations override API for controlled integration/testing
  • managed recovery integration tests cover reopen/power-loss behavior through backend-open wiring
  • managed stress test covers mixed KV/TS/REL operations across repeated crash/power-loss reopen cycles
  • managed stress is sliced into smoke and long CTest lanes for faster default validation and deeper fault runs
  • both lanes now include explicit runtime envelope gates (--max-ms) in addition to CTest timeouts
  • lane budgets are calibrated through CMake cache vars: LOX_MANAGED_STRESS_SMOKE_MAX_MS and LOX_MANAGED_STRESS_LONG_MAX_MS (see docs/MANAGED_STRESS_BASELINES.md)
  • CI uses CMakePresets.json (ci-debug-linux, ci-debug-windows) to apply platform-specific stress budgets consistently
  • release workflow now uses CMakePresets.json (release-linux, release-windows) with profile-specific stress budgets
  • scheduled baseline refresh workflow publishes runtime artifacts for ongoing threshold calibration
  • scripts/recommend-managed-baselines.ps1 computes recommended thresholds from historical baseline artifacts (p95 + margin)
  • baseline refresh workflow now also publishes aggregated recommendation artifacts (json + md)
  • scripts/apply-managed-thresholds.ps1 can apply recommendation JSON directly to preset budgets (--dry-run supported)
  • baseline refresh workflow publishes candidate preset patch artifacts (candidate CMakePresets.json + diff)

Storage contract (fail-fast at lox_init):

  • erase_size must be > 0
  • write_size must be exactly 1
  • write_size == 0 or write_size > 1 currently returns LOX_ERR_INVALID

Read-only diagnostics API

System stats are exposed through read-only APIs (not user KV keys):

  • lox_get_db_stats(...)
  • lox_get_kv_stats(...)
  • lox_get_ts_stats(...)
  • lox_get_rel_stats(...)
  • lox_get_effective_capacity(...)
  • lox_get_pressure(...)
  • lox_selfcheck(...)
  • lox_admit_kv_set(...)
  • lox_admit_ts_insert(...)
  • lox_admit_rel_insert(...)

Admission preflight semantics:

  • API return value reports API-level validity (LOX_OK when request was analyzed)
  • final operation decision is in lox_admission_t.status
  • would_compact indicates compact pressure for the projected write
  • would_degrade indicates policy-driven degradation path (for example overwrite/drop-oldest)
  • deterministic_budget_ok indicates whether request fits deterministic budget

Pressure semantics:

  • lox_get_pressure(...) exposes kv/ts/rel/wal fill percentages
  • compact_pressure_pct expresses WAL fill relative to compact threshold
  • near_full_risk_pct is max pressure signal across engines/WAL

Migrations vs Snapshots

loxdb has three different concepts that are easy to mix up:

  1. Schema migrations (public API)
  • REL schema upgrades are driven by schema_version + cfg.on_migrate.
  • Trigger point is lox_table_create(...) when an existing table version differs.
  • Without on_migrate, version mismatch returns LOX_ERR_SCHEMA.
  • See docs/SCHEMA_MIGRATION_GUIDE.md.
  1. Durable snapshots (internal durability mechanism)
  • WAL/compact/recovery uses internal snapshot banks (A/B) + superblocks.
  • This is an internal storage safety/recovery mechanism, not a user-facing snapshot API.
  • There is currently no public API to create/list/restore named snapshots.
  1. Query-time snapshot semantics (iteration consistency)
  • TS/REL query/iter paths capture mutation state and validate after callback re-lock.
  • If concurrent mutation is detected, APIs return LOX_ERR_MODIFIED.
  • This protects traversal consistency; it is separate from durable storage snapshots.

Semantics:

  • last_runtime_error is sticky last non-LOX_OK runtime status since lox_init
  • last_recovery_status is status of the last open/recovery path step in this process lifetime
  • compact_count, reopen_count, recovery_count are runtime-only counters (not persistent)
  • REL uses rows_free (free slots), not a historical deleted-rows counter

RAM budget guide

LOX_RAM_KB KV entries (est.) TS samples/stream (est.) REL rows (est.) Typical use
8 KB ~3 ~32 ~4 Ultra-tiny KV-focused profile
16 KB ~40 ~136 ~8 Small MCU baseline
32 KB ~64 ~1 500 ~30 General embedded node
64 KB ~150 ~3 000 ~80 Rich sensing / control node
128 KB ~300 ~6 000 ~160 MCU + external RAM
256 KB ~600 ~12 000 ~320 High-retention edge node
512 KB ~1 200 ~24 000 ~640 Linux embedded
1024 KB ~2 500 ~48 000 ~1 300 Resource-rich MCU
txn staging overhead LOX_TXN_STAGE_KEYS * sizeof(lox_txn_stage_entry_t) bytes same same Reserved from KV slice

Estimates assume default 40/40/20 RAM split and default column sizes. Override with LOX_RAM_KV_PCT, LOX_RAM_TS_PCT, LOX_RAM_REL_PCT.

Capacity planning helper:

  • open tools/lox_capacity_estimator.html for profile-based storage/layout estimation (2/4/8/16/32 MiB) and rough record-fit planning.

Design decisions and known limitations

Single malloc at init. loxdb allocates exactly once in lox_init() and never again. This makes memory usage predictable and eliminates heap fragmentation - a critical property for long-running embedded systems.

Fixed RAM slices per engine. Each engine gets a fixed percentage of the RAM budget at init time. There is no automatic redistribution if one engine fills up while another has free space. This is intentional - dynamic redistribution would require a runtime allocator, breaking the single-malloc guarantee. Workaround: tune kv_pct, ts_pct, rel_pct in lox_cfg_t for your use case.

One index per relational table. Each table supports one indexed column for O(log n) lookups. All other column lookups are O(n) linear scans. For embedded tables with <= 100 rows this is acceptable (microseconds on ESP32). Secondary indexes are not planned for v1.x.

LRU eviction is O(n). When KV store is full and overflow policy is OVERWRITE, finding the LRU entry requires scanning all buckets. At LOX_KV_MAX_KEYS=64 this is 64 comparisons. For embedded use cases this is negligible. Not suitable for LOX_KV_MAX_KEYS > 1000.

Optional hook-based thread safety. Enable LOX_THREAD_SAFE=1 and provide lock_create/lock/unlock/lock_destroy hooks in lox_cfg_t to integrate your RTOS/application mutex.

No built-in compression or encryption. If your product needs those, apply them in your application layer before calling loxdb APIs.

Test coverage

The repository covers:

  • KV engine behavior and overflow variants
  • TS engine behavior and overflow variants
  • REL schema, indexing, and iteration
  • WAL recovery, corruption handling, and disabled-WAL mode
  • integration flows across RAM-only and persistent modes
  • RAM budget variants from 8 KB through 1024 KB
  • compile-fail validation for invalid percentage sums
  • tiny-footprint profile checks (test_tiny_footprint + test_tiny_size_guard)
  • canonical footprint-min baseline + hard section/linkage gate (test_footprint_min_baseline + test_footprint_min_size_gate_release, Release contract)

Integration note

loxdb is storage-focused. Transport, serialization, and cryptography are handled by surrounding application components.

Wiki

GitHub Wiki source pages are stored in wiki/. That keeps documentation versioned in the main repository and ready to publish into the GitHub wiki repo.

License

MIT.

License details and file-level SPDX policy:

About

Embedded database for microcontrollers. Three engines (KV, time-series, relational), single malloc, zero dependencies, power-loss safe WAL. C99 · ESP32 · 175+ tests

Topics

c iot microcontroller embedded database time-series key-value firmware esp32 c99 embedded-database bare-metal no-heap zero-depedency

Resources

Readme

License

MIT license

Code of conduct

Code of conduct

Contributing

Contributing

Security policy

Security policy
Activity

Stars

1 star

Watchers

0 watching

Forks

0 forks
Report repository

Releases 4

v1.4.0 Latest
Apr 23, 2026
+ 3 releases

Packages

 
 
 

Contributors

Languages