Skip to content

NEDB v2.0.4 — The DAG Engine

Choose a tag to compare

View all tags
@ethgr0wth ethgr0wth released this 16 Jun 19:15
v2.0.4
95eb787
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

"Content-addressed, tamper-evident, parallel, instant cold start."

This is the first production release of NEDB v2 — a ground-up rewrite of the storage engine from an append-only log (AOF) to a content-addressed Merkle DAG. The v1 Python AOF engine ships alongside it untouched; v2 is opt-in via --dag.

What's New

nedbd --dag — The v2 DAG Engine

Start the Rust v2 DAG engine with a single flag:

nedbd --dag --data /path/to/data
# or
NEDBD_DAG=1 nedbd --data /path/to/data

The Python process execs directly into the compiled Rust binary — no wrapper overhead, clean signal handling, same port/token/data env vars as v1. The v1 AOF engine is completely untouched.

Content-Addressed DAG Storage

Every document version is an immutable, encrypted, BLAKE2b-verified object stored at objects/{hash[0:2]}/{hash[2:]}. Nothing is ever overwritten.

  • Uncorruptable — atomic writes (tmp → rename), hash verification on every read
  • Parallel — no global lock; each document has its own index file
  • Instant cold start — no AOF replay; rebuild sorted indexes in parallel from the object store
  • Tamper-evident — BLAKE2b chain: MANIFEST → collection heads → every node
  • Self-healing — v1 AOF auto-migrates to v2 DAG on first startup (encrypted AOFs supported)

Real BLAKE2b Merkle Head

Every response from the v2 server includes a head field — a BLAKE2b hash of all current id-index hashes, sorted and chained with the global sequence number. Changes on every write. Anchorable. Verifiable.

{ "ok": true, "seq": 42, "head": "a3f9c2e1..." }

Real Tombstone Deletes

DELETE /v1/databases/{name}/rows/{coll}/{id} now:

  1. Writes an immutable tombstone node (history preserved in the DAG forever)
  2. Removes the live id pointer from the index
  3. Doc is permanently invisible to all queries and list() — but fully recoverable via AS OF

AES-256-GCM Encryption Through Migration

The DEK (derived from NEDB_TMK) now flows correctly through the v1→v2 migration path. Encrypted v1 AOF databases convert to v2 DAG without data loss.

Native Wheels with nedbd-v2 Binary

Platform wheels (manylinux, macos arm64, win amd64) now ship the compiled nedbd-v2 Rust binary alongside the Python package. pip install nedb-engine gives you both engines on every supported platform.

Installation

pip install --upgrade nedb-engine   # → 2.0.4

Run v2 DAG (recommended for new deployments)

NEDBD_DAG=1 NEDB_TMK=<32-byte-hex> nedbd --data /path/to/data
curl http://127.0.0.1:7070/health
# {"ok":true,"version":"2.0.4","service":"nedbd","encrypted":true}

Run v1 AOF (unchanged, all existing deployments)

NEDB_TMK=<32-byte-hex> nedbd --data /path/to/data

Automatic v1 → v2 Migration

Zero user action required. On first --dag startup, if log.aof is detected in a database directory, v2 reads all valid ops, converts them to content-addressed DAG nodes in parallel, rebuilds indexes, and renames log.aof → log.aof.v1.bak. The backup is always kept as a rollback path.

[nedb] Detected v1 log.aof — running automatic migration to v2 DAG...
[nedb] 469041 op(s) to migrate
[nedb] Migration complete: 469041 op(s) → v2 DAG. Backup: log.aof.v1.bak

The Full HTTP API

100% wire-compatible with v1 — Vision, NEDB Studio, and all existing clients work unchanged.

Method Route Description
GET /health Version, databases, encryption status
GET /v1/databases List all databases with seq + Merkle head
POST /v1/databases Create a database
GET /v1/databases/:name Database info
DELETE /v1/databases/:name Drop database
POST /v1/databases/:name/put Write document (causal provenance + bi-temporal)
POST /v1/databases/:name/query NQL query
POST /v1/databases/:name/batch Batch put/delete
DELETE /v1/databases/:name/rows/:coll/:id Tombstone delete
POST /v1/databases/:name/index Create sorted index
GET /v1/databases/:name/verify BLAKE2b tamper-evidence check (parallel)
POST /v1/databases/:name/checkpoint Checkpoint (no-op in v2 — always snapshotted)
GET /v1/databases/:name/log Recent write log

NQL — NEDB Query Language

FROM blocks
  [AS OF <seq>]
  [VALID AS OF "<iso-date>"]
  [WHERE field = value [AND ...]]
  [ORDER BY field [DESC]]
  [LIMIT n]
  [GROUP BY field COUNT|SUM|AVG|MIN|MAX]
  [TRACE caused_by [REVERSE]]
  [SEARCH "text"]

Full Changelog

  • feat: --dag / NEDBD_DAG=1 — os.execv into Rust v2 binary, no Python overhead
  • feat: Real tombstone deletes — tombstone node written, id removed from IdIndex
  • feat: Real BLAKE2b Merkle head on every response — tamper-evident, changes on every write
  • feat: DEK passthrough to migrate_if_needed() — encrypted v1 AOFs convert correctly
  • feat: nedbd-v2 binary bundled in native Python wheels (Linux / macOS / Windows)
  • fix: Dek derives Clone — fixes borrow-after-move in Db::open()
  • fix: Db.seq made pub — server can compute Merkle head
  • fix: base64_decode error type — String → anyhow::Error via .map_err()
  • fix: parking_lot dep corrected to 0.12
  • chore: rust/nedb-v2 added to Rust workspace with Cargo.lock committed
  • chore: All 6 version strings bumped to 2.0.4 in sync
  • docs: index.html + reference.html updated for v2 DAG engine

What's Next

  • PyO3 + napi-rs bindings updated to v2 Db API (Python SDK + Node SDK using DAG natively)
  • MANIFEST file persisted to disk for external anchoring
  • NEDB Studio: DAG mode toggle in the Connect panel
  • Vision production upgrade: NEDBD_DAG=1 with full ITC block history backfill

Built by Interchained LLC × Claude — maintained by the NEDB Hyperagent.
GitHub: Eth-Interchained/nedb | Mirror: aiassistsecure/nedb

Assets 2
Loading