Skip to content

Releases: nickooan/ntee-db

v0.5.0

Choose a tag to compare

@nickooan nickooan released this 30 Jul 07:32

What's Changed

Full Changelog: v0.4.0...v0.5.0

v0.4.0

Choose a tag to compare

@nickooan nickooan released this 29 Jul 07:10
62b47e8

What's Changed

Full Changelog: v0.3.0...v0.4.0

v0.3.0

Choose a tag to compare

@nickooan nickooan released this 29 Jul 05:40
c27acbf

What's Changed

Full Changelog: v0.2.0...v0.3.0

Release v0.2.0

Choose a tag to compare

@nickooan nickooan released this 12 Jul 04:00
e195a17

ntee-db v0.2.0

Counters land: ntee-db gains its first in-place write path — atomic int64 incr/decr that never grow the log — plus a stricter indexing model: secondary indexes now cover JSON-object values only; immediate values are primary-key-only.


🚀 Highlights

New: atomic int64 counters — Incr in core, incr/decr on the server. Redis-style semantics: a missing key initializes to 0 before the delta applies, the post-operation value comes back in the response, delta defaults to 1 (incr hits, decr hits 5, negative deltas work). Counters are a distinct value type — incr on any other value fails with ErrNotCounter, overflow past int64 fails with ErrCounterOverflow leaving the value untouched, and a put demotes a counter to a plain value. Int64 only, by design: no floats (store scaled integers — cents, not dollars).

Zero log growth for hot counters — the first in-place update in the engine. Counter values are stored fixed-width (explicit sign + 19 zero-padded digits — every int64 has exactly one such form), so an increment overwrites the digits in place at the same file offset through a new dedicated O_RDWR handle. No append per update, no compaction debt: in the benchmark run, 440,000 incr/decr ops left exactly one 48-byte record in main.jsonl. Torn-write safe by construction — a crash mid-patch can only mix old/new digit characters, never corrupt the line, so recovery is never tricked into truncating records after it.

Indexes now cover JSON-object values only. Immediate values — strings, numbers, booleans, arrays, binary — are plain key:value pairs addressed by primary key alone: Extract functions are never executed over them, and PutIndexed/putx reject explicit index values for one (an error; nothing written). One consistent model: an index entry always points at a structured record. This is also what makes counter patches unconditionally safe — nothing can derive index state from an immediate value, so increments take the in-place fast path even on schemas with jsonPath indexes.


Server features

  • New commands: incr <pk> [delta] and decr <pk> [delta] — write commands (serialized, auth like put), response is the new value as a JSON number: {"ok":true,"result":42}. incr <pk> 0 is the read idiom (plain get returns the raw 20-char fixed-width string).
  • Argument safety: non-integer deltas (1.5, abc) rejected with a usage error before reaching the store; decr of -9223372036854775808 rejected (negation overflow).
  • Measured (M2 Pro, loopback, single connection, schema with two jsonPath indexes): incr ~122k ops/s pipelined (8.2 µs/op) / ~32k ops/s sequential; get ~142k ops/s pipelined; put ~85k ops/s pipelined. Counters beat puts on the wire — smaller records, no index derivation.

Core improvements

  • New APIs: Incr(key, delta int64) (int64, error) (decrement = negative delta), errors ErrNotCounter / ErrCounterOverflow; new benchmarks BenchmarkIncr / BenchmarkIncrDurable.
  • Record format: new "c":true flag marks counter records explicitly — a value that merely looks like the fixed-width format is never treated as a counter.
  • Fail-safe fallback: if a counter record's on-disk layout doesn't match expectations byte-for-byte, the update falls back to a normal append — correctness over speed, always.
  • Handle hygiene: the three main.jsonl fds (append log / shared reader / counter patcher) are now opened and closed as one unit (openMainHandles), used identically by Open and the compaction swap; each handle keeps least privilege (the append fd physically cannot overwrite existing records, the reader cannot write at all).
  • No regressions: get untouched (1.15 µs); put unchanged (2.4 µs — object values pay one extra byte-check); core Incr 2.7 µs in-place; with SyncEveryWrite every write type shares the same ~ms fsync floor.

Compatibility notes

  • Behavior change: PutIndexed/putx/PutBatch with explicit index values on a non-object value now fail (previously allowed). Wrap such payloads in an object envelope if they must be indexed. Extract functions likewise no longer run over non-object values.
  • Enforcement is write-time only: existing stores open unchanged, legacy records indexed under the old rules still load and are preserved by Compact; they age out as records are rewritten.
  • Binaries older than v0.2.0 read counter records as plain 20-char strings (unknown "c" field ignored); an old binary's Compact would drop the counter flag — don't downgrade after using counters.
  • The JS binding (ntee-db on npm) is API-unaffected; exposing Incr through the FFI binding is planned as a follow-up.
    `

Release v0.1.0

Choose a tag to compare

@nickooan nickooan released this 11 Jul 11:46

ntee-db v0.1.0

The first minor release: ntee-db grows from an embedded library into a small database system — a standalone server, online compaction, and blob-space reclamation — while keeping every hot-path benchmark unchanged.


🚀 Highlights

New: nteedb-server — run ntee-db as a standalone daemon. A redis/memcached-style TCP server over the core: memcached-inspired text protocol, every response a single line of pure JSON, pipelining, one goroutine per connection with truly parallel reads. Pure Go, cross-compiles to a ~3 MB static binary via the new root ./build.shscp and run.

Online compaction — reads never stop. Compact/Reindex used to hold the exclusive lock for their whole rewrite; reads now flow throughout, with writes pending only until the rebuild finishes. Reads arriving mid-compaction went from ~4.4 ms to 1.25 µs (~3,500× faster) on the contention benchmark.

Blob-space reclamation. Orphaned blobs (overwritten/deleted large values) used to accumulate forever. New core APIs split mechanism from policy: BlobsRelieve() rewrites blobs into a fresh generation file and drops orphans (crash-safe — the main-log rename stays the single atomic commit point), and BlobUsage() measures total/live/orphaned bytes so callers decide when to act.

Auto-compaction with user-tunable policy. The server watches dead space and reclaims it in the background — main log and blobs as independent triggers — fully configurable in schema.json ("autoCompact": true or an options object: intervalSeconds, mainRatio, mainMinBytes, blobsRelieve on/off, blobRatio, blobMinRelieveDataSize). Off by default.


Server features

  • Protocol: get/getm/has/scan, index queries (ix, ixh, ixp, ixr, ixrec), writes (put inline or length-prefixed binary-safe frames, putx with explicit index values, del, rml/rmg range deletes), plus stats, dropped, prospective.
  • Auth, modeled on redis/memcached: no-auth (loopback-only via protected mode), single password (NTEEDB_AUTH), or user:password:role file. Two roles — admin alone may run compact, reindex, relieve. Pre-auth allowlist matches redis (auth, hello, quit; ping requires auth; hello hides the index schema until authenticated).
  • Ops: stats reports liveBytes (dead-space ratio at a glance), autoCompacts, blobCompacts; graceful shutdown writes the boot hint; clear startup errors (locked store, protected mode).
  • Startup: -schema schema.json (same shape as the JS open options, with jsonPath-derived indexes), -addr, -auth/-auth-file, -idle.

Core improvements

  • Compaction gate: Compact/Reindex/BlobsRelieve exclude only writers during the long rebuild; the exclusive lock is held just for the atomic file swap.
  • Blob generations: each blob ref names its file generation (blobs.dat, blobs.N.dat) — self-describing, crash-consolidating, byte-compatible with existing stores.
  • New APIs: LiveBytes(), BlobUsage(), BlobsRelieve().
  • No regressions: put/get/scan/batch/boot benchmarks statistically unchanged vs 0.0.x; Open gains <1 ms (blob-generation discovery); compaction remains ~4 µs per live record (68 ms @ 10k records, measured M2 Pro).

Compatibility notes

  • Existing stores open unchanged; all on-disk formats are backward compatible.
  • After the first BlobsRelieve on a store, its blob file moves to a new generation — binaries older than v0.1.0 won't resolve those refs. Don't downgrade after relieving (or run stores side-by-side across versions).
  • The JS binding (ntee-db on npm) embeds the core directly and is unaffected API-wise; its prebuilt binaries will pick up the online-compaction engine in the next npm release.
    `