Releases: nickooan/ntee-db
Release list
v0.5.0
v0.4.0
v0.3.0
Release v0.2.0
ntee-db v0.2.0
Counters land: ntee-db gains its first in-place write path — atomic int64 incr/decr that never grow the log — plus a stricter indexing model: secondary indexes now cover JSON-object values only; immediate values are primary-key-only.
🚀 Highlights
New: atomic int64 counters — Incr in core, incr/decr on the server. Redis-style semantics: a missing key initializes to 0 before the delta applies, the post-operation value comes back in the response, delta defaults to 1 (incr hits, decr hits 5, negative deltas work). Counters are a distinct value type — incr on any other value fails with ErrNotCounter, overflow past int64 fails with ErrCounterOverflow leaving the value untouched, and a put demotes a counter to a plain value. Int64 only, by design: no floats (store scaled integers — cents, not dollars).
Zero log growth for hot counters — the first in-place update in the engine. Counter values are stored fixed-width (explicit sign + 19 zero-padded digits — every int64 has exactly one such form), so an increment overwrites the digits in place at the same file offset through a new dedicated O_RDWR handle. No append per update, no compaction debt: in the benchmark run, 440,000 incr/decr ops left exactly one 48-byte record in main.jsonl. Torn-write safe by construction — a crash mid-patch can only mix old/new digit characters, never corrupt the line, so recovery is never tricked into truncating records after it.
Indexes now cover JSON-object values only. Immediate values — strings, numbers, booleans, arrays, binary — are plain key:value pairs addressed by primary key alone: Extract functions are never executed over them, and PutIndexed/putx reject explicit index values for one (an error; nothing written). One consistent model: an index entry always points at a structured record. This is also what makes counter patches unconditionally safe — nothing can derive index state from an immediate value, so increments take the in-place fast path even on schemas with jsonPath indexes.
Server features
- New commands:
incr <pk> [delta]anddecr <pk> [delta]— write commands (serialized, auth likeput), response is the new value as a JSON number:{"ok":true,"result":42}.incr <pk> 0is the read idiom (plaingetreturns the raw 20-char fixed-width string). - Argument safety: non-integer deltas (
1.5,abc) rejected with a usage error before reaching the store;decrof-9223372036854775808rejected (negation overflow). - Measured (M2 Pro, loopback, single connection, schema with two
jsonPathindexes):incr~122k ops/s pipelined (8.2 µs/op) / ~32k ops/s sequential;get~142k ops/s pipelined;put~85k ops/s pipelined. Counters beat puts on the wire — smaller records, no index derivation.
Core improvements
- New APIs:
Incr(key, delta int64) (int64, error)(decrement = negative delta), errorsErrNotCounter/ErrCounterOverflow; new benchmarksBenchmarkIncr/BenchmarkIncrDurable. - Record format: new
"c":trueflag marks counter records explicitly — a value that merely looks like the fixed-width format is never treated as a counter. - Fail-safe fallback: if a counter record's on-disk layout doesn't match expectations byte-for-byte, the update falls back to a normal append — correctness over speed, always.
- Handle hygiene: the three
main.jsonlfds (append log / shared reader / counter patcher) are now opened and closed as one unit (openMainHandles), used identically byOpenand the compaction swap; each handle keeps least privilege (the append fd physically cannot overwrite existing records, the reader cannot write at all). - No regressions:
getuntouched (1.15 µs);putunchanged (2.4 µs — object values pay one extra byte-check); coreIncr2.7 µs in-place; withSyncEveryWriteevery write type shares the same ~ms fsync floor.
Compatibility notes
- Behavior change:
PutIndexed/putx/PutBatchwith explicit index values on a non-object value now fail (previously allowed). Wrap such payloads in an object envelope if they must be indexed.Extractfunctions likewise no longer run over non-object values. - Enforcement is write-time only: existing stores open unchanged, legacy records indexed under the old rules still load and are preserved by
Compact; they age out as records are rewritten. - Binaries older than v0.2.0 read counter records as plain 20-char strings (unknown
"c"field ignored); an old binary'sCompactwould drop the counter flag — don't downgrade after using counters. - The JS binding (
ntee-dbon npm) is API-unaffected; exposingIncrthrough the FFI binding is planned as a follow-up.
`
Release v0.1.0
ntee-db v0.1.0
The first minor release: ntee-db grows from an embedded library into a small database system — a standalone server, online compaction, and blob-space reclamation — while keeping every hot-path benchmark unchanged.
🚀 Highlights
New: nteedb-server — run ntee-db as a standalone daemon. A redis/memcached-style TCP server over the core: memcached-inspired text protocol, every response a single line of pure JSON, pipelining, one goroutine per connection with truly parallel reads. Pure Go, cross-compiles to a ~3 MB static binary via the new root ./build.sh — scp and run.
Online compaction — reads never stop. Compact/Reindex used to hold the exclusive lock for their whole rewrite; reads now flow throughout, with writes pending only until the rebuild finishes. Reads arriving mid-compaction went from ~4.4 ms to 1.25 µs (~3,500× faster) on the contention benchmark.
Blob-space reclamation. Orphaned blobs (overwritten/deleted large values) used to accumulate forever. New core APIs split mechanism from policy: BlobsRelieve() rewrites blobs into a fresh generation file and drops orphans (crash-safe — the main-log rename stays the single atomic commit point), and BlobUsage() measures total/live/orphaned bytes so callers decide when to act.
Auto-compaction with user-tunable policy. The server watches dead space and reclaims it in the background — main log and blobs as independent triggers — fully configurable in schema.json ("autoCompact": true or an options object: intervalSeconds, mainRatio, mainMinBytes, blobsRelieve on/off, blobRatio, blobMinRelieveDataSize). Off by default.
Server features
- Protocol:
get/getm/has/scan, index queries (ix,ixh,ixp,ixr,ixrec), writes (putinline or length-prefixed binary-safe frames,putxwith explicit index values,del,rml/rmgrange deletes), plusstats,dropped,prospective. - Auth, modeled on redis/memcached: no-auth (loopback-only via protected mode), single password (
NTEEDB_AUTH), oruser:password:rolefile. Two roles —adminalone may runcompact,reindex,relieve. Pre-auth allowlist matches redis (auth,hello,quit;pingrequires auth;hellohides the index schema until authenticated). - Ops:
statsreportsliveBytes(dead-space ratio at a glance),autoCompacts,blobCompacts; graceful shutdown writes the boot hint; clear startup errors (locked store, protected mode). - Startup:
-schema schema.json(same shape as the JS open options, withjsonPath-derived indexes),-addr,-auth/-auth-file,-idle.
Core improvements
- Compaction gate:
Compact/Reindex/BlobsRelieveexclude only writers during the long rebuild; the exclusive lock is held just for the atomic file swap. - Blob generations: each blob ref names its file generation (
blobs.dat,blobs.N.dat) — self-describing, crash-consolidating, byte-compatible with existing stores. - New APIs:
LiveBytes(),BlobUsage(),BlobsRelieve(). - No regressions: put/get/scan/batch/boot benchmarks statistically unchanged vs 0.0.x;
Opengains <1 ms (blob-generation discovery); compaction remains ~4 µs per live record (68 ms @ 10k records, measured M2 Pro).
Compatibility notes
- Existing stores open unchanged; all on-disk formats are backward compatible.
- After the first
BlobsRelieveon a store, its blob file moves to a new generation — binaries older than v0.1.0 won't resolve those refs. Don't downgrade after relieving (or run stores side-by-side across versions). - The JS binding (
ntee-dbon npm) embeds the core directly and is unaffected API-wise; its prebuilt binaries will pick up the online-compaction engine in the next npm release.
`