v0.7.0 — startup, memory and concurrency

Performance release. The work in 0.7.0 is about what the server holds in memory and how long it takes to be ready, not about what it answers. Every change is either identical in result to 0.6.3 or opt-in, so an operator who upgrades and changes nothing gets the same answers with a smaller footprint — with one narrow exception, stated under Compatibility.

Startup no longer reads the corpus out of SQLite

Startup used to decode every stored blob into the search matrices: linear in the corpus, peaking at about twice the payload, and the reason a million-row corpus of 768 dimensions does not start on an 8 GB host. The multiply that answers a query was never the cost; getting the matrix into memory was.

The vectors now live in a contiguous file next to the index database (magic, version, a JSON header carrying a watermark and the per-group ids, 64-byte-aligned float32 blocks). At startup the file is mapped rather than parsed, a rows-only reconciliation against SQLite decides which mapped rows are still live, and anything written above the watermark is read from SQLite as before. SQLite remains the durable record; the file is a cache that can be deleted at any time.

scripts/bench_sidecar_startup.py --rows 100000	time to first search	peak RSS
SQLite	0.23 s	1.1 GiB
sidecar	0.09 s	402 MiB

Default EMBEDDING_SIDECAR=auto, but nothing happens below EMBEDDING_SIDECAR_MIN_ROWS (10,000), so small deployments are untouched. EMBEDDING_SIDECAR=off keeps the previous path exactly. The file is rebuilt when the rows written since the last build exceed EMBEDDING_SIDECAR_MAX_TAIL (0.25) of it, and at shutdown when it is behind. cembedding-sidecar build|status drives it by hand. Design and the two things implementation found — SQLite reusing a deleted rowid, and two segments meaning two BLAS calls — are in docs/CONTIGUOUS_SIDECAR_DESIGN.md.

One resident matrix per namespace

The index kept every vector twice: a per-item array map plus a stacked search matrix that every write discarded and the next search rebuilt, which made write-then-search linear in the namespace. There is now one float32 matrix per namespace and dimension with spare capacity: appends are amortised constant time, an overwrite touches one row, a removal flips a liveness flag, and compaction runs once tombstones outnumber live rows. Row order never changes, so results and tie order are identical — verified by running the old and new index side by side over upserts, removals, re-adds, dimension switches and exact ties, comparing ids, order and scores with ==.

20k x 384	30 write+search rounds	resident (payload 29.3 MiB)
0.6.3	153 ms	58.7 MiB
0.7.0	17 ms	48.0 MiB

Concurrent embed requests share a model run

Each local provider ran one forward pass per request behind a lock, so under load every request paid for its own pass while the rest queued: throughput was flat at about 150 req/s from 1 to 32 clients. Requests that arrive while a pass is in flight now accumulate and share the next one. Nothing waits for a batch to fill, a lone request is never split, and a failing pass fails exactly the requests in it.

/embed, one text per request, jina-v5-nano fp32, CPU	before p50 / p99 / rps	after p50 / p99 / rps
1 client	7.3 / 7.6 / 137	7.3 / 7.8 / 136
8 clients	54.8 / 58.4 / 147	29.0 / 30.1 / 280
32 clients	219.5 / 272 / 150	89.5 / 113.6 / 358

EMBEDDING_MAX_BATCH (default 64) caps how many texts a merged pass carries; 0 restores one pass per request, measured identical to 0.6.3.

Opt-in fp16 / int8 weights, and why the default did not change

The jina-v5-nano repository ships the same graph at several precisions. EMBEDDING_MODEL_VARIANT selects fp32 (default), fp16 or int8, and cembedding-download-model --model jina-v5-nano --variant <v> fetches one ahead of time.

variant	download	resident	single query (laptop / 4-core x86)	cosine to fp32 median / worst	top-10 agreement
fp32	810 MB	906 MB	5.8 / 71 ms	—	—
fp16	405 MB	933 MB	6.3 ms	1.0 / 1.0	1.00
int8	236 MB	628 MB	46 / 319 ms	0.9998 / 0.9981	0.97

fp16 saves nothing at run time on CPU, and int8 saves about 280 MB but costs four to eight times more per query on both architectures measured. That is worth having for memory-bound hosts, not as a default. Vectors already stored were produced at fp32, so changing the variant makes queries and corpus mixed-precision — read the README section before setting it.

Configurable ONNX session

The three ONNX providers built their session with no options, in two places. There is now one construction path, with ONNX_INTRA_OP_THREADS (default 0 = the runtime decides) and ONNX_GRAPH_OPT_LEVEL (default all). Both defaults reproduce the runtime's own behaviour, so nothing changes unless set. The thread knob exists for hosts where the runtime reads the wrong core count — a container quota, a shared box — and not as a speed-up: over-subscribing measured 2.3 times slower.

Compatibility

Additive or identical in result, except for one case worth stating plainly: when the sidecar splits a namespace into a mapped segment and an in-memory tail, the two segments are two BLAS calls, and a kernel chosen for a small segment can accumulate in a different order. Scores are rounded to four decimals in the payload, so the only observable effect is the order among rows that tie exactly — identical vectors. EMBEDDING_SIDECAR=off avoids it entirely.

On accelerator execution providers, the low bits of an embedding (about 1e-6) have always depended on the shape of the batch a text ran in; with request merging a single-text request under load can now see that same variance. EMBEDDING_MAX_BATCH=0 restores the previous behaviour exactly.

No schema change, no configuration change required, no endpoint change.

Verification

113 tests (54 in 0.6.3). The two behaviour-preserving claims are held by equivalence fixtures that compare ids, order and scores with == against the previous implementation, plus a containment property against SQLite after random writes. New invariants were each checked against a deliberate mutation; where a mutant survived — the ORDER BY rowid clauses that SQLite already satisfies through the integer primary key — it is stated rather than claimed as covered.

Full changelog: v0.6.3...v0.7.0