VectorDB.search() silently omits stored rows (~6-12%); db.get() still returns them, higher efSearch does not recover

[vidaunited]

Summary

VectorDB.search() intermittently omits a stored row from its results. The row is still in the store — db.get(id) returns it, vector and metadata intact — so the index has lost the node rather than the write having failed.

It is not a breadth-of-search limit: it reproduces with k and efSearch far above the row count, and re-running the same query or raising efSearch from 256 to 1024 returns the same short result. Once a row is lost for a given store, it stays lost.

Environment

ruvector	0.2.41
node	v22.23.0
npm	10.9.8
OS	Darwin 25.6.0 arm64 (macOS, Apple silicon)
native pkgs	@ruvector/core, @ruvector/rvf-node-darwin-arm64, etc. (default install)

Reproduction

Needs only npm i ruvector — no embedder, deterministic vectors.

// repro.cjs — node repro.cjs
const { VectorDB } = require('ruvector');
const fs = require('node:fs');
const os = require('node:os');
const path = require('node:path');

const DIM = 384;
const ROWS = 3;
const TRIALS = 200;

const unitVector = (seed) => {
  let x = (seed * 2654435761) % 2 ** 31;
  const v = new Float32Array(DIM);
  let norm = 0;
  for (let i = 0; i < DIM; i += 1) {
    x = (x * 1103515245 + 12345) % 2 ** 31;
    v[i] = x / 2 ** 30 - 1;
    norm += v[i] * v[i];
  }
  norm = Math.sqrt(norm);
  for (let i = 0; i < DIM; i += 1) v[i] /= norm;
  return v;
};

(async () => {
  let short = 0;
  const examples = [];
  for (let t = 0; t < TRIALS; t += 1) {
    const storagePath = path.join(fs.mkdtempSync(path.join(os.tmpdir(), 'rv-')), 's.db');
    const db = new VectorDB({ dimensions: DIM, distanceMetric: 'cosine', storagePath });

    for (let i = 0; i < ROWS; i += 1) {
      await db.insert({ id: `m${i}`, vector: unitVector(t * 100 + i), metadata: { tag: 'x' } });
    }

    const len = await db.len();
    const hits = await db.search({ vector: unitVector(t * 100), k: 64, efSearch: 256 });

    if (hits.length !== len) {
      short += 1;
      const seen = new Set(hits.map((h) => h.id));
      const missing = [...Array(ROWS).keys()].map((i) => `m${i}`).filter((id) => !seen.has(id));
      const byId = await Promise.all(missing.map((id) => db.get(id)));
      const retry = await db.search({ vector: unitVector(t * 100), k: 64, efSearch: 1024 });
      examples.push({
        trial: t, len, returned: hits.length, missing,
        getWorks: byId.every(Boolean), retryReturned: retry.length,
      });
    }
  }
  console.log(`short results: ${short}/${TRIALS}`);
  for (const e of examples.slice(0, 5)) console.log(' ', JSON.stringify(e));
})().catch((e) => { console.error(e); process.exit(1); });

Observed

short results: 12/200
  {"trial":7,"len":3,"returned":2,"missing":["m2"],"getWorks":true,"retryReturned":2}
  {"trial":51,"len":3,"returned":2,"missing":["m1"],"getWorks":true,"retryReturned":2}
  {"trial":83,"len":3,"returned":1,"missing":["m1","m2"],"getWorks":true,"retryReturned":1}
  {"trial":93,"len":3,"returned":2,"missing":["m2"],"getWorks":true,"retryReturned":2}
  {"trial":105,"len":3,"returned":2,"missing":["m2"],"getWorks":true,"retryReturned":2}

len is 3 in every case, getWorks is always true, and retryReturned (efSearch 1024) never recovers the row. Trial 83 lost two of three rows.

Expected: with k=64 and only 3 rows stored, search() returns all 3.

Rates measured

vectors	rate
deterministic pseudo-random (above)	12/200 (6%)
real sentence embeddings, OnnxEmbedder all-MiniLM-L6-v2, 384d	3/25 and 2/25 (~8–12%)

The rate appears higher with real embeddings, which are clustered, than with near-orthogonal random vectors — consistent with a neighbour-linking problem during index construction, though I have not read the index code.

Repair

Deleting and re-inserting a lost row restores it: 4 of 4 cases recovered, all within a single pass. So the stored vector is fine and only the graph entry is wrong.

const row = await db.get(id);
await db.delete(id);
await db.insert({ id, vector: row.vector, metadata: row.metadata });

Detection, if useful to anyone hitting this: probe each known id with its own vector. If a nearest-neighbour search for a vector does not return the row holding that exact vector, the index has lost it.

Impact

Silent, and it survives every obvious check. insert() resolves successfully, len() counts the row, get() returns it — only search omits it. Any code that enumerates by searching therefore operates on a random subset while reporting success. In our case an audit-trail aggregation folded ~90% of its rows and a compaction step skipped the same ~10%, both reporting completion.

Our workaround is to keep a roster row listing every id and enumerate with list() + get() instead of search, plus a verify()/reindex() pair using the probe and repair above. That covers enumeration; search() itself is still lossy, which for a vector database is the part that matters.

Happy to test a patch or provide more traces.

[vidaunited]

Follow-up: I built the repo and ran the benchmark suite to see whether anything existing would have caught this. Nothing does, and I think the reason is structural rather than an oversight.

benches/hnsw_search.rs cannot detect a lost node

It is the only benchmark covering the affected path, and it measures latency exclusively. The whole file is 56 lines; the relevant part:

db.insert_batch(vectors).unwrap();          // 1000 vectors, never verified
// ...
for k in [1, 10, 100].iter() {
    group.bench_with_input(BenchmarkId::from_parameter(k), k, |bench, &k| {
        bench.iter(|| {
            db.search(SearchQuery { vector: black_box(query.clone()), k: black_box(k),
                                    filter: None, ef_search: None }).unwrap()
        });
    });
}

The returned Vec is discarded. There is no recall computation, no brute-force oracle, and no assertion on result length — so a store that has silently dropped 12% of its nodes benchmarks faster, not worse, because there is less graph to traverse. The failure mode is invisible to, and mildly rewarded by, this measurement.

Two smaller things in the same file that would weaken it as a correctness check even if a length assertion were added: the dataset is vector[j] = (i + j) * 0.1, which is nearly collinear rather than distributed, and ef_search: None leaves the default in place, so it never exercises the parameter I varied while reproducing.

Results on my machine for reference (M-series, release, criterion 100 samples):

k	median	95% CI
1	4.75 µs	4.726 – 4.775
10	10.09 µs	10.047 – 10.140
100	24.20 µs	24.077 – 24.336

The suite is otherwise green

ruvector-lsm-ann, ruvector-temporal-coherence, ruvector-capgated and ruvector-matryoshka all pass their own acceptance thresholds. Notably ruvector-lsm-ann does compute recall@10 against a brute-force oracle and reports Baseline recall@10 = 1.0000 — but it exercises the LSM-ANN crate, not the core HNSW path that the Node VectorDB sits on. So the one suite with an oracle covers a different index.

A debug_assert-style check in hnsw_search that results.len() == min(k, inserted) would have caught the original report at essentially zero cost.

Possibly relevant to the root cause

The workspace vendors a patched hnsw_rs rather than using the published crate:

[patch.crates-io]
hnsw_rs = { path = "./patches/hnsw_rs" }

If the dropped node originates there, the bug may not reproduce against upstream hnsw_rs 0.3.3 — worth checking the fork's insertion path before looking further afield. I have not read that code, so this is a pointer rather than a diagnosis.

Version detail

More precise than my original report: the npm CLI is 0.2.41 but @ruvector/core reports 0.1.29, and the two version independently. The reproduction was against that pair. (Minor aside: ruvector info prints Core Version: [object Object] — it interpolates core.version without calling it, since it is exported as a function.)