Concurrency hardening for BfTree parallel workloads

[JordanMaples]

Problem

NeighborProvider::append_vector performs a read-modify-write cycle (read existing neighbors → append → write back) without synchronization. Under concurrent mutation, the last writer wins and silently drops the other's edges. Stress testing shows edge loss under multi-thread contention on a single vertex.

Additionally, the dual-store set_element wrote quant before full-precision, meaning if the full-precision write failed after quant succeeded, it would leave a quantized ghost with no backing data. And delete did not clean up the quant store entry.

Solution

Striped Mutex table for per-vertex synchronization

• StripedLocks: a fixed-size table of Mutex<()> stripes (diskann-bftree/src/locks.rs).
• Stripe count is derived from hardware parallelism: (cpus * 4).next_power_of_two().max(64) — constant memory regardless of dataset size, suitable for billion-scale workloads.
• Vertex IDs map to stripes via Fibonacci multiply-shift hashing (floor(2^64 / phi)), keeping false contention negligible.
• Each mutating op acquires exactly one stripe lock for the target vertex, does its work, and drops the guard — no nested locking, so deadlock is structurally impossible.

Why striped instead of per-vertex locks?

BfTree's value proposition is spilling to disk for datasets exceeding RAM. At 1B vectors, per-vertex locks would consume gigabytes of in-memory locks — defeating the purpose. Striped locks keep memory constant at the cost of rare false contention.

Reads are lock-free

get_neighbors and get_vector acquire no stripe lock. They rely on bf-tree's internal thread safety and the fact that DiskANN search is approximate by design: a reader may briefly observe a partially-updated neighbor list, which affects recall momentarily rather than correctness. Writers (set_element, delete, set_neighbors/write_neighbors, append_vector/write_append) serialize on the stripe.

Dual-store write ordering

• Full-precision is now written first in set_element.
• Failure mode is benign: vector reachable in full-precision but not yet in quantized search (rather than a quantized ghost with no backing data).

Quant store cleanup on delete

• New DeleteQuant trait: real impl for QuantVectorProvider (deletes the quant entry), no-op for NoStore.
• delete now removes both full-precision and quant entries under the stripe lock. Neighbor-topology cleanup remains in the upper DiskANNIndex layer (drop_adj_list).

Testing

Three new concurrent stress tests in diskann-bftree/src/neighbors.rs:

• test_concurrent_append_no_lost_edges — 8 threads × 10 edges to the same vertex; asserts every edge survives.
• test_concurrent_append_independent_vertices — 8 threads appending to disjoint vertex ranges; asserts total edge count.
• test_concurrent_read_write_consistency — mixed readers + writers on the same vertex; validates no torn reads.

[Copilot]

Pull request overview

This PR hardens diskann-bftree for concurrent workloads by synchronizing neighbor-list mutations (preventing lost updates), correcting dual-store (full + quant) lifecycle behavior, and adding concurrent stress tests to validate the new behavior.

Changes:

• Add striped per-vertex RwLock synchronization to NeighborProvider (read locks for reads, write locks for mutations) and introduce concurrent stress tests.
• Adjust hard-delete behavior to also delete quantized vectors when present via a small abstraction over the quant store.
• Reorder dual-store writes in set_element to write full-precision first, then quantized.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.

File	Description
diskann-bftree/src/neighbors.rs	Adds striped locking around neighbor-list read/modify/write and adds concurrent stress tests.
diskann-bftree/src/provider.rs	Adds quant-store cleanup on delete via DeleteQuant and reorders full/quant write sequencing in set_element.
diskann-bftree/src/quant.rs	Adds quant-store delete support used by hard-delete cleanup.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[harsha-simhadri]

Jordan, what is the perf impact of this change? Is ingestion speed the same as before? thanks

[codecov-commenter]

Codecov Report

❌ Patch coverage is 98.99329% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.85%. Comparing base (53257e9) to head (b8ca306).

Files with missing lines	Patch %	Lines
diskann-bftree/src/provider.rs	88.46%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1158      +/-   ##
==========================================
+ Coverage   89.79%   90.85%   +1.06%     
==========================================
  Files         488      489       +1     
  Lines       93312    93575     +263     
==========================================
+ Hits        83791    85022    +1231     
+ Misses       9521     8553     -968     

Flag	Coverage Δ
miri	90.85% <98.99%> (+1.06%)	⬆️
unittests	90.82% <98.99%> (+1.36%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
diskann-bftree/src/lib.rs	59.09% <ø> (ø)
diskann-bftree/src/locks.rs	100.00% <100.00%> (ø)
diskann-bftree/src/neighbors.rs	96.67% <100.00%> (+2.58%)	⬆️
diskann-bftree/src/quant.rs	89.12% <100.00%> (+0.18%)	⬆️
diskann-bftree/src/provider.rs	90.45% <88.46%> (-0.11%)	⬇️

... and 40 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[JordanMaples]

@copilot give it another review.

[Copilot]

@copilot give it another review.

I re-reviewed and found one remaining concurrency issue: beam expansion paths were reading neighbors without taking the striped read lock. I fixed both call sites in commit 562c341 by acquiring self.provider.locks.read(n.into_usize()) before get_neighbors.

Validation run:

• cargo test -p diskann-bftree --profile ci ✅
• cargo clippy -p diskann-bftree --all-targets --config 'build.rustflags=["-Dwarnings"]' -- -D warnings ✅
• Copilot code review ✅ (no additional comments)
• CodeQL scan timed out in the validation tool (no rerun per tool guidance).

[JordanMaples]

@harsha-simhadri after chatting a bit with Mark and and changing the approach to locking in the provider. I ran a few more wiki100k streaming benchmarks on my devbox instance it looks like the perf overhead is negligible.

[harsha-simhadri]

Left some comments inline. Ack on perf numbers.

[harsha-simhadri]

Left some comments inline. Ack on perf numbers.