Reindex: missing-vector detection probes chunks_vec row-by-row (O(total chunks) per reindex)

[samkeen]

What

The projection pass detects chunks that still need embedding with a LEFT JOIN against the chunks_vec vec0 virtual table:

SELECT c.note_b2id, n.path, c.id, c.text
FROM chunks c
JOIN notes n ON n.b2id = c.note_b2id
LEFT JOIN chunks_vec v ON v.chunk_id = c.id
WHERE v.chunk_id IS NULL
ORDER BY n.path, c.seq

To evaluate that join, sqlite-vec probes its shadow rowid table once per existing chunk:

SELECT id, chunk_id, chunk_offset FROM "main"."chunks_vec_rowids" WHERE rowid = ?

Observed

On a trivial 6-note vault, a reindex that embedded 0 chunks (everything already vectorized) still emitted 126 of 206 logged SQLite statements as these per-row chunks_vec_rowids lookups — 61% of all query traffic. Captured via B2_LOG_FILE:

126  SELECT id, chunk_id, chunk_offset FROM "main"."chunks_vec_rowids" WHERE rowid = ?

The probe count scales with the total number of chunks in the vault, and runs on every reindex regardless of how many chunks are actually new (0 here). Harmless at this scale (~4ms total), but on a large vault the "is this chunk already embedded?" check becomes O(total chunks) single-row virtual-table probes on every incremental reindex — potentially the dominant cost of a no-op reindex.

Why it matters

The projection/embedding split (specs/completed/projection-embedding-split.md) is meant to make the model-free project_vault pass cheap. A per-chunk probe into the vec shadow table on every reindex works against that: the cost of deciding there's nothing to embed grows with the vault, not with the delta.

To investigate

• Whether the pending-set can be derived without a per-row vec probe — e.g. maintain the "has vector" bit in the base chunks table (or a plain side table) that embed_vault updates, so detection is a normal indexed scan instead of a virtual-table join.
• Whether restricting the candidate set upstream (only chunks whose note changed this reindex) avoids the full-vault probe on incremental runs.
• Confirm the query plan / probe pattern with EXPLAIN QUERY PLAN on a larger vault before optimizing.

Repro

B2_LOG_FILE=./logs/debug.jsonl cargo run -p b2-cli -- -C <vault> reindex
jq -r '.sql' logs/debug.jsonl | sed -E 's/[0-9]+/N/g; s/"[^"]*"/"X"/g' | sort | uniq -c | sort -rn | head

Not a correctness bug — incremental still equals full rebuild. Pure scaling watch-item for the project_vault pass.