Reconcile symmetric vs asymmetric bge-small convention between Lattice embedding providers

[ruvnet]

Summary

PR #648 (LatticeEmbedding in crates/ruvector-core) and PR #651 (LatticeWasmEmbeddings in npm/packages/ruvector-extensions) both add a "Lattice"-backed bge-small embedding provider, but they disagree on whether the model is symmetric or asymmetric:

• feat(extensions): add LatticeWasmEmbeddings provider #651 (WASM): treats bge-small as symmetric — no query-side instruction prefix is applied.
• Optional LatticeEmbedding provider for ruvector-core (feature: lattice-embeddings) #648 (Rust core): treats bge-small-en-v1.5 as asymmetric — applies BGE's retrieval-instruction prefix in embed_query.

Same declared model, same 384 dimensions, different vector spaces. A user who indexes a corpus with one Lattice provider and queries with the other gets no error — just silently degraded retrieval, since query vectors and passage vectors won't be prefixed consistently.

Suggested fix

Reconcile which convention is authoritative for bge-small/bge-small-en-v1.5 across both providers — likely by matching whatever the upstream lattice-embed crate does natively for this model — so the two "Lattice embeddings" surfaces stay interchangeable for users who expect a shared name to mean a shared embedding space.

Secondary, non-blocking notes from review (bundle in if convenient)

• Model-name registry mismatch: feat(extensions): add LatticeWasmEmbeddings provider #651 uses short names (bge-small), Optional LatticeEmbedding provider for ruvector-core (feature: lattice-embeddings) #648 delegates to lattice-embed's FromStr table (bge-small-en-v1.5) — an id valid in one throws "unknown model" in the other.
• Normalization-guarantee mismatch: feat(extensions): add LatticeWasmEmbeddings provider #651 asserts L2-normalized output; Optional LatticeEmbedding provider for ruvector-core (feature: lattice-embeddings) #648 doesn't document a normalization guarantee. Matters if either vector set is used with a non-cosine (dot-product) index.

cc @ohdearquant — thanks again for both PRs, just want to make sure the two surfaces stay safely interchangeable before folks build on them.

[ohdearquant]

on it

[ohdearquant]

Comment authored by Claude (Anthropic agent) on behalf of @ohdearquant.

Thanks for catching this — you're right that the two providers need to share one embedding space, and the reconciliation is straightforward because upstream lattice-embed is unambiguous here.

Authoritative convention: bge-small-en-v1.5 is asymmetric in lattice-embed. EmbeddingModel::query_instruction() returns the BGE-v1.5 retrieval instruction ("Represent this sentence for searching relevant passages: ") for queries, and document_instruction() returns None, so passages are embedded as raw text. That makes the ruvector-core provider from #648 the one that matches upstream, and #663 brings the WASM provider in line with it rather than the other way around.

Opened #663 to reconcile all three points:

1. Query prefix: LatticeWasmEmbeddings gains an embedQuery() that applies the BGE query instruction; embedText/embedTexts stay raw for passages, matching LatticeEmbedding::embed_query() / embed() in core.
2. Name registry: the WASM provider now accepts the same alias set as lattice-embed's FromStr (bge-small, bge-small-en, bge-small-en-v1.5, BAAI/bge-small-en-v1.5, case/underscore-insensitive), so an id valid in one provider is valid in the other.
3. Normalization: documented the L2-normalization guarantee on the core provider as well — lattice-embed's encode path normalizes unconditionally, so both vector sets are safe for dot-product indexes.

Test evidence and one disclosed limitation (the ruvector-extensions package currently has no CI workflow, so the TS results in the PR are from local runs) are in the PR body.