Summary
@ruvector/ruvllm@2.6.0's embed()/similarity produce non-discriminative embeddings: unrelated text pairs frequently score a higher cosine similarity than genuine paraphrase pairs. This makes the package unusable for its stated purpose (semantic similarity / vector memory).
Environment
@ruvector/ruvllm@2.6.0 (current npm latest, confirmed no newer version published)- Native binary confirmed loaded (
@ruvector/ruvllm-darwin-arm64, not the JS/WASM fallback)
- macOS, arm64 (Apple Silicon), Node v26.4.0
How to reproduce
npm install @ruvector/ruvllm
npx ruvllm similarity "how to connect gmail to claude for email automation" \
"recipe for chocolate lava cake with raspberry coulis"
npx ruvllm similarity "how to connect gmail to claude for email automation" \
"steps for linking your gmail account with claude to send emails"
npx ruvllm similarity "connect gmail to claude" "connect gmail to claude"
Additional runs with bin/cli.js similarity against the native darwin-arm64 module (not the JS fallback) confirm this is not environment-specific:
| Pair type |
Similarity |
| Paraphrase |
97.85% |
| Paraphrase |
97.89% |
| Paraphrase |
99.53% |
| Identical |
100.00% |
| Unrelated |
91.24% |
| Unrelated |
97.78% |
| Unrelated |
98.75% |
| Unrelated |
99.02% |
Two of four unrelated pairs score higher than two of four paraphrase pairs. All eight values fall in a 91–100% band with no separation by actual semantic content.
Expected behavior
A trained sentence embedder should clearly separate unrelated text (low similarity, roughly 0–0.4 cosine on a typical MiniLM-class model) from paraphrases (high similarity, roughly 0.7–0.95), with unrelated pairs always scoring below genuine paraphrases.
Root cause (confirmed by reading source, examples/ruvLLM/src/embedding.rs, current main as of 2026-07-10)
There is no pretrained-weight-loading code path in this file at all — this is architecturally a random-projection character-bag embedder, not a case of pretrained weights failing to load:
Tokenizer::new() (~line 54) builds a character-level vocabulary (individual letters/digits/punctuation), not word- or subword-level tokens.EmbeddingService::new() (~lines 159–166) initializes embedding_matrix via rand::thread_rng().gen_range(-0.1..0.1) per token — freshly randomized on every process start, never loaded from any checkpoint. grep-ing the file for pretrained|load_weight returns no matches.mean_pooling() (~lines 327–352) averages random per-character token vectors plus sinusoidal position embeddings across the whole sequence, then L2-normalizes. Averaging many random unit-ish vectors drives all output embeddings toward a similar central direction (law of large numbers) — this is the mechanism that collapses cosine similarity toward 1.0 regardless of semantic content, for any sufficiently long input.
Sample use cases affected
- Semantic search / RAG retrieval ranking by cosine similarity
- Vector-memory deduplication or nearest-neighbor clustering
- Any downstream code that thresholds on similarity score to decide relatedness
Workaround
Use a real pretrained sentence embedder instead (e.g. @xenova/transformers/@huggingface/transformers with a MiniLM-class model) — confirmed to correctly order paraphrase pairs above unrelated pairs on the same test set.
Proposed fix
- Load actual pretrained weights (or drop the "embedding" framing until a trained model backs it), and/or switch tokenization to word/subword level.
- Add a regression test in CI asserting
sim(text, paraphrase(text)) > sim(text, unrelated) across a small fixed corpus, run on every release.
- Expose a debug/verbose flag reporting whether weights are pretrained vs. randomly initialized, so this failure mode is visible rather than silent.
Related issues
ruvnet/ruvector#316 ("Bug: embed() always returns hash vectors — breaks cosine similarity when ONNX is active") is related but distinct — it's scoped to the core ruvector package's ONNX embedder silently falling back to a hash vector. @ruvector/ruvllm's embedder has no ONNX/pretrained path to fall back from; it's random by construction. Also loosely related for context: #523, #402. None of these cover this specific defect.
Summary
@ruvector/ruvllm@2.6.0'sembed()/similarityproduce non-discriminative embeddings: unrelated text pairs frequently score a higher cosine similarity than genuine paraphrase pairs. This makes the package unusable for its stated purpose (semantic similarity / vector memory).Environment
@ruvector/ruvllm@2.6.0(current npmlatest, confirmed no newer version published)@ruvector/ruvllm-darwin-arm64, not the JS/WASM fallback)How to reproduce
Additional runs with
bin/cli.js similarityagainst the native darwin-arm64 module (not the JS fallback) confirm this is not environment-specific:Two of four unrelated pairs score higher than two of four paraphrase pairs. All eight values fall in a 91–100% band with no separation by actual semantic content.
Expected behavior
A trained sentence embedder should clearly separate unrelated text (low similarity, roughly 0–0.4 cosine on a typical MiniLM-class model) from paraphrases (high similarity, roughly 0.7–0.95), with unrelated pairs always scoring below genuine paraphrases.
Root cause (confirmed by reading source,
examples/ruvLLM/src/embedding.rs, currentmainas of 2026-07-10)There is no pretrained-weight-loading code path in this file at all — this is architecturally a random-projection character-bag embedder, not a case of pretrained weights failing to load:
Tokenizer::new()(~line 54) builds a character-level vocabulary (individual letters/digits/punctuation), not word- or subword-level tokens.EmbeddingService::new()(~lines 159–166) initializesembedding_matrixviarand::thread_rng().gen_range(-0.1..0.1)per token — freshly randomized on every process start, never loaded from any checkpoint.grep-ing the file forpretrained|load_weightreturns no matches.mean_pooling()(~lines 327–352) averages random per-character token vectors plus sinusoidal position embeddings across the whole sequence, then L2-normalizes. Averaging many random unit-ish vectors drives all output embeddings toward a similar central direction (law of large numbers) — this is the mechanism that collapses cosine similarity toward 1.0 regardless of semantic content, for any sufficiently long input.Sample use cases affected
Workaround
Use a real pretrained sentence embedder instead (e.g.
@xenova/transformers/@huggingface/transformerswith a MiniLM-class model) — confirmed to correctly order paraphrase pairs above unrelated pairs on the same test set.Proposed fix
sim(text, paraphrase(text)) > sim(text, unrelated)across a small fixed corpus, run on every release.Related issues
ruvnet/ruvector#316("Bug: embed() always returns hash vectors — breaks cosine similarity when ONNX is active") is related but distinct — it's scoped to the coreruvectorpackage's ONNX embedder silently falling back to a hash vector.@ruvector/ruvllm's embedder has no ONNX/pretrained path to fall back from; it's random by construction. Also loosely related for context:#523,#402. None of these cover this specific defect.