Design decision: pretrained (lattice-backed) embeddings — default-on or opt-in? (fix path for #655)

[ohdearquant]

Issue filed by an AI agent on behalf of @ohdearquant.

Context

#655 reports that @ruvector/ruvllm's embed()/similarity are non-discriminative — the
embedder is a random-init character-level matrix (examples/ruvLLM/src/embedding.rs), never
loaded from a checkpoint, so unrelated pairs often out-score paraphrases.

A fix is in progress that backs EmbeddingService with a real pretrained embedder —
ruvector_core::embeddings::LatticeEmbedding (already merged into ruvector-core behind the
lattice-embeddings feature; supports MiniLM, BGE-small/base/large, E5, Qwen3-Embedding). It's
API-compatible with the existing EmbeddingService surface, so napi.rs doesn't change.

The one real decision isn't code — it's distribution, and it's yours to make. Filing this to
ask before assuming.

The fork

A. Default-on. embed()/similarity produce real semantic embeddings out of the box —
fixes #655 for every user with zero opt-in.
Cost: a pretrained model (~90 MB MiniLM / ~130 MB BGE-small) becomes a runtime dependency of the
published package (download-on-first-use unless bundled); the feature build requires Rust ≥ 1.93
(lattice-embed 0.6, edition 2024); heavier build.

B. Opt-in (what the in-progress PR does today). Default build, footprint, and MSRV unchanged;
the real embedder is available when the lattice-embeddings feature is compiled in and a model is
selected (via config/env). A visibility flag (pretrained: bool on the stats) makes the
random-vs-pretrained state explicit so the #655 failure mode is never silent. Downside: users
hitting #655 have to opt in to get the fix.

If you want A (default-on), three sub-choices

1. Default model — MiniLM (smaller, symmetric, solid STS/similarity) vs BGE-small (asymmetric
   query/passage prefixing, stronger retrieval).
2. Bundle vs lazy-download — ship the weights inside the npm package (bigger install, works
   offline immediately) vs fetch on first embed() (small install, needs network on first run).
3. The random path — keep it as an explicitly-selected offline/no-download fallback, or drop
   the "embedding" framing there entirely (per ruvllm: embed()/similarity produce non-discriminative embeddings (random-init char-level embedder, no pretrained weights) #655's own third suggestion).

Ask

Which do you want — A or B, and if A, your calls on the three sub-choices? The PR is wired for B
right now; flipping it to default-on (plus wiring the chosen model and bundle/lazy behavior) is a
small change once you decide. Not going to change the default footprint of your published package
without your say-so.

[ohdearquant]

Numbers for the fork above

We asked A or B here on 14 July without having measured what the provider is actually worth, which
made the question harder to answer than it needed to be. Here is the measurement, plus a correction
to something we wrote in the issue body itself.

A correction to our own text, and it runs against us. This issue says the feature "requires
Rust >= 1.93 (lattice-embed 0.6, edition 2024)". That number is right as written, and the comment
in this repo's crates/ruvector-core/Cargo.toml carrying it is right too: published
lattice-embed 0.6.1 declares rust-version = 1.93, so cargo refuses to build it on anything
lower before compiling a line. What we measured is a narrower thing. The crate's source checks
clean on Rust 1.90.0, and getting that reading at all required --ignore-rust-version, which
measures the code rather than the manifest. So the declared floor is conservative for the code by
at least three minor versions, and it is still the floor you would actually hit today. None of
that changes the cost: 1.93 is what adopting this needs, and even edition 2024's hard floor of
1.85 sits eight minors above this repo's workspace 1.77.

Sub-choice 1, MiniLM versus BGE-small, now has a head-to-head. On BEIR SciFact, nDCG@10, one
harness, exact brute-force search, shuffled control on every arm:

arm	document window	nDCG@10
lattice-embed / bge-small-en-v1.5	512 tokens (native)	0.7141
lattice-embed / bge-small-en-v1.5	128 tokens (matched)	0.6773
all-MiniLM-L6-v2 via this repo's OnnxEmbedding	128 tokens (native)	0.6239

Read the window column first. The two providers do not see the same amount of each document, because
the MiniLM artifact's own tokenizer.json sets truncation.max_length = 128. The matched row is the
one that isolates model quality, and at a matched window BGE-small leads by 0.0534 rather than the
0.0902 the as-configured numbers suggest. Roughly 41% of the raw gap was window rather than model.
Full method, diagnostics, controls and caveats are below.

Sub-choice 2, bundle versus lazy-download. We can only contribute the size: bge-small-en-v1.5 is
128.2 MiB across four files as cached, of which model.safetensors is 133,466,304 bytes. Whether
that belongs inside the published package is a distribution call we are not going to make from
outside.

Sub-choice 3 is untouched by any of this, and we want to be explicit rather than let a number drift
across a boundary. The zero-scoring arm in our benchmark is ruvector-core's HashEmbedding, which
is a different piece of code from the random-init character-level embedder in examples/ruvLLM. Our
result says nothing about the latter, and we are not going to let one stand in for the other.

Our answer to the fork itself: B, opt-in. Not because the provider measures badly, it measures
well, but because the Rust-version cost lands on every consumer of the default build and it is a cost
we impose rather than one you chose. B keeps that contained to people who opt in and leaves your
default footprint and MSRV where they are. If you would rather have A, the sub-choice-1 evidence above
is the part that was missing and the rest is your call.

What follows is the full measurement so the numbers above can be checked rather than taken on trust.

---

The measurement

BEIR SciFact, nDCG@10, on one harness. lattice-embed 0.6.1 is the version this repo pins, and
the MiniLM arm runs through this repo's own OnnxEmbedding rather than a reimplementation.

arm	document window	nDCG@10	distinct top-1	mean pairwise cosine
lattice-embed 0.6.1	512 tokens (native)	0.7141	265 / 300	0.6282
lattice-embed 0.6.1	128 tokens (matched)	0.6773	262 / 300	0.6076
all-MiniLM-L6-v2 via OnnxEmbedding	128 tokens (native)	0.6239	260 / 300	0.0992
HashEmbedding	n/a	0.0000	33 / 300	0.9654
shuffled control, lattice vectors, 512	512 tokens	0.0000	265 / 300	0.6372
shuffled control, lattice vectors, 128	128 tokens	0.0000	262 / 300	0.6164
shuffled control, MiniLM vectors	128 tokens	0.0000	260 / 300	0.1124

Read the window column before the score column. The two providers do not see the same amount of
each document, and comparing them without saying so would be the more flattering presentation
rather than the true one.

OnnxEmbedding embeds the first 128 wordpiece tokens of each document, because the model
artifact it fetches sets truncation.max_length = 128 in its tokenizer.json. That was measured
rather than read off the config: two texts sharing a 240-word prefix and differing only afterwards
embed to bit-identical vectors, while a pair differing inside the first 128 tokens does not.
lattice-embed embeds up to 512 for this model. Measured against lattice's tokenizer, 5073 of
the 5183 documents exceed 128 tokens, so on the native rows the two providers differed on almost the
whole corpus rather than on a handful of long outliers.

Hence two lattice rows. The matched row re-runs the same harness with every document cut to the
same 128-token window, at token granularity using the tokenizer lattice-embed itself loads, and is
the row that isolates model quality. The native rows are what each provider produces as
configured today, which is the question a user actually faces. They answer different things and both
belong here.

At the matched window the distance between the two providers is 0.0534 rather than the 0.0902
the native rows show, so roughly 41% of the unmatched difference was window rather than model. The
remainder is reported here without being attributed to a cause. The two models tokenize against
different vocabularies, so 126 tokens covers slightly different amounts of text on each side; they
run on different runtimes; and the converse experiment, raising the ONNX arm to a 512-token window,
was not run. How much of the 0.0534 is model rather than method is therefore open.

The matched run's truncation control is a hard gate rather than a printed diagnostic, and it fired
once. The first attempt built its shared prefix long enough to exceed 512 tokens as well, so the
uncut pair embedded identically too, the control failed its own discrimination check, and the
harness exited without printing a score. The prefix was shortened to sit between the two windows and
the run repeated. Only the second run's number is in the table above, and nothing else about the run
was changed to get the control to pass.

Every arm carried conditions written into the harness source before it ran, the matched re-run
included: lattice ≥ 0.60, hash < 0.10, shuffled control < 0.05. The lattice threshold was set below
the published figure for this model so the result would not depend on recalling that figure
correctly.

Method, so it can be checked rather than taken on trust:

• BEIR SciFact from the public UKP mirror, unmodified. 5183 documents, 300 test queries graded by
  qrels/test.tsv. Document text is title and body concatenated, the BEIR convention for this set.
• The lattice arm resolves to BAAI/bge-small-en-v1.5, 384 dimensions, with embed_query for
  queries so the BGE retrieval instruction is applied and embed_passage for documents. See the
  deployment note below, because that asymmetry matters at the call site.
• The ONNX arm resolves to sentence-transformers/all-MiniLM-L6-v2 through OnnxEmbedding, used
  exactly as the module docs use it: raw text on both sides, since MiniLM is symmetric and the
  EmbeddingProvider trait offers a single embed. Each model therefore got the convention it was
  trained under, and neither side was handicapped by the other's.
• Exact brute-force search over all 5183 documents. No ANN index, so index recall is not a
  confound in either direction. This measures the embeddings alone.
• nDCG@10 with binary relevance, using the same formula ruvector-sota-bench implements in
  src/runners/mteb.rs, for queries with at least one relevant document.
• Zero embed failures, and zero queries excluded for having no relevant document, on every arm.
• The lattice arm was run twice independently and produced identical scores.

On the arms reading exactly 0.0000. A round zero is also the shape a broken harness makes, so
every arm prints diagnostics that tell the cases apart, and here they say different things. The
hash arm answers all 300 queries with 33 distinct top-1 documents at a mean pairwise cosine of
0.9654, a collapsed vector space rather than a weak ranking, consistent with its documented
non-semantic role. Each shuffled control carries its own arm's real document vectors permuted by a
cyclic shift of n/2 + 1 so that no document can keep its own vector; all three keep healthy
diagnostics and all three still score zero. That is what establishes the qrels-to-corpus mapping is
real rather than the pipeline handing each query its own answer back, and it is why a headline
number from any arm is reportable at all. Had a control not collapsed, the arm above it would be
void and this page would say so instead of quoting it.

The two providers sit at very different mean pairwise cosines, 0.6282 and 0.0992. That is a
property of how each model spreads its space, not a ranking-quality signal, and neither number
should be read as one.

One deliberate divergence in our harness, disclosed because it should be visible now rather than
found later. ruvector-sota-bench's ndcg_at_k returns 1.0 when a query has no relevant
documents. Ours excludes those queries and prints the excluded count, on the reasoning that a
query with nothing to find scoring perfectly is a value emitted by an instrument that resolved
nothing. On SciFact's test split the count is zero, so the two agree exactly here.

Is it the implementation, or just the model?

Worth separating, because a provider can wire a good model and still degrade it in the forward
pass. lattice-embed's implementation of bge-small-en-v1.5 is asserted against HuggingFace
reference outputs in its own test suite, tolerances cosine ≥ 0.9990 and max-abs-diff ≤ 5e-3:

[bge-small] 'The quick brown fox jumps over the lazy '  cosine=0.999868  max_diff=2.85e-3
[bge-small] 'Pure Rust transformer inference engine.'   cosine=0.999935  max_diff=1.63e-3
[bge-small] 'Café résumé naïve façade, Unicode test.'   cosine=0.999907  max_diff=1.81e-3
[bge-small] '   leading whitespace and    multiple   '  cosine=0.999872  max_diff=2.38e-3
[bge-small] '短い日本語のテストです。'                      cosine=0.999903  max_diff=2.22e-3
aggregate: min_cosine=0.999868  max_abs_diff=2.85e-3  failures=0/5

So the score is the model's retrieval quality arriving intact, not a reimplementation that happens
to rank well.

What this provider is actually for

The module docs already recommend OnnxEmbedding with all-MiniLM-L6-v2 for production, and that
is a local path, so locality is not what lattice-embed adds. The difference is the dependency
story, and building both arms for the benchmark above put numbers on it.

The ONNX arm resolved ort and linked a prebuilt libonnxruntime.a of 70.3 MiB, plus the
model artifact at 86.2 MiB, roughly 156 MiB fetched before it can link and run on a cold
machine. The one component built from non-Rust source was onig, the C regex library the
tokenizers feature pulls in. None of that is a problem on a developer laptop with a warm cache,
where the whole tree compiled in about a minute.

lattice-embeddings pulls lattice-embed and tokio, and is Rust to the bottom: no prebuilt
native archive to locate for the host triple and no C toolchain in the build. We built and ran on
macOS only, so treat cross-target behaviour as unmeasured rather than asserted. What is measured is
the dependency edge itself, which carries no ONNX Runtime archive and no C regex library. Its cost
is the model download: 128.2 MiB for bge-small-en-v1.5 across four files, of which
model.safetensors is 133,466,304 bytes, measured on the cache this benchmark ran against.

For a project that ships a single static binary, builds in restricted CI, or cross-compiles to
targets where a prebuilt ONNX Runtime is not sitting ready, the dependency shape is the reason to
look at all, subject to the caveat above that we have not built for those targets ourselves. For a
project already comfortable with ONNX Runtime it may be worth nothing. That judgment is yours and we
are not going to make it from outside.

The Rust-version cost, stated plainly

This is the real cost of the feature, and it is ours rather than yours. Your manifest already
records it:

NOTE: lattice-embed 0.6 requires Rust >= 1.93 (edition 2024). Cargo cannot express a per-feature rust-version, so enabling the lattice-embeddings feature raises the effective MSRV above this crate's workspace-inherited 1.77 for anyone who turns it on. The default build is unaffected.

We measured what the crate actually needs rather than what it declares. lattice-embed checks
clean on Rust 1.90.0, so the declared 1.93 is a workspace-wide floor across all of lattice's
crates and is conservative for this one by at least three minor versions. Below 1.90 the binding
constraint stops being our source: at 1.85 the failures are all in the image crate, a hard
dependency, using library features unstable at that version. Edition 2024 puts a hard floor at
1.85 regardless.

To be exact about what that does and does not establish: 1.90.0 is a version we checked and it
passed, not a minimum we located. We did not test 1.86 through 1.89, and we stopped there
deliberately rather than for lack of time, because the lowest number reachable inside edition 2024
is 1.85 and that is still eight minors above your 1.77, so a tighter answer would not change any
decision you have to make.

So the honest bound is that no floor we can reach comes near 1.77. We are not proposing anything
that changes your default feature set, and we are not asking you to move your workspace
rust-version. If a lower number than 1.90 would change your answer, say which number and we will
go measure whether we can get there.

What we are offering

One offer, and one question we would rather ask than answer for you. Both keep
lattice-embeddings opt-in and leave the existing default feature set unchanged.

1. Documentation. The provider list already names LatticeEmbedding. If it is useful, a short
   subsection in the same style as the ONNX one, showing the working call and stating the Rust
   version and the model download next to it, so the tradeoff is visible at the point of the
   decision. No code, no new dependency.

2. A convenience constructor, which we are deliberately not offering, and why. The obvious
   shape would be AgenticDB::with_lattice_embeddings(options, model_id), folding
   Arc::new(LatticeEmbedding::from_pretrained(model_id)?) and the options.dimensions agreement
   into one call. We sketched it, then went and read what it would actually do, and it is not worth
   offering on its own.

   AgenticDB::generate_text_embedding calls EmbeddingProvider::embed, and the search entry
   points call it the same way the indexing ones do. embed is the passage side. embed_query is
   an inherent method on LatticeEmbedding, not part of the trait, so a boxed provider cannot
   expose it. A convenience constructor would install the provider correctly and still embed every
   query without the retrieval instruction, which is exactly the failure the deployment note below
   warns about. Your module documentation already tells a reader to call embed_query directly for
   the query side, so the working path today is direct provider use, and that is the path the
   documentation offer above would show.

   Whether AgenticDB should grow a role-aware query seam is an API design question that is yours
   rather than ours, and it is not specific to this provider: it applies to any asymmetric
   retriever, which your own docs identify as BGE, E5 and Qwen3-Embedding. If you want one scoped,
   we are glad to help. If you would rather keep the trait role-neutral and let callers reach for
   the inherent method, that is a coherent position and the documentation item stands on its own
   either way.

Deployment notes a reader will want before selecting it

• First run downloads the model. bge-small-en-v1.5 is a 384-dimension model fetched from
  HuggingFace on first use, 128.2 MiB on disk as cached here. There is no bundled-weights option, so
  a fully offline first run needs the cache pre-populated.
• Query and passage embeddings are not symmetric for this model. BGE prepends a retrieval
  instruction to queries and nothing to passages, which is why the benchmark above uses
  embed_query for queries and embed_passage for documents. The EmbeddingProvider trait's
  single embed is the passage-side call. Anything routing both sides through embed loses the
  instruction and measurably loses retrieval quality, so this is the one integration detail worth
  documenting wherever the provider is advertised.
• Platform reach. SIMD on x86_64 and aarch64, no GPU requirement, no macOS gate. lattice-embed
  does also build for wasm32, but that is a different API from the provider described here: it takes
  model bytes in memory and has no download or filesystem cache path, so it is not the shape
  LatticeEmbedding uses and should not be read as wasm support for this integration.
• bge-small-en-v1.5 is English. Default model choice, and its size against first-run download
  weight, is a design question rather than a technical constraint, and it is yours.

What this does not establish

• One dataset. SciFact is a small scientific-claim corpus. Workloads of code, chat logs or
  product text should expect different absolute numbers.
• One direction of the window comparison. The matched row cuts our arm down to 128 tokens. The
  converse, giving the ONNX arm a 512-token window, was not run, so how each model behaves with more
  context per document is open in both directions.
• No throughput or latency measurement. The benchmark was not run under conditions that make
  timing meaningful, so no performance claim is made here in either direction.
• No comparison against hosted API providers. Re-deriving that would have meant sending the
  corpus to an external service, and the published MTEB numbers for these models are available to
  anyone who wants that comparison.

Happy to open the documentation item as a PR if it is useful, or to leave all of this here as a data
point. Either is a fine outcome from our side.

[ohdearquant]

One correction to the comment above, which I have edited in place rather than leave standing.

I wrote that the issue's "Rust >= 1.93" figure was too high. That is wrong in the direction that
flatters us, so it is worth saying plainly. Published lattice-embed 0.6.1 declares
rust-version = 1.93, and cargo enforces that before compiling anything, so 1.93 is the number
you would actually hit. Our 1.90 result came from --ignore-rust-version and describes the source
rather than the manifest. The Cargo.toml comment in this repo that states 1.93 is correct and I
am not proposing any change to it. The recommendation is unaffected: still B, opt-in.