research: small CPU-only code embeddings + adoptable optimizations from peer tools (embed tier roadmap)

[ajianaz]

Goal

Pick the next embedding tier for Brain Mode (between today's hashing 256d ≈15-20 CoIR and the planned 154 MB ONNX jina-v2-base-code ≈57 in #500), and harvest optimization ideas from peer tools. CPU-only is a hard requirement (cora is BYOK/local-first).

Candidate: EmbeddingGemma-300m (Q4, ONNX)

• ~300M params, runs in-browser/on-device, QAT-quantized for edge
• Code retrieval strength: arXiv 2509.20354 reports large gains on CosQA and AppsRetrieval vs same-size models
• License: Gemma terms (usable, not Apache) — flag vs our Apache-2.0 stance
• Caveats: HF thread reports weak out-of-box results without proper prompt/task prefixes

Alternatives

• jina-embeddings-v2-base-code (154 MB q8, Apache-2.0, ~57 CoIR est) — the feat(brain): ONNX runtime integration with Jina v2 base code embeddings #500 plan, proven license
• nomic-embed-text v1.5 (137M, Apache-2.0) — general text, weaker on code
• codet5p-110m (BSD-3) — code-specific, seq 512, no ONNX
• Vendor upgrade path: nomic-embed-code 7B / jina-code-0.5b — too big or NC-licensed

What peers actually use

• codebase-memory-mcp (40.9k★): bundles nomic-embed-code 768d int8 into the binary; local CPU inference; hybrid FTS5-BM25 + vectors + graph + 11-signal ranking; MinHash/LSH clone detection; Louvain communities; team-shared zstd graph artifact; Linux kernel 28M LOC in ~3 min
• graphify: knowledge-graph MCP; embeddings roadmap = OpenAI API or nomic-embed-text via Ollama (local)
• Tabby: default embedding = Nomic-Embed-Text (local)
• Continue.dev: recommends nomic-embed-text via Ollama; voyage-code-3 if cloud OK
• aider / code-review-graph: NO embeddings — pure tree-sitter repo/structural maps

⇒ Industry local default is Nomic-family on CPU; nobody ships a small code-specific transformer locally — a gap EmbeddingGemma-300m or jina-v2-code would fill for cora.

Adoptable optimizations for cora

1. int8/f16 storage of embeddings (pairs with vecq backend, feat(brain): vecq as opt-in vector backend for v0.15.0 (replace usearch long-term) #542)
2. Fused multi-signal ranking notes (we already do RRF; compare against their 11-signal scorer)
3. MinHash+LSH near-clone detection as a cheap cora extra
4. Team-shared compressed index artifact (graph.db.zst pattern) for onboarding
5. Louvain community detection on our call graph for cora arch
6. Background git-watcher incremental re-index (we have cora index --watch; consider auto-trigger)

Recommendation

1. Short term: keep hashing (tier 0) + nomic static (tier 1)
2. v0.15/0.16: ship jina-v2-base-code ONNX q8 (tier 2, feat(brain): ONNX runtime integration with Jina v2 base code embeddings #500) — Apache-clean, known quality
3. Evaluate EmbeddingGemma-300m QAT Q4 as tier 2 alternative: smaller than jina-q8? (~170 MB q4) — license + prompt-prefix caveats
4. vecq store (feat(brain): vecq as opt-in vector backend for v0.15.0 (replace usearch long-term) #542) composes with any of these

Sources: arXiv 2509.20354 · codebase-memory-mcp README · graphify#1 · tabby docs · ollama nomic-embed-text

[ajianaz]

Research input from the vecq width study (2026-08-31)

Measured on cora's own pipeline (not literature): real Rust symbol name + signature texts through embed_code_dispatch (hashing 256d, the default provider), recall@10 vs exact cosine ground truth across 1k/5k/13k-symbol scales, all vecq 0.3.0 quantization modes:

mode	recall@10 @1k	@5k	@13k	query @13k
plain 4-bit	0.780	0.773	0.710	0.6 ms
plain 5-bit	0.880	0.850	0.840	2.2 ms
plain 6-bit	0.917	0.907	0.870	2.3 ms
4-bit + residual	0.930	0.917	0.863	1.2 ms

Finding: quantization is not the bottleneck — the hashing embedding is

Even the best quantization mode tops out at ~0.86–0.93, and the neighbor structure these numbers measure is the same one usearch-f32 serves losslessly. The ceiling is set by the embedding, not the store. Corollary for this issue: the hashing→real-model gap (CoIR ~15-20 → ~57) is worth more recall than any store/quantization decision downstream. This strengthens the case for a new embedding tier and de-risks the storage side (vecq residual keeps a 768d index at ~5x compression with measured recall cost).

Candidate assessment vs cora constraints (Apache-2.0, CPU-only, bundle size)

• jina-embeddings-v2-base-code q8 (154 MB, Apache-2.0, ~57 CoIR) — stays the safe default track (feat(brain): ONNX runtime integration with Jina v2 base code embeddings #500): proven license, proven code retrieval.
• EmbeddingGemma-300m Q4 — attractive size/quality, but (a) Gemma license conflicts with our Apache-2.0 stance for a bundled asset, (b) the reported code-retrieval gains depend on task-prompt prefixes (per the HF thread) — evaluate with prefixes or not at all.
• nomic-embed-text v1.5 / peer defaults — the industry local default, but general-text (weaker on code) and redundant with the nomic-embed-code distilled path we already ship behind pretrained-embed.
• Zero-MB middle tier worth one experiment first: upgrade the hashing provider (token-pair/bigram features, identifier-subword weighting) before bundling 100 MB+. The recall harness below measures it in an afternoon; if it closes even a third of the gap, it's a free tier for users who never opt into a model download.

Evaluation protocol (so candidates stay comparable)

Dump real symbol texts from an actual cora index run ({name} {signature} format), embed with the candidate model, measure recall@10 vs exact cosine at 1k/5k/13k scales plus CPU build/query latency. The offline harness used for the width study does exactly this against a dumped embeddings file.

Recommendation: keep #500 (jina) as the tracked path; run the bigram-hashing experiment first (cheap, zero size cost); revisit EmbeddingGemma only if the license stance changes and a prompt-prefixed eval clears the bar.