Embeddings

Embeddings

An embedding turns a piece of text into a vector — a point in a few-hundred-dimensional space where meaning is position. Two texts about the same thing land close together even with no words in common. That is what lets you search for "insurance renewal letter" and find a PDF that never uses those words.

Everything searchable in SenseTree exists because something was embedded. This page covers who produces the vectors, what text actually gets embedded, and where the vectors live.

Two engines, one interface

	Local engine (default)	HTTP server
embedding.mode	"local"	"openai"
Runtime	fastembed + ONNX Runtime, in-process	Any OpenAI-compatible /v1/embeddings
Network	none	your server
Setup	none — model downloads itself	Ollama / LM Studio / vLLM / a hosted API
Hardware	CPU, or CUDA if use_gpu	whatever the server has

Both implement the same internal interface (embed_documents, embed_query, dimensions), so nothing downstream — chunking, storage, search, reranking — knows or cares which one is active.

The default is deliberately the local one: semantic search works out of the box, offline, with no server to install.

The local engine

Available models

Identifiers accepted in embedding.model when mode: "local". The dimensions field must match — it defines the vector table's schema.

Model	Dims	Multilingual
multilingual-e5-small (default)	384	✅
multilingual-e5-base	768	✅
multilingual-e5-large	1024	✅
bge-small-en-v1.5	384	❌ English
bge-base-en-v1.5	768	❌ English
bge-large-en-v1.5	1024	❌ English
gte-base-en-v1.5	768	❌ English
gte-large-en-v1.5	1024	❌ English
modernbert-embed-large	1024	❌ English
all-minilm	384	❌ English
nomic-embed-text	768	❌ English
mxbai-embed-large	1024	❌ English

The multilingual column is the one that matters and it is invisible in the names. Only the E5 family is multilingual here. The others are trained on English and collapse on a French corpus — picking one by its benchmark score alone leads to a mediocre index you can only fix by re-indexing everything. The in-app catalog shows this flag next to each model.

An unknown identifier falls back to multilingual-e5-small with a warning rather than failing.

nomic-embed-text and mxbai-embed-large also exist in the Ollama library, so you can run the same model locally on CPU or remotely on a GPU and compare.

E5 prefixes, handled for you

E5 models were trained with asymmetric prefixes and lose accuracy without them. SenseTree applies them automatically, and only for that family:

• indexing → passage: <text>
• searching → query: <text>

In openai mode this is not done — your text is sent as-is, and it is the server's job to add prefixes if its model needs them.

ONNX Runtime provisioning

fastembed is compiled with dynamic ORT loading, so the app ships one binary and fetches the right runtime at first use:

1. On first need, ONNX Runtime 1.20.0 for win-x64 is downloaded from the official Microsoft release — the CPU build, or the GPU build if use_gpu is on and an NVIDIA driver is detected (nvcuda.dll present in System32).
2. It is unpacked into the app data directory and ORT_DYLIB_PATH is set before any ORT initialisation.
3. With use_gpu, the CUDA execution provider is requested with a CPU provider behind it: no GPU, wrong driver, or CPU-only runtime simply falls back, no error.

Provisioning happens exactly once per session, guarded so that whichever comes first — health check, worker, reranker, CLIP — triggers it safely.

Model cache

Weights are downloaded from Hugging Face on first use and cached in:

%APPDATA%\com.virgi.sensetree\models\models--<org>--<repo>\

The catalog shows which models are already cached and can pre-download one (it loads then immediately frees it) so your first indexing run doesn't stall on a download.

Loading, batching, unloading

• The embedder is built lazily and cached under the key mode | model | base_url | use_gpu. Change any of those in Settings and the next call rebuilds it.
• fastembed is synchronous and CPU-bound, so every embed runs on a blocking thread pool — never on the async runtime.
• Batch size is indexing.batch_size (default 32), used identically by both engines.
• The model is unloaded when idle. ONNX Runtime keeps an intra-op thread pool that spins (and burns CPU) as long as a session exists. So SenseTree drops the embedder when indexing is paused, and after ~15 seconds with an empty queue. It reloads on demand — the next file, or the next search.

What actually gets embedded

This is where most of the retrieval quality is decided, and the answer is not "the chunk".

For every file, SenseTree keeps two parallel texts per chunk: the one it stores (used for snippets, BM25 keyword matching and reranking) and the one it embeds (used for the dense vector).

Chunk	Stored text	Embedded text
#0	[pdf] <full qualification> + blank line + chunk	identical to stored
#1..n	the raw chunk	<filename> · <qualification, 180 chars> + blank line + chunk

Two distinct mechanisms are at play.

The qualification rides on chunk #0's stored text. The LLM-written "sense" of the document (what it is, plus key facts) is prepended to the first chunk, which makes it findable by keywords too. A French ID card whose OCR never contains the words "carte d'identité" is still matched by a BM25 query for them, because the qualification says so.

Contextual retrieval enriches the embedded text of every other chunk. A chunk reading "the amount is €90" is meaningless alone; embedded as "EDF-invoice-2024.pdf · Invoice from EDF for the March 2024 electricity bill... / the amount is €90" it lands near "electricity bill" in vector space. The snippet you see stays clean, because the enrichment is only in what was vectorized.

Non-document paths embed one vector each:

• Images — qualification + vision caption + context descriptor (name, folder, type, neighbours).
• Media — the transcription and/or visual description, then chunked and treated exactly like a document.
• Context-only files — the LLM's guess plus the factual descriptor.
• Block folders — the one-sentence LLM description plus the folder's facts (item count, dominant extensions, sample names).

See Indexing Pipeline for how each of those is produced, and Retrieval & RAG for how they are searched.

Chunking

Chunks are cut on semantic boundaries, not every N characters (chunker.rs):

1. Split on paragraphs (\n\n).
2. A paragraph longer than chunk_size is split into sentences (after ., !, ? followed by whitespace, or on a newline).
3. A sentence still too long is hard-windowed at chunk_size.
4. Units are then greedily packed up to chunk_size, with overlap characters carried over from the previous chunk — restarted after a space, so the overlap never begins mid-word.

Defaults: chunk_size: 1000, overlap: 200. Larger chunks mean more context per vector but blurrier matching; smaller chunks sharpen matching but fragment meaning.

max_chunks_per_file (default 0 = unlimited) caps how many vectors one file may produce. It is off by default on purpose: size says nothing about semantic value — a thesis, a book, a large code corpus deserve full indexing, and truncating silently would lose content. When you do enable it, the truncation is written into the stored extract, so you can see that a document is only partially searchable.

Storage

Vectors live in LanceDB (embedded, serverless, file-based), in the app data directory. Two tables:

Table	Contents	Dimension
chunks	id, path, chunk_index, text, content_hash, mtime, vector	embedding.dimensions
images	path, vector (CLIP)	512, fixed

• Writes are delete-then-insert per path, so re-indexing a file never leaves stale chunks.
• Dense search is cosine; a scope restricts it with a path LIKE 'prefix%' filter (Windows separators and LIKE metacharacters escaped).
• A BM25 full-text index on the text column is built lazily and rebuilt whenever new chunks were written since the last search.
• Renames update the path column without re-embedding — moving a folder costs nothing.

Changing the embedding model

Vectors from two different models are not comparable — they are points in unrelated spaces. So when you change embedding.model or embedding.dimensions and save:

1. the vector table's dimension is updated,
2. the vector store is cleared,
3. the SQLite index state and folder profiles are reset,
4. a full re-scan of every root starts.

This is automatic and unavoidable. Choosing your embedding model early is worth a few minutes of thought — the catalog exists to make that comparison concrete, with live MTEB scores per language board.

Changing reasoning, vision, transcription or video settings never triggers a re-index.

Two other vector models

Both are local fastembed/ONNX models, downloaded on first use into the same cache, and unrelated to the embedding slot.

Reranker (cross-encoder). Scores a (query, passage) pair jointly instead of comparing two independent vectors — far more accurate, far more expensive, so it only ever sees the handful of candidates hybrid search already selected. Options: bge-reranker-v2-m3 (default, multilingual), bge-reranker-base, jina-reranker-v2-base-multilingual. Toggle: retrieval.rerank.

CLIP ViT-B/32. Encodes images and text into the same 512-dimensional visual space, which is what makes Image Search work: you type a description, it is embedded as text, and the nearest image vectors come back. Entirely separate from document indexing — it is built on demand from the Image Search panel.

Tuning notes

• Multilingual corpus → an E5 model. No exception. Benchmarks that look better are usually English-only boards.
• Bigger isn't automatically better. multilingual-e5-large is 1024-d and roughly 3× the cost of -small per chunk; on filename-and-summary-heavy corpora the gain is often modest. There is a built-in benchmark harness (cargo test --lib banc_embedding_local -- --ignored --nocapture, with EMBED_MODEL=...) that measures chunks/s and MB/s on your own machine.
• Remote embedding is worth it when you have a real GPU elsewhere — a home server or a second PC. Set mode: openai, point at it, and set dimensions to what Test connection reports.
• batch_size trades throughput against memory on both paths. Lower it if a remote server rejects large requests.