Skip to content

Embedding models

Jump to bottom
Administrator edited this page Aug 12, 2026 · 2 revisions

Embedding models

Hybrid recall pairs keyword search with dense vectors from a small embedding model. Prebuilt release binaries embed with candle and default to BAAI/bge-small-en-v1.5 (about 130 MB on disk), downloaded from HuggingFace once on first use and cached.

dmem doctor          # active embedder, model, cache dir, whether it is already cached, CPU features
dmem doctor --json   # the same, machine-parseable

Footprint and requirements

Process RSS is dominated by the model the server keeps resident, not by the SQLite store (live data is usually tens of megabytes). Figures below are order-of-magnitude, once warm, on a small Linux host. Treat them as planning guides; measure with dmem doctor and your process monitor.

Setup Typical process RSS Languages / notes
Prebuilt default: candle + BAAI/bge-small-en-v1.5 ~0.8-1.2 GiB English-first. What the release ships.
candle + intfloat/multilingual-e5-small (env override) ~1.2-1.5 GiB Strong pick for English + Malay (BM/MS) and other languages. Needs mean pooling + e5 prefixes (below).
candle + other 384-d Bert (env override) similar to model size Any HF Bert checkpoint with tokenizer.json and 384-d hidden size. Match pooling/prefixes to the model card.
Build with model2vec (e.g. potion-base-8M) a few hundred MB Rebuild required. Much smaller; weaker semantic recall.
No real embedder (hash placeholder) tens of MB Keyword/FTS only. Not for production semantic recall.
Client-only against a remote dmem serve negligible local model RAM Embeddings run on the server.

Disk: model cache is separate from the store. Multilingual checkpoints are often larger on disk than bge-small.

DM_CANDLE_F16=1 loads weights as f16 and can cut model RAM roughly in half. The fastembed Cargo feature is an alternative ONNX path; release binaries ship candle, not ONNX.

Disclaimer: usage scales with embedder choice. If the prebuilt default is heavier than you want, rebuild with model2vec or run without a real embedder, or keep a thin client pointed at a remote server that holds the model.

Offline / air-gapped

Pre-populate the model cache once on a connected machine, then carry it over (or point at a shared path):

# Pre-warm the cache (run once with network), then start dmem offline:
HF_HOME=/srv/hf-cache python -c \
  "from huggingface_hub import snapshot_download; snapshot_download('BAAI/bge-small-en-v1.5')"
HF_HOME=/srv/hf-cache dmem serve --addr 127.0.0.1:8088

dmem honours HF_HOME and HUGGINGFACE_HUB_CACHE (the standard HuggingFace cache), and dmem serve logs the cache dir and model on startup. dmem doctor prints the exact directory it expects and whether the model is present, so you know up front if a first run needs network.

Multilingual and custom models

The default model is English. If your memory is bilingual (for example English + Malay: queries in one language should recall records written in another), swap in a multilingual checkpoint by env, no recompile. Any 384-d BertModel-architecture checkpoint on HuggingFace works. Needs a build with the candle multilingual knobs (release 0.2+ / candle pooling support).

Recommended starting points for EN and MY users:

  1. BAAI/bge-small-en-v1.5 (prebuilt default) - English-first, best zero-config path.
  2. intfloat/multilingual-e5-small - EN + Malay and many other languages; higher RAM than bge-small.
  3. Another 384-d Bert on HF - if you already trust a checkpoint; set pooling/prefixes to match its card.
  4. model2vec build - when RAM is tight and you accept weaker vectors.
  5. Hash placeholder - smoke tests and keyword-only installs.

Example: multilingual-e5-small

export DM_CANDLE_MODEL="intfloat/multilingual-e5-small"
export DM_CANDLE_POOLING=mean            # e5 and paraphrase-multilingual are mean-pooled (default: CLS, the bge way)
export DM_CANDLE_PREFIX_QUERY="query: "  # e5-style role prefixes
export DM_CANDLE_PREFIX_DOC="passage: "

Set the pooling and prefixes to whatever your model's card documents; the defaults (CLS pooling, no prefixes) match the bge family exactly. The doc prefix is injected into every chunk window of a long record, not just the first, so late windows keep their role marker. With a model override the embedder reports itself as candle-custom in dmem doctor and the logs, since similarity scores are model-relative and should not be read against bge-calibrated expectations.

These variables take effect wherever the embedding happens: the server process in client/server mode, or your local process in embedded mode. Set them for the service (systemd/launchd unit) if you run one, not just your shell.

After switching models, re-embed the existing records so old and new vectors live in one space:

dmem reindex-embeddings

Build-time embedder choices

Cargo features (see Cargo.toml):

  • dist (what releases ship): candle + server + client + wizard + ui, default model bge-small-en-v1.5
  • candle: pure-Rust transformer path (production default)
  • fastembed: ONNX bge-small (heavier runtime)
  • model2vec: static embeddings, small RSS
  • none of the above: HashEmbedder placeholder

Changing feature set requires a rebuild. Changing the candle HF model id does not.

Clone this wiki locally