Embedding Migration

Can you migrate a vector database to a new embedding model without re-embedding the whole corpus?

We embedded ~1M MS MARCO passages with 6 embedding models across 3 dimensionalities (768, 1024, 2560), trained linear translators between every pair of embedding spaces, and measured how much retrieval quality is preserved. The core metric is recall@10 ratio -- the translated recall divided by the native ceiling.

Key Results

50 paired examples is enough

Linear translation achieves 95-100% of native retrieval quality with as few as 50 aligned document pairs. For compatible model pairs, Procrustes at n=50 matches or exceeds Procrustes at n=5000:

Prefix mismatch is the real obstacle, not dimension mismatch

E5->GTE (both 768-dim) scores 0.090 with Procrustes because E5 uses query:/passage: prefixes while GTE uses none. Meanwhile GTE->Qwen-0.6B (768->1024) scores 0.858. Ridge regression rescues prefix-mismatched pairs: E5->GTE jumps from 0.090 to 0.814.

Method comparison on MS MARCO (30 pairs each)

Method	Best use case	Top ratio	Prefix-mismatch ratio
Orthogonal Procrustes	Same-dim compatible pairs	0.974 (BGE->E5)	0.090 (E5->GTE)
Least-squares	Cross-dim compatible pairs	1.006 (Qwen-4B->0.6B)	--
Ridge regression	Prefix-mismatched pairs	0.961 (BGE->E5)	0.814 (E5->GTE)
MLP (InfoNCE)	None -- no improvement over linear	0.942 (Qwen-4B->E5)	0.824 (E5->GTE)
Relative repr (1000 anchors)	None -- training-free but low quality	0.588 (Qwen-4B->0.6B)	0.000 (E5->GTE)

Cross-domain generalization

Translators trained on MS MARCO and evaluated on BEIR datasets (SciFact, FiQA, NFCorpus):

• Same-family pairs generalize well: Qwen-4B->Qwen-0.6B scores 0.94-1.01 across all domains
• BGE->E5 holds at 0.87-0.97
• Prefix-mismatched pairs degrade 10-30%, worst on financial text (FiQA)

Models

Model	Dim	Prefix
intfloat/e5-base-v2	768	query/passage
BAAI/bge-base-en-v1.5	768	query only
Alibaba-NLP/gte-base-en-v1.5	768	none
nomic-ai/nomic-embed-text-v1.5	768	search_query/search_document
Qwen/Qwen3-Embedding-0.6B	1024	instruction format
Qwen/Qwen3-Embedding-4B	2560	instruction format

Evaluation

• Corpus: ~1M MS MARCO v1 passages (995,000 docs, 6,980 queries with qrels)
• Metric: recall@10 ratio = recall@10(translated queries, target docs) / recall@10(native target queries, target docs)
• Training: 5,000 paired document embeddings (source, target) + all query embeddings
• Cross-domain: BEIR SciFact (5,183 docs), FiQA (57,638 docs), NFCorpus (3,633 docs)

Repository Structure

embedding-migration/
  prepare_corpus.py      # Convert MS MARCO to parquet + manifest
  prepare_corpus_1m.py   # Scale corpus to 1M passages
  prepare_beir.py        # Convert BEIR datasets to parquet + manifest
  embed.py               # Embed corpus/queries with any HF model
  sanity_checks.py       # Pre-flight checks before embedding
  eval_retrieval.py      # Compute native recall@10 ceilings
  translate.py           # Train translators (Procrustes, ridge, MLP) and evaluate recall
  fewshot_curve.py       # Few-shot learning curve experiment
  relative_repr.py       # Relative representations (Moschella et al.) baseline
  make_figures.py        # Generate all figures from results
  FINDINGS.md            # Detailed findings and data inventory
  figures/               # Generated PNG figures

Usage

Setup

python -m venv venv && source venv/bin/activate
pip install -r requirements.txt

Prepare corpus

python prepare_corpus_1m.py          # MS MARCO 1M subset
python prepare_beir.py               # BEIR datasets

Embed

python embed.py --model intfloat/e5-base-v2 \
    --input data/corpus_1m/corpus.parquet \
    --output data/embeddings_1m/intfloat_e5-base-v2 \
    --prefix-type doc --batch-size 256

Translate and evaluate

# All 30 pairs with Procrustes/least-squares
python translate.py --method procrustes --results-dir results/msmarco_procrustes

# Ridge regression
python translate.py --method ridge --results-dir results/msmarco_ridge

# Cross-domain evaluation
python translate.py --method ridge --eval-dataset scifact --results-dir results/beir_scifact

# Few-shot learning curve
python fewshot_curve.py --train-sizes 50 100 250 500 1000 2500 5000

# Relative representations (training-free baseline)
python relative_repr.py --anchor-sizes 50 100 250 500 1000

Critical Invariant

Row N = document N. The .npy embedding files have no ID column -- alignment depends on the parquet being in the same order it was during embedding. prepare_corpus.py refuses to run if a manifest already exists, and verify_corpus.py checks the hash. Never regenerate the corpus after embeddings exist.

Requirements

• Python 3.10+
• PyTorch 2.0+ with CUDA
• ~50 GB disk for 1M embeddings across 6 models
• GPU with 16+ GB VRAM for embedding (24 GB recommended for Qwen-4B)