Generative Embedding Benchmark: How Much Information Survives in a Dense Embedding?

Overview

Most embedding benchmarks ask whether representations separate the right item from a candidate gallery. GEB asks a complementary question: how much answer-relevant information can a generator recover from a dense embedding?

GEB freezes an embedding model and trains a shared Qwen3-0.6B readout. The decoder receives the embedding and question text, but never the source image or intermediate visual features. The benchmark contains 1,800 development and 900 held-out test questions spanning natural images, scene text, and visual documents. This generative readout exposes information bottlenecks that ranking-based evaluation cannot directly measure.

Results

Across seven embedding models, visual-only readout is tightly clustered, while question-conditioned VL-joint embeddings reveal much larger differences. The strongest embedding model reaches 65.56% overall, compared with 33.21% when one image embedding must be reused across questions.

Qualitative examples show that question-conditioned embeddings can recover fine-grained text and numerical details that are often unavailable from a reused visual-only embedding.

Quick evaluation

This README covers the shortest path for evaluating the released checkpoints. For training data preparation, embedding precomputation, decoder training, and full reproduction details, see docs/reproduction.md.

Environment

Python 3.10 or 3.11 and a CUDA-capable machine are recommended.

git clone https://github.com/LimitedMouse/Generative-Embedding-Benchmark.git
cd Generative-Embedding-Benchmark
pip install -e '.[eval]'

Checkpoints and evaluation data

No manual download step is required. Passing auto downloads the matching Qwen3-VL-Embedding-2B/8B decoder from the checkpoint repository. The evaluation command also loads the eight original public datasets through lmms-eval and applies the fixed membership from the GEB dataset repository. GEB does not redistribute the source images, questions, or answers.

Run evaluation

Visual-only evaluation:

NPROC=8 BATCH_SIZE=16 bash scripts/evaluate_paper.sh \
  qwen3vl-2b auto test visual_only

Question-conditioned VL-joint evaluation:

NPROC=8 BATCH_SIZE=16 bash scripts/evaluate_paper.sh \
  qwen3vl-2b auto test vl_joint

Replace qwen3vl-2b with qwen3vl-8b for the 8B release. Then compute the paper's item-level score from the generated sample logs:

python scripts/summarize_geb_results.py \
  --samples_dir outputs_lmms/qwen3vl-2b-test-vl_joint/samples \
  --split test

Do not use the raw lmms-eval group aggregate: source-dataset sizes can survive membership filtering. The full reproduction guide lists all four expected scores and covers evaluation from a locally trained checkpoint.

The paper evaluates Qwen3-VL-Embedding-2B/8B, VLM2Vec-V2, UME-R1-2B, Embed-RL-2B, CLIP ViT-L/14, and SigLIP SO400M.

Citation

Citation information will be updated when the paper is available on arXiv.

License

GEB-authored code and membership metadata are released under Apache-2.0. Models and source datasets remain subject to their original licenses.