A multimodal retrieval system built for a final-year project. It indexes a text knowledge base and a folder of images, then lets you:
- Ask questions and get answers grounded in retrieved document chunks (RAG)
- Search images by typing a description (text → image, via CLIP)
- Upload an image to find related text chunks (image → text, via CLIP)
| Q&A tab (RAG) | Image Search tab |
|---|---|
 |
 |
CLIP (ViT-B/32) maps both images and text into the same 512-dimensional space, which means one FAISS index handles all three retrieval modes above. Text chunks are also indexed separately in Elasticsearch using bge-small-en-v1.5 (a retrieval-tuned text embedder) to support BM25 and dense kNN hybrid search for the Q&A tab.
PDF pages are rendered to images via PyMuPDF and indexed alongside standalone images — so a PDF is simultaneously searchable as text and as a visual.
Streamlit UI
├── Q&A tab ──────────────────────────────────────────────────────┐
│ text query → FAISS (CLIP) or Elasticsearch (BM25 / bge kNN) │
│ retrieved chunks → LLM → grounded answer + source citations │
└── Image Search tab ───────────────────────────────────────────┐ │
text description → CLIP → FAISS → top-k images │ │
uploaded image → CLIP → FAISS → top-k text/image chunks │ │
pip install -r requirements.txt
cp .env.example .env # add your OpenAI key
python -m scripts.build_index # build FAISS index
streamlit run app/main.pyOpen http://localhost:8501.
docker compose up -d # Elasticsearch 8.14
python -m scripts.build_es_index # BM25 + bge-small kNN index
python -m eval.run_eval # benchmark all four modesPick the backend in the sidebar: hybrid / bm25 / dense / faiss.
27 gold-labeled queries (14 exact-terminology, 13 paraphrased/conceptual) over a 201-chunk corpus of numerical-methods lecture PDFs and AI/ML notes.
| Mode | Retriever | Recall@5 | MRR | Avg latency |
|---|---|---|---|---|
faiss |
CLIP text encoder + keyword boost | 59.3% | 0.517 | 1346 ms |
bm25 |
Elasticsearch BM25 | 92.6% | 0.720 | 290 ms |
dense |
ES kNN over bge-small-en-v1.5 |
92.6% | 0.787 | 79 ms |
hybrid |
BM25 + dense kNN fused with RRF | 92.6% | 0.751 | 136 ms |
See eval/results.md for the per-query breakdown.
Why RRF instead of weighted score fusion?
BM25 scores and cosine similarities are on completely different scales — you can't add them directly without normalization that needs per-domain tuning. RRF fuses on rank instead (score = Σ 1/(60 + rank)), so it doesn't care about the scale of either retriever's scores and doesn't need retuning when the corpus changes. The tradeoff is that a perfectly calibrated weighted fusion can beat RRF on a specific domain, but it breaks as soon as the data distribution shifts.
Why bge-small-en-v1.5 for the text path instead of CLIP's text encoder?
CLIP's text encoder is trained to align with images, not with other text. It doesn't do well at text-to-text semantic similarity. bge-small-en-v1.5 is a retrieval-specialized embedder trained on MS-MARCO and similar datasets — it clearly wins on paraphrase retrieval (see MRR difference in the table above). Images stay on the CLIP path because you need the shared embedding space for cross-modal search.
Why IndexFlatIP for FAISS?
With L2-normalized embeddings, inner product equals cosine similarity, so IndexFlatIP gives exact cosine search. Exact search is fine at this scale (~300 chunks, ~100 images). The index type is configured in one place (config.py) and swaps to IVF or HNSW without changing any other code.
Why render PDF pages as images in addition to extracting text? PyMuPDF lets you do both in one pass. A scanned or diagram-heavy PDF will have weak text extraction but good visual content — rendering the page image means CLIP can still surface it through the Image Search tab.
BLIP caption caching
BLIP captions are keyed by SHA-256 of the image file and persisted in data/caption_cache.json. Repeated rebuilds skip BLIP for unchanged images, so rebuilding after adding a few new images only pays the inference cost for those new images, not the whole folder.
| Concept | Location |
|---|---|
| Contrastive vision-language pretraining (CLIP) | app/retrieval/embeddings.py |
| FAISS vector index (cosine similarity via IndexFlatIP) | app/retrieval/vector_store.py |
| Text chunking with overlap | app/ingestion/document_loader.py |
| PDF ingestion + page image rendering | app/ingestion/document_loader.py |
| BLIP image captioning with hash-keyed cache | app/ingestion/image_processor.py |
| Retrieval-Augmented Generation | app/generation/llm.py |
| Cross-modal retrieval (text↔image) | app/main.py Image Search tab |
| BM25 + dense kNN hybrid search | app/retrieval/es_store.py |
| Reciprocal Rank Fusion (manual, no ES license needed) | rrf_fuse() in app/retrieval/es_store.py |
| Retrieval evaluation (Recall@k, MRR) | eval/run_eval.py |
vision-rag/
├── app/
│ ├── main.py # Streamlit app (Q&A + Image Search tabs)
│ ├── config.py # all paths, model names, and constants
│ ├── ingestion/
│ │ ├── document_loader.py # load + chunk .txt/.md/.pdf/.docx
│ │ └── image_processor.py # BLIP captioning with caption cache
│ ├── retrieval/
│ │ ├── embeddings.py # CLIP encoder (shared image/text space)
│ │ ├── text_embeddings.py # bge-small embedder (ES dense side)
│ │ ├── vector_store.py # FAISS: build / save / load / search
│ │ ├── es_store.py # Elasticsearch: BM25 + kNN + RRF
│ │ └── retriever.py # unified retrieve(query, mode=...)
│ └── generation/
│ └── llm.py # RAG: prompt + LLM call + citations
├── data/
│ ├── knowledge_base/ # .txt/.md/.pdf/.docx files to index
│ ├── images/ # images to index
│ └── caption_cache.json # BLIP caption cache (auto-generated)
├── eval/
│ ├── queries.json # gold-labeled benchmark queries
│ ├── run_eval.py # Recall@5 + MRR across all 4 modes
│ └── results.md # benchmark results
├── scripts/
│ ├── build_index.py # CLI: build FAISS index
│ └── build_es_index.py # CLI: build Elasticsearch index
├── docker-compose.yml # Elasticsearch 8.14
├── tests/test_pipeline.py
├── requirements.txt
└── .env.example
MIT