🩺 Medical Case Retrieval with DINOv3 & RAG

🚀 This project demonstrates how to use DINOv3 to extract features from medical images and leverage them to build a RAG (Retrieval-Augmented Generation) system to help doctors and trainees quickly find the most similar cases.

---

📂 Project Overview

• 📦 Data:
  • A subset of the CXR8 dataset from Hclinical center.
  • The dataset contains 5000 images (~4 GB).
• 🧠 Model:
  • facebook/dinov3-vit7b16-pretrain-lvd1689m for extracting embeddings.
  • Each image is represented by a 4096-D feature vector.
• 💾 Storage:
  • Features: ~78 MB (.npz)
  • Metadata: ~340 KB (information about disease and patient).

---

🎯 Project Goal

The idea is to build a medical decision support system by:

1. Receiving a new patient image.
2. Extracting a feature vector using DINOv3.
3. Searching for Top-K closest cases in the archive.
4. Displaying similar results (images + metadata) to doctors:
   • 🧑‍⚕️ Training doctors and trainees.
   • 🆘 Supporting decision-making when a specialist is not available at patient arrival.
   • ⏱️ Speeding up diagnosis by comparing similar cases.

---

🔄 Workflow

flowchart LR
  A["🖼️ Patient Image"] --> B["⚙️ DINOv3 Feature Extractor"]
  B --> C["🔢 4096-D Vector"]
  C --> D["📚 Feature Index (FAISS / HNSW)"]
  D --> E["🔍 Top-K Retrieval"]
  E --> F["📊 Results + Metadata"]
  F --> G["🧑‍⚕️ Doctor Review"]

Loading

📊 Key Numbers

Stage	Size/Count
First batch images	~4 GB
Number of images	5000 images
Features size	78 MB
Metadata size	340 KB
Feature dimension	4096

🛠️ Implementation Steps

1. Feature Extraction

• Use DINOv3 to convert images into embeddings.

2. Building the Index

• Create a FAISS/HNSW index for fast retrieval.

3. RAG-style Retrieval

• Search for the closest cases (Top-K).
• Integrate metadata (diagnosis, age, gender, modality).

4. Evaluation

• Measure Recall@K and NDCG@K.
• Clinical review (Human-in-the-loop).

⚡ Future Applications

• 📚 Train doctors on diverse cases.
• 🆘 Support clinical decisions when a specialist is absent.
• 🧬 Quickly search through a large medical data archive.
• 📈 Improve diagnosis quality and reduce response time.

⚠️ Important Note

• This project is intended as a support tool for doctors and should not be used as a replacement for medical diagnosis.
• All data used is real, but the purpose is research and training only.

💡 Future Idea

• Image-based retrieval can be combined with textual queries (report/case description) to build a more accurate Hybrid Retrieval system.