🚀 Foundational Models Guide Best 🚀

Source-Free Domain Adaptive Breast Cancer Detection from Mammograms

This was part of my thesis titled "Domain Adaptation in Breast Cancer Detection from Mammograms" at IIT DELHI. 🌟 Give it a ⭐ if you find it helpful!

---

🌟 Abstract

We dive deep into source-free domain adaptive object detection (SFDAOD), where the source data decides to take a vacation during the adaptation phase! The goal? Adapt to an unlabeled target domain without any direct access to the source.

Here's the twist: Standard Student-Teacher (ST) frameworks often lead to catastrophic teacher collapses due to source bias and domain shifts. But worry not! Enter vision foundation models like CLIP, DINOv2, and SAM—our saviors with exceptional zero-shot skills.

We propose Fuse, Learn, and Distil (AFaL):

• 📚 Align student features with experts using contrastive knowledge distillation.
• 🧠 Fuse multiple expert models via the novel Expert Domain Fusion (EDF) module.
• 🚀 Avoid mode collapse, handle biases, and conquer challenging domains!

🛠️ Experiments across four natural imaging datasets and two medical datasets prove that AFaL slays the competition with superior performance. 🔥

---

🏗️ Model Zoo

Access pretrained weights, configs, and everything cool here. 🐾

---

📁 Directory Structure

foundation_weights/
|
-- vitb16_clip/
|    |-- source_dataset_1/
|           |-- model_best.pt
|           |-- target_dataset_1/
|               |-- embeddings_save.npy
|           |-- target_dataset_2/
|               |-- embeddings_save.npy
-- vitb16_imgnet/
|    |-- source_dataset_1/
|           |-- model_best.pt
|           |-- target_dataset_1/
|               |-- embeddings_save.npy
|           |-- target_dataset_2/
|               |-- embeddings_save.npy
-- vitdino/
    |-- source_dataset_1/
           |-- model_best.pt
           |-- target_dataset_1/
                |-- embeddings_save.npy
           |-- target_dataset_2/
                |-- embeddings_save.npy

The files model_best.pt and embeddings_save.npy are source trained foundation model and target set embeddings of the foundation model respectively. For source training and gettings embeddings of foundation models, refer foundation_code/README.md

But before doing anything, setup your conda environment following the instructions below:

INSTALLATION OF ENVIRONMENTS:

In foundation_code/ and FUSED/ lies the .yml config files to create conda environment for running the foundation and adaptation code respectively.

1) Install environment for Foundation models

cd ./foundation_code
conda env create -f found.yml

2) Install environment for FocalNet Object detector

Compiling the Deformable Attention Code in FUSED/:

cd ./FUSED
conda env create -f sfda-neurips.yml

cd ./models/dino/ops
python setup.py build install

# unit test (should see all checking is True)
python test.py

After these Installations you are good to go...

Overall Flow of the Code:

This is the only correct order to run this code:

Source Pretraining:

1) FND Source pretraining: Follow through ./FUSED/README.md for FND source training.
2) Foundation Source Pretraining: Follow through ./foundation_code/README.md for Source training all the foundation models necessary. These weights (model_best.pt) will be stored in ./foundation_weights.

Adaptation:

1) Get embeddings from foundation models: Follow through ./foundation_code/README.md for getting the expert/foundation embeddings. These embeddings (embeddings_save.npy) should be stored in ./foundation_weights.
After this and source pretraining, the structure of ./foundation_weights should look like as described above.
2) FUSED Adaptation: Now, the expert embeddings are used in the adaptation process. Follow through ./FUSED/README.md for adaptation of the detector.

If your source dataset is the same as one of our experiments, you may refer to this link to download the concerned source trained foundation models' weights. (model_best.pt)

After source training, get the embeddings (by following instructions in ./foundation_code/README.md) and use these embeddings in adaptation.