Multi-view Test-time Adaptation for Semantic Segmentation in Clinical Cataract Surgery

The repository is the official pytorch implementation of the paper Multi-view Test-time Adaptation for Semantic Segmentation in Clinical Cataract Surgery.

@ARTICLE{10843248,
  author={Li, Heng and Ou, Mingyang and Li, Haojin and Qiu, Zhongxi and Niu, Ke and Fu, Huazhu and Liu, Jiang},
  journal={IEEE Transactions on Medical Imaging}, 
  title={Multi-view Test-time Adaptation for Semantic Segmentation in Clinical Cataract Surgery}, 
  year={2025},
  volume={},
  number={},
  pages={1-1},
  keywords={Surgery;Cataracts;Adaptation models;Semantic segmentation;Data models;Biomedical imaging;Training;Data privacy;Lighting;Decoding;Cataract Surgery;semantic segmentation;test-time adaptation;multi-view learning},
  doi={10.1109/TMI.2025.3529875}}

If you find our work useful in your research please consider citing our paper.

Prerequisite

Dataset

The annotated CatInstSeg and CatS can be found at:

• CatInstSeg
• CatS

Dataset Preprocessing: Category Standardization

Note: A "mapping_merge_tools.csv" file is necessary for your dataset. For example, if your CaDIS dataset is located in "storage/datasets/cadis", then there should be a "mapping_merge_tools.csv" in the same directory. We provide an example file for label mapping in CaDIS dataset. Example

Package dependency

• easydict==5.1.1
• imageio==2.34.2
• matplotlib==3.8.4
• numpy==1.26.4
• pillow==10.4.0
• pytorch==2.3.1
• torchvision==0.18.1
• tqdm==4.66.4
• yaml==0.2.5

Source Pretraining

We provide the checkpoints of pretrain model for convenience:

Deeplabv3_ResNet50

For source pretraining phase of MUTA model, run the following command:

# source_muta
bash storage/scripts/muta_source_cadis.sh # training on dataset CaDIS

Updated: We now provide the pretrained weights of muta_source in storage folder

Alternatively, we provide a pretrained deeplabv3 storage folder:

# source_deeplabv3
bash storage/scripts/deeplabv3_cadis.sh # testing script of deeplabv3

Adaptation

As mentioned in paper, we provide solutions for both domain adaptation scenarios without source.

To conduct Test Time Adaptation(TTA), where each sample is adapted individually, run the following command:

# tta_muta
bash storage/scripts/muta_tta_cadis2catinstseg.sh # tta from CaDIS to CatInstSeg

For Source Free Domain Adaptation(SFDA), where samples from target domain are adapted all together, run the command:

# sfda_muta
bash storage/scripts/muta_sfda_cadis2catinstseg.sh # sfda from CaDIS to CatInstSeg

Similar to the source pretraining procudure, we provide the deeplabv3 versions for TTA and SFDA respectively:

# tta_deeplabv3
bash storage/scripts/deeplanv3_tta_cadis2catinstseg.sh # tta from CaDIS to CatInstSeg

# sfda_deeplabv3
bash storage/scripts/deeplanv3_sfda_cadis2catinstseg.sh # sfda from CaDIS to CatInstSeg

For experiments on CatS dataset, simply replace the 'data_root' in the script. Besides, be careful to check the correctness of the label mapping file.