TASFormer: Task-aware Image Segmentation Transformer

[Springer] [pdf] [BibTeX]

This repo contains the code for our paper TASFormer: Task-aware Image Segmentation Transformer.

Contents

2. Notes
3. Installation Instructions
4. Dataset Preparation
5. Execution Instructions
   • Training
   • Evaluation
6. Results

Notes

ADE20K_multitask_segmentation/ contains codes and configs for ADE20K dataset.

multitask_segmentation/ contains additional codes and configs for GDD, Sun, SceneNet and Trans10k datasets.

transformers_update_for_adapters/ contains files required for TASFormer (HF adapter) and TASFormer (HF adapter++).

You can find $bIoU$ metric implementation inside validation_step() in pl_module.py. Keep in mind, our TASFormer model works with binary masks, and averaging is performed over all masks, regardless of their class. More details about $bIoU$ metric can be found in our Paper.

Installation Instructions

• We use Python 3.8, PyTorch 1.8.1 (CUDA 10.1 build).
• We use PyTorch Lightning 1.5.0.
• For complete installation instructions, please see Installation.

Dataset Preparation

• We experiment on ADE20K benchmark and other datasets.
• Please see Preparing Datasets for complete instructions for preparing the datasets.

Execution Instructions

Training

• Please see Getting Started for training commands.

Evaluation

• Please see Getting Started for evaluation commands.

Results

You can find our pretrained models in Getting Started.

ADE20K

Segmentation results on different Num Classes from ADE20K dataset, $bIoU$, %.

Method	Params	Crop Size	2	12	150
SegFormer (B0)	3.8M	320×320	63.2	52.4	37.9
TASFormer (emb)	4--14M	320×320	60.8	6.1	14.1
TASFormer (VSA emb)	4.1M	320×320	48.6	0.1	0.1
TASFormer (HF adapter)	7.3M	320×320	67.9	59.4	48.3
TASFormer (HF adapter++)	5.7M	320×320	67.9	58.4	47.6

Segmentation results of TASFormer (HF adapter) with different Crop Size on ADE20K dataset, $bIoU$, %.

Method	Params	Crop Size	$mIoU$, %	$bIoU$, %
TASFormer (HF adapter)	7.3M	320×320	14.6	48.3
TASFormer (HF adapter)	7.3M	640×640	14.7	51.1
TASFormer (HF adapter)	7.3M	896×896	14.4	52.0

Citation

If you found DAPS3D useful in your research, please consider starring ⭐ us on GitHub and citing 📚 us in your research!

@InProceedings{yudin2024tasformer,
title={TASFormer: Task-Aware Image Segmentation Transformer},
author={Yudin, Dmitry and Khorin, Aleksandr and Zemskova, Tatiana and Ovchinnikova, Darya},
booktitle={Neural Information Processing},
year={2024},
publisher={Springer Nature Singapore},
pages={305--317},
doi={10.1007/978-981-99-8073-4_24}
}