This is the project page for the paper:

ISTR: End-to-End Instance Segmentation via Transformers.

Updates

• (2022.03.09) New codes for ISTR-PCA, ISTR-DCT, and ISTR-SMT with better performance and speed have been released.
• (2021.05.03) The project page for ISTR is avaliable.

Method	backbone	fps	box AP	mask AP	link
ISTR-PCA	R50-FPN	13.0	46.7	39.8	7p58 (google drive)
ISTR-DCT	R50-FPN	12.5	46.9	40.2	ibi3
ISTR-SMT	R50-FPN	10.4	47.4	41.7	73bs (google drive)
ISTR-PCA	R101-FPN	10.7	48.0	41.1	5rcj
ISTR-DCT	R101-FPN	10.3	48.3	41.6	0mdl (google drive)
ISTR-SMT	R101-FPN	8.9	48.8	42.9	qbr8 (google drive)
ISTR-SMT	Swin-L	3.5	55.8	49.2	nuj8 (google drive)
ISTR-SMT@1088	Swin-L	2.9	56.4	49.7	9uj8

• The inference time is evaluated with a single 2080Ti GPU.
• We use the models pre-trained on ImageNet using torchvision. The ImageNet pre-trained ResNet-101 backbone is obtained from SparseR-CNN.

Installation

The codes are built on top of Detectron2, SparseR-CNN, and AdelaiDet.

Requirements

• Python=3.8
• PyTorch=1.6.0, torchvision=0.7.0, cudatoolkit=10.1
• OpenCV for visualization

Steps

1. Install the repository (we recommend to use Anaconda for installation.)

conda create -n ISTR python=3.8 -y
conda activate ISTR
conda install pytorch==1.6.0 torchvision==0.7.0 cudatoolkit=10.1 -c pytorch
or (conda install pytorch torchvision torchaudio cudatoolkit=11.3 -c pytorch)
pip install opencv-python
pip install scipy
pip install shapely
git clone https://github.com/hujiecpp/ISTR.git
cd ISTR
python setup.py build develop

2. Link coco dataset path

ln -s /coco_dataset_path/coco ./datasets

3. Train ISTR (e.g., with ResNet50 backbone)

python projects/ISTR/train_net.py --num-gpus 4 --config-file projects/ISTR/configs/ISTR-R50-3x.yaml

4. Evaluate ISTR (e.g., with ResNet50 backbone)

python projects/ISTR/train_net.py --num-gpus 4 --config-file projects/ISTR/configs/ISTR-R50-3x.yaml --eval-only MODEL.WEIGHTS ./output/model_final.pth

5. Visualize the detection and segmentation results (e.g., with ResNet50 backbone)

python demo/demo.py --config-file projects/ISTR/configs/ISTR-R50-3x.yaml --input input1.jpg --output ./output --confidence-threshold 0.4 --opts MODEL.WEIGHTS ./output/model_final.pth

Citation

If our paper helps your research, please cite it in your publications:

@article{hu2021istr,
  title={Istr: End-to-end instance segmentation with transformers},
  author={Hu, Jie and Cao, Liujuan and Lu, Yao and Zhang, ShengChuan and Wang, Yan and Li, Ke and Huang, Feiyue and Shao, Ling and Ji, Rongrong},
  journal={arXiv preprint arXiv:2105.00637},
  year={2021}
}
@article{hu2024istr,
  author={Hu, Jie and Lu, Yao and Zhang, Shengchuan and Cao, Liujuan},
  title={ISTR: Mask-Embedding-Based Instance Segmentation Transformer},
  journal={IEEE Transactions on Image Processing},  
  year={2024},
  volume={33},
  pages={2895-2907},
  doi={10.1109/TIP.2024.3385980}
}