Skip to content
/ pcan Public

Prototypical Cross-Attention Networks for Multiple Object Tracking and Segmentation, NeurIPS 2021 Spotlight


Notifications You must be signed in to change notification settings


Folders and files

Last commit message
Last commit date

Latest commit



50 Commits

Repository files navigation


Prototypical Cross-Attention Networks for Multiple Object Tracking and Segmentation

This is the offical implementation of paper PCAN for MOTS. PCAN is also served as the baseline method in BDD100K tracking challenges at CVPR 2022.

We also present a trailer that consists of method illustrations and tracking & segmentation visualizations. Our project website contains more information:

Prototypical Cross-Attention Networks for Multiple Object Tracking and Segmentation
NeurIPS 2021, Spotlight
Lei Ke, Xia Li, Martin Danelljan, Yu-Wing Tai, Chi-Keung Tang, Fisher Yu


Multiple object tracking and segmentation requires detecting, tracking, and segmenting objects belonging to a set of given classes. Most approaches only exploit the temporal dimension to address the association problem, while relying on single frame predictions for the segmentation mask itself. We propose Prototypical Cross-Attention Network (PCAN), capable of leveraging rich spatio-temporal information for online multiple object tracking and segmentation. PCAN first distills a space-time memory into a set of prototypes and then employs cross-attention to retrieve rich information from the past frames. To segment each object, PCAN adopts a prototypical appearance module to learn a set of contrastive foreground and background prototypes, which are then propagated over time. Extensive experiments demonstrate that PCAN outperforms current video instance tracking and segmentation competition winners on both Youtube-VIS and BDD100K datasets, and shows efficacy to both one-stage and two-stage segmentation frameworks.

Prototypical Cross-Attention Networks (PCAN)

Main results

Detector mMOTSA-val mIDF1-val ID Sw.-val Scores-val mMOTSA-test mIDF1-test ID Sw.-test Scores-test Config Weights Preds Visuals
ResNet-50 28.1 45.4 874 scores 31.9 50.4 845 scores config model | MD5 preds visuals


Please refer to for installation instructions.


Please refer to for dataset preparation and detailed running (training, testing, visualization, etc.) instructions.

Related links

Youtube Video | Bilibili Video| Zhihu Reading


If you find PCAN useful in your research or refer to the provided baseline results, please star ⭐ this repository and consider citing 📝:

  title={Prototypical Cross-Attention Networks for Multiple Object Tracking and Segmentation},
  author={Ke, Lei and Li, Xia and Danelljan, Martin and Tai, Yu-Wing and Tang, Chi-Keung and Yu, Fisher},
  booktitle={Advances in Neural Information Processing Systems},