Skip to content

[ICCV'21] Learning Spatio-Temporal Transformer for Visual Tracking


Notifications You must be signed in to change notification settings


Folders and files

Last commit message
Last commit date

Latest commit



60 Commits

Repository files navigation


The official implementation of the ICCV2021 paper Learning Spatio-Temporal Transformer for Visual Tracking

Hiring research interns for visual transformer projects:


  • STARK has been integrated into the mmtracking library!
  • 🏆 We are the Winner of VOT-21 RGB-D challenge
  • 🏆 We won the Runner-ups in VOT-21 Real-Time and Long-term challenges
  • We release an extremely fast version of STARK called STARK-Lightning ⚡ . It can run at 200~300 FPS on a RTX TITAN GPU. Besides, its performance can beat DiMP50, while the model size is even less than that of SiamFC! More details can be found at中文教程
  • The raw results of STARK and other trackers on NOTU (NFS, OTB100, TC128, UAV123) have been uploaded to here STARK_Framework


End-to-End, Post-processing Free

STARK is an end-to-end tracking approach, which directly predicts one accurate bounding box as the tracking result.
Besides, STARK does not use any hyperparameters-sensitive post-processing, leading to stable performances.

Real-Time Speed

STARK-ST50 and STARK-ST101 run at 40FPS and 30FPS respectively on a Tesla V100 GPU.

Strong performance

Tracker LaSOT (AUC) GOT-10K (AO) TrackingNet (AUC)
STARK 67.1 68.8 82.0
TransT 64.9 67.1 81.4
TrDiMP 63.7 67.1 78.4
Siam R-CNN 64.8 64.9 81.2

Purely PyTorch-based Code

STARK is implemented purely based on the PyTorch.

Install the environment

Option1: Use the Anaconda

conda create -n stark python=3.6
conda activate stark

Option2: Use the docker file

We provide the complete docker at here

Data Preparation

Put the tracking datasets in ./data. It should look like:

 -- data
     -- lasot
         |-- airplane
         |-- basketball
         |-- bear
     -- got10k
         |-- test
         |-- train
         |-- val
     -- coco
         |-- annotations
         |-- images
     -- trackingnet
         |-- TRAIN_0
         |-- TRAIN_1
         |-- TRAIN_11
         |-- TEST

Set project paths

Run the following command to set paths for this project

python tracking/ --workspace_dir . --data_dir ./data --save_dir .

After running this command, you can also modify paths by editing these two files

lib/train/admin/  # paths about training
lib/test/evaluation/  # paths about testing


Training with multiple GPUs using DDP

python tracking/ --script stark_s --config baseline --save_dir . --mode multiple --nproc_per_node 8  # STARK-S50
python tracking/ --script stark_st1 --config baseline --save_dir . --mode multiple --nproc_per_node 8  # STARK-ST50 Stage1
python tracking/ --script stark_st2 --config baseline --save_dir . --mode multiple --nproc_per_node 8 --script_prv stark_st1 --config_prv baseline  # STARK-ST50 Stage2
python tracking/ --script stark_st1 --config baseline_R101 --save_dir . --mode multiple --nproc_per_node 8  # STARK-ST101 Stage1
python tracking/ --script stark_st2 --config baseline_R101 --save_dir . --mode multiple --nproc_per_node 8 --script_prv stark_st1 --config_prv baseline_R101  # STARK-ST101 Stage2

(Optionally) Debugging training with a single GPU

python tracking/ --script stark_s --config baseline --save_dir . --mode single

Test and evaluate STARK on benchmarks

  • LaSOT
python tracking/ stark_st baseline --dataset lasot --threads 32
python tracking/ # need to modify tracker configs and names
  • GOT10K-test
python tracking/ stark_st baseline_got10k_only --dataset got10k_test --threads 32
python lib/test/utils/ --tracker_name stark_st --cfg_name baseline_got10k_only
  • TrackingNet
python tracking/ stark_st baseline --dataset trackingnet --threads 32
python lib/test/utils/ --tracker_name stark_st --cfg_name baseline
  • VOT2020
    Before evaluating "STARK+AR" on VOT2020, please install some extra packages following external/AR/
cd external/vot20/<workspace_dir>
export PYTHONPATH=<path to the stark project>:$PYTHONPATH
  • VOT2020-LT
cd external/vot20_lt/<workspace_dir>
export PYTHONPATH=<path to the stark project>:$PYTHONPATH

Test FLOPs, Params, and Speed

# Profiling STARK-S50 model
python tracking/ --script stark_s --config baseline
# Profiling STARK-ST50 model
python tracking/ --script stark_st2 --config baseline
# Profiling STARK-ST101 model
python tracking/ --script stark_st2 --config baseline_R101
# Profiling STARK-Lightning-X-trt
python tracking/

Model Zoo

The trained models, the training logs, and the raw tracking results are provided in the model zoo



[ICCV'21] Learning Spatio-Temporal Transformer for Visual Tracking








No releases published


No packages published