HVOT Research Hub

This repository accompanies Hyperspectral video object tracking: A review and benchmark. It collects HVOT papers, official implementations, and benchmark metadata, together with the RGB and multimodal tracking methods most closely connected to spectral tracking. The main catalog contains the 41 HSP/MSP tracker variants evaluated in the review and a separate list of related methods.

HVOT Papers and Code

2018-2021: Early Spectral Tracking

Year	Tracker	Full paper title	Venue	Paper	Official implementation
2018	CNHT	Object Tracking in Hyperspectral Videos with Convolutional Features and Kernelized Correlation Filter	Smart Multimedia / LNCS	DOI	Not confirmed
2019	DeepHKCF	Tracking in Aerial Hyperspectral Videos Using Deep Kernelized Correlation Filters	IEEE TGRS	DOI	Code
2020	MHT	Material Based Object Tracking in Hyperspectral Videos	IEEE TIP	DOI	Code archive
2020	BAE-Net	BAE-Net: A Band Attention Aware Ensemble Network for Hyperspectral Object Tracking	ICIP	DOI	Not confirmed
2021	MFI	Multi-Features Integration Based Hyperspectral Videos Tracker	WHISPERS	DOI	Not confirmed
2021	SST-Net	Spectral-Spatial-Temporal Attention Network for Hyperspectral Tracking	WHISPERS	DOI	Not confirmed

2022: Band Selection, Correlation Filters, and Siamese Transfer

Year	Tracker	Full paper title	Venue	Paper	Official implementation
2022	SiamHYPER	SiamHYPER: Learning a Hyperspectral Object Tracker From an RGB-Based Tracker	IEEE TIP	DOI	Results
2022	TSCFW	Spatial-Spectral Weighted and Regularized Tensor Sparse Correlation Filter for Object Tracking in Hyperspectral Videos	IEEE TGRS	DOI	Not confirmed
2022	BAHT	Robust Hyperspectral Object Tracking by Exploiting Background-Aware Spectral Information With Band Selection Network	IEEE GRSL	DOI	Not confirmed
2022	TASSCF	Target-Aware and Spatial-Spectral Discriminant Feature Joint Correlation Filters for Hyperspectral Video Object Tracking	CVIU	DOI	Not confirmed
2022	DeepTASSCF	Deep-feature variant reported with TASSCF	CVIU	DOI	Not confirmed
2022	SiamF	Material-Guided Siamese Fusion Network for Hyperspectral Object Tracking	ICASSP	DOI	Not confirmed
2022	HOMG	Histograms of Oriented Mosaic Gradients for Snapshot Spectral Image Description	ISPRS JPRS	DOI	Code

2023: Deep Band Modeling and Transformer Prototypes

Year	Tracker	Full paper title	Venue	Paper	Official implementation
2023	ViPT*	Visual Prompt Multi-Modal Tracking	CVPR	CVF	Code
2023	SEE-Net	Learning a Deep Ensemble Network With Band Importance for Hyperspectral Object Tracking	IEEE TIP	DOI	Code
2023	SiamOHOT	SiamOHOT: A Lightweight Dual Siamese Network for Onboard Hyperspectral Object Tracking via Joint Spatial-Spectral Knowledge Distillation	IEEE TGRS	DOI	Not confirmed
2023	SiamBAG	SiamBAG: Band Attention Grouping-Based Siamese Object Tracking Network for Hyperspectral Videos	IEEE TGRS	DOI	Code
2023	CBFF-Net	CBFF-Net: A New Framework for Efficient and Accurate Hyperspectral Object Tracking	IEEE TGRS	DOI	Not confirmed
2023	Trans-HST	A Transformer-Based Network for Hyperspectral Object Tracking	IEEE TGRS	DOI	Not confirmed
2023	SiamHT	A Siamese Network-Based Tracking Framework for Hyperspectral Video	Neural Computing and Applications	DOI	Not confirmed
2023	BABS	Background-Aware Band Selection for Object Tracking in Hyperspectral Videos	IEEE GRSL	IEEE	Not confirmed

ViPT* is an adapted multimodal prompt tracker rather than a native HVOT method.

2024: Prompt Learning, Domain Adaptation, and Dynamic Templates

Year	Tracker	Full paper title	Venue	Paper	Official implementation
2024	SPIRIT	SPIRIT: Spectral Awareness Interaction Network With Dynamic Template for Hyperspectral Object Tracking	IEEE TGRS	DOI	Results
2024	TBR-Net	Transformer-Based Band Regrouping With Feature Refinement for Hyperspectral Object Tracking	IEEE TGRS	DOI	Not confirmed
2024	SENSE	SENSE: Hyperspectral Video Object Tracker via Fusing Material and Motion Cues	Information Fusion	DOI	Code
2024	MMF-Net	Material-Guided Multiview Fusion Network for Hyperspectral Object Tracking	IEEE TGRS	DOI	Code
2024	PHTrack	PHTrack: Prompting for Hyperspectral Video Tracking	IEEE TGRS	DOI	Code
2024	HA-Net	A Deep Temporal-Spectral-Spatial Anchor-Free Siamese Tracking Network for Hyperspectral Video Object Tracking	IEEE TGRS	DOI	Results
2024	Hy-Tracker	Hy-Tracker: A Novel Framework for Enhancing Efficiency and Accuracy of Object Tracking in Hyperspectral Videos	IEEE TGRS	DOI	Not confirmed
2024	SiamCAT	A Channel Adaptive Dual Siamese Network for Hyperspectral Object Tracking	IEEE TGRS	DOI	Not confirmed
2024	Trans-DAT	Domain Adaptation-Aware Transformer for Hyperspectral Object Tracking	IEEE TCSVT	DOI	Code

2025: Unified Models, Memory, MoE, and Larger Benchmarks

Year	Tracker	Full paper title	Venue	Paper	Official implementation
2025	HDSP	Hyperspectral Object Tracking With Dual-Stream Prompt	IEEE TGRS	DOI	Code
2025	SSTtrack	SSTtrack: A Unified Hyperspectral Video Tracking Framework via Modeling Spectral-Spatial-Temporal Conditions	Information Fusion	DOI	Code
2025	DaSSP-Net	Multi-Domain Universal Representation Learning for Hyperspectral Object Tracking	Pattern Recognition	DOI	Code
2025	HotMoE	HotMoE: Exploring Sparse Mixture-of-Experts for Hyperspectral Object Tracking	IEEE TMM	DOI	Code
2025	SASUNet	SASU-Net: Hyperspectral Video Tracker Based on Spectral Adaptive Aggregation Weighting and Scale Updating	Expert Systems with Applications	DOI	Not confirmed
2025	SPDAN	BihoT: A Large-Scale Dataset and Benchmark for Hyperspectral Camouflaged Object Tracking	IEEE TNNLS	DOI	Not confirmed
2025	SSF-Net	SSF-Net: Spatial-Spectral Fusion Network With Spectral Angle Awareness for Hyperspectral Object Tracking	IEEE TIP	DOI	Not confirmed
2025	SpectralTrack	Hyperspectral Video Tracking With Spectral-Spatial Fusion and Memory Enhancement	IEEE TIP	DOI	Code
2025	SpectralTrack+	Enhanced variant reported in the SpectralTrack paper	IEEE TIP	DOI	Code
2025	ProFiT	ProFiT: A Prompt-Guided Frequency-Aware Filtering and Template-Enhanced Interaction Framework for Hyperspectral Video Tracking	ISPRS JPRS	DOI	Code
2025	UNTrack	MUST: The First Dataset and Unified Framework for Multispectral UAV Single Object Tracking	CVPR	CVF	Code and data

Additional Related Methods

The following papers are not part of the 41-variant comparison table.

Year	Tracker	Full paper title	Venue	Paper	Official implementation
2022	H3Net	Unsupervised Deep Hyperspectral Video Target Tracking and High Spectral-Spatial-Temporal Resolution Benchmark Dataset	IEEE TGRS	DOI	Results
2022	TrTSN	A Transformer-Based Three-Branch Siamese Network for Hyperspectral Object Tracking	WHISPERS	DOI	Not confirmed
2023	HELIOS	HELIOS: Hyperspectral Hindsight OSTracker	WHISPERS	DOI	Code
2024	HySSTP	HySSTP: Hyperspectral Video Tracker Embedding Multi-Modal Spatial-Spectral-Temporal Features	WHISPERS	DOI	Not confirmed
2025	HCSMP	Hyperspectral Video Object Tracking with Cross-Modal Spectral Complementary and Memory Prompt Network	Knowledge-Based Systems	DOI	Not confirmed
2025	VSS	A Multi-Stream Visual-Spectral-Spatial Adaptive Hyperspectral Object Tracking	ICMR	DOI	Not confirmed
2025	UBSTrack	UBSTrack: Unified Band Selection and Multi-Model Ensemble for Hyperspectral Object Tracking	IEEE TGRS	DOI	Code
2026	HyperTrack	HyperTrack: A Unified Network for Hyperspectral Video Object Tracking	IEEE TCSVT	Publisher record pending	Code

RGB Tracking Foundations

Many deep HVOT methods inherit their matching architecture, backbone, prompt design, or temporal update mechanism from RGB tracking.

Year	Tracker	Full paper title	Venue	Paper	Code	Connection to HVOT
2015	KCF	High-Speed Tracking with Kernelized Correlation Filters	IEEE TPAMI	DOI	Not listed	Online correlation filtering
2016	SiamFC	Fully-Convolutional Siamese Networks for Object Tracking	ECCV Workshops	Paper	Code	Template-search matching
2018	CSRDCF	Discriminative Correlation Filter Tracker with Channel and Spatial Reliability	IJCV	DOI	Code	Channel reliability
2018	SiamRPN	High Performance Visual Tracking with Siamese Region Proposal Network	CVPR	CVF	Code	Proposal-based state estimation
2019	SiamRPN++	SiamRPN++: Evolution of Siamese Visual Tracking With Very Deep Networks	CVPR	CVF	Code	Deep Siamese backbones
2019	ATOM	ATOM: Accurate Tracking by Overlap Maximization	CVPR	CVF	Code	Classification and IoU estimation
2019	DiMP	Learning Discriminative Model Prediction for Tracking	ICCV	CVF	Code	Online discriminative prediction
2020	SiamBAN	Siamese Box Adaptive Network for Visual Tracking	CVPR	CVF	Code	Anchor-free box prediction
2020	SiamCAR	SiamCAR: Siamese Fully Convolutional Classification and Regression for Visual Tracking	CVPR	CVF	Code	Joint classification and regression
2021	TransT	Transformer Tracking	CVPR	CVF	Code	Transformer feature fusion
2021	STARK	Learning Spatio-Temporal Transformer for Visual Tracking	ICCV	CVF	Code	Spatio-temporal state prediction
2022	OSTrack	Joint Feature Learning and Relation Modeling for Tracking: A One-Stream Framework	ECCV	Paper	Code	One-stream prompting backbone
2022	MixFormer	MixFormer: End-to-End Tracking With Iterative Mixed Attention	CVPR	CVF	Code	Mixed attention and template interaction
2023	ARTrack	Autoregressive Visual Tracking	CVPR	CVF	Code	Sequence generation
2024	ODTrack	ODTrack: Online Dense Temporal Token Learning for Visual Tracking	AAAI	AAAI	Code	Online temporal tokens
2024	SAMURAI	SAMURAI: Adapting Segment Anything Model for Zero-Shot Visual Tracking with Motion-Aware Memory	arXiv	Paper	Code	Motion-aware memory

Multimodal and Unified Tracking

Year	Tracker	Modalities	Full paper title	Venue	Paper	Code	Connection to HVOT
2021	DeT	RGB-D	DepthTrack: Unveiling the Power of RGBD Tracking	ICCV	CVF	Code	Auxiliary-modality feature extraction
2023	ViPT	RGB-D/T/E	Visual Prompt Multi-Modal Tracking	CVPR	CVF	Code	Parameter-efficient modality prompts
2024	BAT	RGB-D/T/E	Bi-Directional Adapter for Multi-Modal Tracking	AAAI	AAAI	Code	Bidirectional cross-modal adaptation
2024	OneTracker	RGB-D/T/E/L	OneTracker: Unifying Visual Object Tracking with Foundation Models and Efficient Tuning	CVPR	CVF	Not listed	Unified architecture and efficient tuning
2024	Un-Track	Any RGB-X subset	Single-Model and Any-Modality for Video Object Tracking	CVPR	CVF	Code	One model across modality combinations
2025	SUTrack	RGB/D/T/E/L	SUTrack: Towards Simple and Unified Single Object Tracking	AAAI	AAAI	Code	Unified task representation
2025	UM-ODTrack	RGB/D/T/E	Towards Universal Modal Tracking With Online Dense Temporal Token Learning	IEEE TPAMI	DOI	Code	Universal modalities and temporal memory

RGB-X tracking usually combines observations from separate sensors and must handle calibration, alignment, or missing modalities. Snapshot HSP/MSP tracking instead works with co-registered bands from one spectral sensor, so RGB-X fusion modules require adaptation to within-cube band relationships.

Dataset Landscape

The tables below show the dataset versions and splits used in the review. Sequence count, frame count, spatial resolution, wavelength range, band count, and task setting are given directly for comparison.

Hyperspectral and Multispectral Tracking Datasets

Dataset	Modality	Sequences	Frames	Resolution	Spectrum / bands	Resources
HOTC20	HSP	75	29,390	512 x 256	470-620 nm / 16	Paper
HOTC23-NIR	HSP	70	13,017	Not reported	665-960 nm / 25	Dataset owner
HOTC23-RedNIR	HSP	26	9,912	Not reported	600-850 nm / 15	Dataset owner
HOTC23-VIS	HSP	101	39,302	Not reported	460-600 nm / 16	Dataset owner
HOTC24-NIR	HSP	100	18,160	Not reported	665-960 nm / 25	Dataset owner
HOTC24-RedNIR	HSP	56	24,258	Not reported	600-850 nm / 15	Dataset owner
HOTC24-VIS	HSP	178	82,585	Not reported	460-600 nm / 16	Dataset owner
WHU-Hi-H3	HSP	69 evaluation sequences	29,722	409 x 216	600-900 nm / 25	Paper
BihoT	HSP camouflage	49	41,912	409 x 217	600-974 nm / 25	Paper / Preprint
MSVT	MSP	135	28,167	2045 x 1080	680-960 nm / 25	Paper / Code
MSSOT	MSP	185	63,082	2045 x 1080	680-960 nm / 25	Paper / Code and data
BihoT-130K-TRA	HSP camouflage	163	128,205	409 x 217	600-974 nm / 25	Paper
MSITrack	MSP	300	129,140	1200 x 900	395-950 nm / 8	Paper / Data
MUST	MSP UAV	250	42,671	1200 x 900	390-950 nm / 8	Paper / Code and data

Representative RGB Datasets

Dataset	Main role	Resource
OTB100	Classical one-pass evaluation and attributes	Paper
UAV123	Aerial, small-object, and camera-motion tracking	Dataset
TrackingNet	Large-scale RGB training and evaluation	Paper
LaSOT	Long-term category-balanced tracking	Paper
GOT-10k	One-shot and class-disjoint evaluation	Project
TOTB	Transparent-object tracking	Project
VOT	Reset-based short- and long-term challenges	Challenge

Representative Multimodal Datasets

Modality	Dataset	Main role	Resource
RGB-D	DepthTrack	Large RGB-D tracking benchmark	Paper / Data
RGB-D	D2CUBE	RGB-D aerial tracking	Paper
RGB-T	LasHeR	Large-scale visible-thermal tracking	Data
RGB-E	VisEvent	Paired frame-event tracking	Project / Data
RGB-E	COESOT	Event-assisted object tracking	Paper / Data
Event	EventVOT	Event-only tracking	Paper / Data
RGB-L	MGIT	Language-guided global instance tracking	Paper
RGB-L	VastTrack	Large-scale vision-language tracking	Paper

Method Taxonomy

The categories are complementary: a tracker may use more than one input representation, spectral mechanism, or temporal strategy.

Input Representation

Representation	Description	Benefits	Main risks	Representative methods
Native spectral cube	Processes most or all spectral bands	Preserves spectral continuity	High dimensionality and sensor-specific channels	CNHT, MHT, TSCFW, SPIRIT, Trans-DAT
Selected bands	Chooses a compact subset of bands	Low cost and RGB compatibility	Selection instability and information loss	BAHT, BABS, Hy-Tracker, UBSTrack
Single false-color view	Maps bands to three channels	Direct reuse of RGB trackers	Spectral compression and color-prior mismatch	DeepHKCF, SiamHT, adapted ViPT
Multiple false-color views	Builds several three-channel band groups	Greater spectral diversity	Fragmented spectra and repeated inference	BAE-Net, SEE-Net, SiamBAG, HotMoE
Native plus false-color branches	Combines spectral and RGB-compatible views	Balances spectral cues and RGB priors	Fusion cost and branch dominance	SiamHYPER, HA-Net, HDSP, SpectralTrack
Material or abundance view	Adds unmixing/material features	Interpretable physical cues	Sensitive to unmixing quality	MHT, SENSE, MMF-Net

Tracker Prototype

Family	Core mechanism	Representative HVOT methods
Correlation filter	Online discriminative filtering in the Fourier domain	CNHT, DeepHKCF, MHT, TSCFW, TASSCF
Deep ensemble	Several band-derived weak trackers are combined	BAE-Net, SEE-Net
Siamese matching	Shared template/search encoders with similarity or regression heads	SiamHYPER, SiamBAG, SiamOHOT, SiamCAT
Transformer	Global template-search and cross-band interaction	Trans-HST, TBR-Net, Trans-DAT
Prompt/adaptation	A pretrained RGB/foundation tracker is adapted with a small spectral module	PHTrack, HDSP, ProFiT, DaSSP-Net
Detection-assisted	Detection or candidate generation supports tracking and re-localization	Hy-Tracker, UNTrack
Mixture of experts	Sparse routing selects spectral experts	HotMoE

Spectral Modeling

• Band importance: estimate which wavelengths contribute to the target/background distinction.
• Band grouping: form several band subsets or false-color views for independent feature extraction.
• Cross-band interaction: use attention, fusion, or graph-like operations between wavelengths.
• Spectral angle or material similarity: compare physical signatures rather than only texture.
• Domain adaptation: normalize sensor-specific band counts and wavelength ranges.
• Frequency-aware filtering: separate complementary spatial or spectral frequency components.

Temporal Modeling

Temporal design	What it contributes	Representative methods
Fixed first-frame template	Stable initialization and simple inference	Early CF and Siamese trackers
Confidence-based template update	Adaptation to appearance and spectral change	SPIRIT, SASUNet
Motion cue fusion	Explicit displacement or trajectory information	SENSE, Hy-Tracker
Online discrimination	Target-specific learning during tracking	HA-Net
Memory bank or temporal prompts	Long-range context and re-localization support	SSTtrack, SpectralTrack, HCSMP
Video-level token learning	Joint modeling across a longer clip	Unified RGB-X methods such as UM-ODTrack

Adaptation and Generalization

The small size of most spectral-video datasets makes adaptation strategy a first-order design choice:

1. Full fine-tuning offers high flexibility but can overfit.
2. Frozen RGB backbone plus spectral branch preserves RGB knowledge but may let the RGB branch dominate.
3. Adapters and prompts reduce trainable parameters and help transfer foundation trackers.
4. Multi-domain learning aims to share representations across sensors and wavelength ranges.
5. Unified RGB-HSP-MSP models reduce duplicated architectures, but must preserve transparent modality-specific preprocessing.