Depth-Guided Video Object Counting in Crowded Scenes

Accepted by ACM Multimedia 2026

Official PyTorch implementation of Depth-Guided Video Object Counting in Crowded Scenes, a depth-aware method for video instance counting. The repository supports training, test-set evaluation, and inference with visual exemplars.

Contents

• Installation
• Data Preparation
• Checkpoints
• Reproduce Test Results
• Demo
• Training
• Results
• Citation
• Acknowledgements and License

Installation

The release was tested on Linux with Python 3.10.18, CUDA Toolkit 12.8, GCC 11.3, and an NVIDIA GeForce RTX 5090 GPU. An NVIDIA driver compatible with CUDA 12.8 is required.

Create the Conda environment and install the pinned Python dependencies from the repository root:

conda env create -f environment.yml
conda activate depth-vic
pip install -r requirements.txt

Build the Multi-Scale Deformable Attention CUDA extension after PyTorch is installed:

cd src/models/ops
python setup.py build install
cd ../../..
python -c "import torch, MultiScaleDeformableAttention"

environment.yml fixes the Python, CUDA toolkit, and compiler environment. requirements.txt fixes the Python packages, including the CUDA-enabled PyTorch wheels.

Data Preparation

The RGBD-VideoCount dataset is distributed separately through Hugging Face Datasets. Download the dataset archive from the project Hugging Face dataset repository, extract it, and arrange the files as follows:

data/
├── RGBD-VideoCount/
│   ├── images/
│   ├── Depth_Data/
│   ├── object_annotations/
│   ├── count_annotations/
│   ├── dataset_split.json
│   ├── video_class.txt
│   ├── exemplars_train.json
│   ├── exemplars_val.json
│   └── exemplars_test.json

The three exemplar JSON files are required for visual-exemplar training and evaluation. See data/README.md for the dataset directory contract.

Checkpoints

Place the following files under checkpoints/ before running training or evaluation:

checkpoints/
├── bert-base-uncased/
├── countgd_box.pth
└── DG-Net.pth

countgd_box.pth initializes the detector for training. DG-Net.pth is the released inference-only checkpoint. Both are distributed separately through the project Hugging Face model repository; see checkpoints/README.md for the expected local filenames.

Reproduce Test Results

Run test-set evaluation with visual exemplars:

python scripts/evaluate.py --weights_path checkpoints/DG-Net.pth --config configs/config_test.py --split test --data_root data/RGBD-VideoCount --output_json_path results/test_with_visual_exemplars.json --use_exemplar_infer --exemplars_file data/RGBD-VideoCount/exemplars_test.json

The command evaluates 30 test videos and 44 video-prompt entries. The output JSON stores global metrics, per-entry predicted counts and ground truth, and frame-wise boxes. It intentionally excludes per-class AP and IoU breakdowns.

Run test-set evaluation without visual exemplars:

python scripts/evaluate.py --weights_path checkpoints/DG-Net.pth --config configs/config_test.py --split test --data_root data/RGBD-VideoCount --output_json_path results/test_without_visual_exemplars.json

The reproduced artifact and qualitative media are documented in results/README.md.

Demo

Qualitative examples from the released test predictions are shown below.

To render a GIF from an evaluation JSON file, run:

python scripts/demo.py --predictions results/test_with_visual_exemplars.json --data_root data/RGBD-VideoCount --videos RGB0009__beverage --output_dir demo

Training

Train Depth-VIC from the CountGD initialization checkpoint:

python scripts/train.py --data_root data/RGBD-VideoCount --config configs/config_val.py --pretrain_model_path checkpoints/countgd_box.pth --output_dir outputs/DG-Net

Training writes TensorBoard events and the best validation-loss checkpoint to the selected output directory.

Results

The results/ directory contains the reproduced visual-exemplar test output, the framework overview, and qualitative GIFs. The GIFs use 600 ms per frame and preserve the source resolution.

• results/beverage.gif
• results/books.gif
• results/beverage_and_bag.gif
• results/cabinet.gif
• results/overview.png
• results/test_with_visual_exemplars.json

Citation

If you use this code, please cite our paper:

@inproceedings{xu2026depth,
  title     = {Depth-Guided Video Object Counting in Crowded Scenes},
  author    = {Xu, Yuanjing and Liu, Xinyan and Chen, Weidong and Zou, Zixuan and Zhang, Linhao and Meng, Zhuangzhe and Chan, Antoni B. and Zhang, Weigang},
  booktitle = {Proceedings of the 34th ACM International Conference on Multimedia},
  year      = {2026},
  doi       = {10.1145/3767308.3835482}
}

Acknowledgements and License

This project builds on CountVid, CountGD, GroundingDINO, DETR, Deformable DETR, and Swin Transformer. Their copyright notices are retained in the derived source files.

Depth-VIC project-owned code is released under the MIT License. Third-party components retain their original licenses and notices; see THIRD_PARTY_NOTICES.md.