Skip to content
JerryYLiPublic

About

Code for CVPR 2022 paper "Improving Video Model Transfer with Dynamic Representation Learning"

Resources

Stars

0 stars

Watchers

1 watching

Forks

Latest commit

 

History

1 Commit

Folders and files

Repository files navigation

Improving Video Model Transfer with Dynamic Representation Learning (CVPR 2022)

Yi Li, Nuno Vasconcelos

Project page | Paper

This repository contains PyTorch implementation of dynamic representation learning (DRL), a framework for evaluating and improving the temporality of video representations.

Get started

The code is tested with Python 3.7+ and PyTorch 1.5.0+. To install required packages:

pip install torch torchvision torchaudio pyyaml tensorboard av opencv-python scikit-image pandas

Usage

Paths to video datasets should be updated to datasets/_paths.py. All examples below use 3D ResNet-18 as the backbone; additional configs are provided in configs/.

Baseline pre-training

python main.py configs/k400/res3d18.yaml

Dynamic representation learning

DRL by adversarial augmentation (FGSM, eps=0.03)

python main-drl.py configs/k400/res3d18.yaml configs/drl/fgsm-e3-i5.yaml

DRL by direct optimization (freeze, lambda=0.5)

python main-drl-dir.py configs/k400/res3d18.yaml configs/drl/kd0.5-freeze.yaml

Evaluation

Dynamic scoring on Kinetics (2D ResNet-18 as spatial model)

python main-drl.py configs/k400/res3d18.yaml configs/scoring/res2d18.yaml --pretrain <path-to-pretrained-model>

Fine-tuning on HMDB

python main.py configs/hmdb/res3d18.yaml --pretrain <path-to-pretrained-model>

Few-shot learning on Something-Something v2 (K=1)

python main-fsl.py configs/sthv2-fs/res3d18-1shot.yaml --pretrain <path-to-pretrained-model>

Acknowledgements

This work was funded in part by NSF awards IIS-1924937, IIS-2041009, a gift from Amazon, and a gift from Qualcomm. We acknowledge the use of Nautilus platform for some of the experiments in the paper.

Part of the code is adapted from the following repos:

Citation

If you find this repo useful, please cite our work. Thanks!

@inproceedings{li2022improving,
  title={Improving Video Model Transfer with Dynamic Representation Learning},
  author={Li, Yi and Vasconcelos, Nuno},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={19280--19291},
  year={2022}
}

About

Code for CVPR 2022 paper "Improving Video Model Transfer with Dynamic Representation Learning"

Resources

Stars

0 stars

Watchers

1 watching

Forks

Contributors

Languages