Source code of our ICCV 2023 paper Dual Learning with Dynamic Knowledge Distillation for Partially Relevant Video Retrieval.
- python 3.8
- pytorch 1.9.0
- torchvision 0.10.0
- tensorboard 2.6.0
- tqdm 4.62.0
- easydict 1.9
- h5py 2.10.0
- cuda 11.1
We used Anaconda to setup a deep learning workspace that supports PyTorch. Run the following script to install the required packages.
conda create --name DLDKD python=3.8
conda activate DLDKD
git clone https://github.com/HuiGuanLab/DL-DKD.git
cd DL-DKD
pip install -r requirements.txt
conda deactivate
Run the following script to download the video feature and text feature of the TVR dataset and place them in the specified path. The data can also be downloaded from Kuake pan. Please refer to here for more description of the dataset.
Run the following script to train DL-DKD
network on TVR. It will save the chechpoint that performs best on the validation set as the final model.
ROOTPATH=$HOME/VisualSearch
mkdir -p $ROOTPATH && cd $ROOTPATH
unzip activitynet.zip
#Add project root to PYTHONPATH (Note that you need to do this each time you start a new session.)
source setup.sh
conda activate DLDKD
ROOTPATH=$HOME/VisualSearch
./do_tvr.sh $ROOTPATH
The model is placed in the directory $ROOTPATH/$DATASET/results/$MODELDIR after training. To evaluate it, please run the following script:
DATASET=tvr
FEATURE=i3d_resnet
ROOTPATH=$HOME/VisualSearch
MODELDIR=tvr-double_dim384_kl_decay0.02_eval_3_7-2022_11_04_09_16_28
./do_test.sh $DATASET $FEATURE $ROOTPATH $MODELDIR
We also provide the trained checkpoint on TVR, run the following script to evaluate it. The model can also be downloaded from Here.
DATASET=tvr
FEATURE=i3d_resnet
ROOTPATH=$HOME/VisualSearch
MODELDIR=checkpoint_tvr
unzip checkpoint_tvr.zip -d $ROOTPATH/$DATASET/results
./do_test.sh $DATASET $FEATURE $ROOTPATH $MODELDIR
$DATASET
is the dataset that the model trained and evaluate on.
$FEATURE
is the video feature corresponding to the dataset.
$MODELDIR
is the path of checkpoints saved.
R@1 | R@5 | R@10 | R@100 | SumR | |
---|---|---|---|---|---|
Text-to-Video | 14.4 | 34.9 | 45.8 | 84.9 | 179.9 |
Run the following script to download the video feature and text feature of the Activitynet dataset and place them in the specified path. The data can also be downloaded from Kuake pan. Please refer to here for more description of the dataset.
ROOTPATH=$HOME/VisualSearch
mkdir -p $ROOTPATH && cd $ROOTPATH
unzip activitynet.zip
Run the following script to train DL-DKD
network on Activitynet.
#Add project root to PYTHONPATH (Note that you need to do this each time you start a new session.)
source setup.sh
conda activate DLDKD
ROOTPATH=$HOME/VisualSearch
./do_activitynet.sh $ROOTPATH
The model is placed in the directory $ROOTPATH/$DATASET/results/$MODELDIR after training. To evaluate it, please run the following script:
DATASET=activitynet
FEATURE=i3d
ROOTPATH=$HOME/VisualSearch
MODELDIR=activitynet-double_kl_8_ex_up_k800.0_loss_scale_weight1.0-2022_11_11_14_12_27
./do_test.sh $DATASET $FEATURE $ROOTPATH $MODELDIR
We also provide the trained checkpoint on Activitynet, run the following script to evaluate it. The model can also be downloaded from Here.
DATASET=activitynet
FEATURE=i3d
ROOTPATH=$HOME/VisualSearch
MODELDIR=checkpoint_activitynet
unzip checkpoint_activitynet.zip -d $ROOTPATH/$DATASET/results
./do_test.sh $DATASET $FEATURE $ROOTPATH $MODELDIR
R@1 | R@5 | R@10 | R@100 | SumR | |
---|---|---|---|---|---|
Text-to-Video | 8.0 | 25.0 | 37.5 | 77.1 | 147.6 |
@inproceedings{dong2023DLDKD,
title = {Dual Learning with Dynamic Knowledge Distillation for Partially Relevant Video Retrieval},
author = {Jianfeng Dong and Minsong Zhang and Zheng Zhang and Xianke Chen and Daizong Liu and Xiaoye Qu and Xun Wang and Baolong Liu},
booktitle = {IEEE International Conference on Computer Vision},
year = {2023},
}
The codes are modified from TVRetrieval and ReLoCLNet.
This work was supported by the National Key R&D Program of China (2018YFB1404102), NSFC (62172420,61902347, 61976188, 62002323), the Public Welfare Technology Research Project of Zhejiang Province (LGF21F020010), the Open Projects Program of the National Laboratory of Pattern Recognition, the Fundamental Research Funds for the Provincial Universities of Zhejiang, and Public Computing Cloud of RUC.