Skip to content

AAAI‘20 - Learning 2D Temporal Localization Networks for Moment Localization with Natural Language

License

Notifications You must be signed in to change notification settings

zhzhuangxue/2D-TAN

 
 

Repository files navigation

2D-TAN

we are hiring talented interns: houwen.peng@microsoft.com

In this paper, we study the problem of moment localization with natural language, and propose a novel 2D Temporal Adjacent Networks(2D-TAN) method. The core idea is to retrieve a moment on a two-dimensional temporal map, which considers adjacent moment candidates as the temporal context. 2D-TAN is capable of encoding adjacent temporal relation, while learning discriminative feature for matching video moments with referring expressions. Our model is simple in design and achieves competitive performance in comparison with the state-of-the-art methods on three benchmark datasets.

Arxiv Preprint

News

Framework

alt text

Main Results

Main results on Charades-STA

Method Rank1@0.5 Rank1@0.7 Rank5@0.5 Rank5@0.7
Pool 39.70 23.31 80.32 51.26
Conv 39.81 23.25 79.33 52.15

Main results on ActivityNet Captions

Method Rank1@0.3 Rank1@0.5 Rank1@0.7 Rank5@0.3 Rank5@0.5 Rank5@0.7
Pool 59.45 44.51 26.54 85.53 77.13 61.96
Conv 58.75 44.05 27.38 85.65 76.65 62.26

Main results on TACoS

Method Rank1@0.1 Rank1@0.3 Rank1@0.5 Rank5@0.1 Rank5@0.3 Rank5@0.5
Pool 47.59 37.29 25.32 70.31 57.81 45.04
Conv 46.39 35.17 25.17 74.46 56.99 44.24

Prerequisites

  • pytorch 1.1.0
  • python 3.7
  • torchtext
  • easydict
  • terminaltables

Quick Start

Please download the visual features from onedrive and save it to the data/ folder.

Training

Use the following commands for training:

# Evaluate "Pool" in Table 1
python moment_localization/train.py --cfg experiments/charades/2D-TAN-16x16-K5L8-pool.yaml --verbose
# Evaluate "Conv" in Table 1
python moment_localization/train.py --cfg experiments/charades/2D-TAN-16x16-K5L8-conv.yaml --verbose

# Evaluate "Pool" in Table 2
python moment_localization/train.py --cfg experiments/activitynet/2D-TAN-64x64-K9L4-pool.yaml --verbose
# Evaluate "Conv" in Table 2
python moment_localization/train.py --cfg experiments/activitynet/2D-TAN-64x64-K9L4-conv.yaml --verbose

# Evaluate "Pool" in Table 3
python moment_localization/train.py --cfg experiments/tacos/2D-TAN-128x128-K5L8-pool.yaml --verbose
# Evaluate "Conv" in Table 3
python moment_localization/train.py --cfg experiments/tacos/2D-TAN-128x128-K5L8-conv.yaml --verbose

Testing

Our trained model are provided in onedrive. Please download them to the checkpoints folder. Then, run the following commands for evaluation:

# Evaluate "Pool" in Table 1
python moment_localization/test.py --cfg experiments/charades/2D-TAN-16x16-K5L8-pool.yaml --verbose --split test
# Evaluate "Conv" in Table 1
python moment_localization/test.py --cfg experiments/charades/2D-TAN-16x16-K5L8-conv.yaml --verbose --split test

# Evaluate "Pool" in Table 2
python moment_localization/test.py --cfg experiments/activitynet/2D-TAN-64x64-K9L4-pool.yaml --verbose --split test
# Evaluate "Conv" in Table 2
python moment_localization/test.py --cfg experiments/activitynet/2D-TAN-64x64-K9L4-conv.yaml --verbose --split test

# Evaluate "Pool" in Table 3
python moment_localization/test.py --cfg experiments/tacos/2D-TAN-128x128-K5L8-pool.yaml --verbose --split test
# Evaluate "Conv" in Table 3
python moment_localization/test.py --cfg experiments/tacos/2D-TAN-128x128-K5L8-conv.yaml --verbose --split test

Citation

If any part of our paper and code is helpful to your work, please generously cite with:

@InProceedings{2DTAN_2020_AAAI,
author = {Zhang, Songyang and Peng, Houwen and Fu, Jianlong and Luo, Jiebo},
title = {Learning 2D Temporal Adjacent Networks forMoment Localization with Natural Language},
booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition (AAAI)},
year = {2020}
} 

About

AAAI‘20 - Learning 2D Temporal Localization Networks for Moment Localization with Natural Language

Resources

License

Code of conduct

Security policy

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Python 98.6%
  • Shell 1.4%