Skip to content
A Dataset for Grounded Video Description
Branch: master
Clone or download
Type Name Latest commit message Commit time
Failed to load latest commit information.
data update the skeleton file Feb 21, 2019
demo initial commit Jan 25, 2019
scripts sanitize eval script Feb 19, 2019 initial commit Jan 25, 2019
LICENSE update README Feb 19, 2019

ActivityNet Entities dataset

This repo hosts the dataset used in our paper Grounded Video Description.

ActivityNet-Entities, is based on the video description dataset ActivityNet Captions and augments it with 158k bounding box annotations, each grounding a noun phrase (NP). Here we release the complete set of NP-based annotations as well as the pre-processed object-based annotations.

dataset teaser


We have the following dataset files under the data directory:

  • anet_entities_trainval.json: The raw dataset file with noun phrase and bounding box annotations. We only release the training and the validation splits for now.
  • anet_entities_cleaned_class_thresh50_trainval.json: Pre-processed dataset file with object class and bounding box annotations. For training and validation splits only.
  • anet_entities_cleaned_class_thresh50_test_skeleton.json: Object class annotation for the testing split. This file is for evaluation server purpose and the bounding box annotation is not given. See below for more details.
  • anet_entities_skeleton.txt: Specify the expected structure of the JSON annotation files.
  • split_ids_anet_entities.json: Video IDs included in the training/validation/testing splits.

Note: Both the raw dataset file and the pre-processed dataset file contain all the 12469 videos in our training and validation split (training + one half of the validation split as in ActivityNet Captions, which is based on ActivityNet 1.3). This includes 626 videos without box annotations.


Under the scripts directory, we include:

  • The preprocessing scripts to obtain the NP/object annotation files.
  •, The scripts that print the dataset stats.
  • The evaluation script for object grounding on GT captions. PyTorch is required. To evaluate your results, simply run:
python scripts/ -s YOUR_SUBMISSION_FILE.JSON

Please follow the example in data/anet_entities_skeleton.txt to format your submission file.


Please contact if you have any trouble running the code. Please cite the following paper if you use the dataset.

  title={Grounded Video Description},
  author={Zhou, Luowei and Kalantidis, Yannis and Chen, Xinlei and Corso, Jason J and Rohrbach, Marcus},
  journal={arXiv preprint arXiv:1812.06587},


This project is licensed under the license found in the LICENSE file in the root directory of this source tree.

The noun phrases in these annotations are based on ActivityNet Captions, which are linked to videos in ActivityNet 1.3

You can’t perform that action at this time.