empty dataset and video&description to .npy&csv&json

[Tikquuss]

When I execute scripts/train.sh yc2 mart, I get this error.

len(train_dataset) gives me 0, yet all the steps prior to the training stage have been respected, and have been successfully completed.
How can I solve the problem please? @jayleicn
I work on google colab.

[jayleicn]

Maybe the script was not configured with the correct path to the feature directory?

[Tikquuss]

I haven't changed much to the configuration in place, except dur_file="/content/mart/densevid_eval/yc2_data/yc2_duration_frame.csv" in train.sh.
Here is my little notebook.
Thank you for your attention.

[jayleicn]

Can you try setting the video feature path to your feature path? And verify the features exist in your feature path.

[Tikquuss]

Thank you very much.

One last question please:

• I have my own dataset of videos and descriptions of these videos, how to switch from these videos to .npy (and everything that goes with it : duration_frame, json).
• I also don't understand the structure of the yc2_duration_frame.csv file, what do the three columns represent?

It would be really useful for the users to know how to switch from their video (and descriptions) to a dataset that can be used by this repo : since I'm setting up an application that will take a video and provide its description.

If I manage to have a clear and simple pipeline I will publish it.

[jayleicn]

Thanks @Tikquuss. It would great if you can help set up a simple pipeline for preparing features!

• For this work, we use the video features (.npy) extracted with https://github.com/LuoweiZhou/anet2016-cuhk-feature. But you can actually use any strong video features available in the community. For example, you can use i3d features + ResNet-152 features, I have code here to extract them https://github.com/jayleicn/TVRetrieval/tree/master/utils/video_feature. Another recent popular choice is the slowfast from FAIR: https://github.com/facebookresearch/SlowFast/tree/master/slowfast.

• For yc2_duration_frame.csv, the 1st column is the YouTube video id of the video, for example, in the first row of https://github.com/jayleicn/recurrent-transformer/blob/master/densevid_eval/yc2_data/yc2_duration_frame.csv, the value of the first column is -xbTvALWCIg, you can view the corresponding video on YouTube at https://www.youtube.com/watch?v=-xbTvALWCIg. The 2nd and 3rd column is the length of the video in terms of seconds and #frames.

Hope this helps!
Jie

[Tikquuss]

I think this repo can do the job.
Another thing I don't understand is the structure of the json files (val_yc2.json for example).

[jayleicn]

{
  "v_xHr8X2Wpmno": {
    "duration": 206.86,
    "timestamps": [
      [47, 60],
      [67, 89],
      [91, 98],
      ...
    ],
    "sentences": [
      "pick the ends off the verdalago",
      "combine lemon juice sumac garlic salt and oil in a bowl",
      "chop lettuce and place it in a bowl",
      ...
    ]
  },
  ...
}

Here is the first entry from the file. v_xHr8X2Wpmno is the video name, timestamps indicate the various segments in the video and sentences are the corresponding captions to these segments. For example, the sentence pick the ends off the verdalago describes 47-60seconds of the video v_xHr8X2Wpmno.