questions about mosei dataset

[TianhaoFu]

Hi, thanks for your code.

When I use your dataloader loading mosei affect dataset[I used the dataset provided in your repo]. I found that the batch video data shape is [batchsize, 50, 35]. batch audio data shape is [batchsize, 50 74]. What does 50 and 35 in video date shape mean? BTW, the regular video batch data format should be [batchsize, channel, clip_length, crop_size, crop_size]. It seems that [batchsize, 50, 35] doesn't follow this format. What is the reason for this?

Thanks!

[Vanvan2017]

Thanks a lot for the question~ the default format of the video and audio data is [batch_size, sequence_length, feature_dim], because the feature here are preprocessed from the videos (using OpenFace or Facet), so the format didn't follow the default video data format~ And seems you are more familiar with the video data format, and the video data is here, you can deal with the video directly if you want to try some video models by yourself ~ It really rare for people to deal with it that way, so it may be a good direction~

[TianhaoFu]

Thanks a lot for the question~ the default format of the video and audio data is [batch_size, sequence_length, feature_dim], because the feature here are preprocessed from the videos (using OpenFace or Facet), so the format didn't follow the default video data format~ And seems you are more familiar with the video data data format, and the video data is here, you can deal with the video directly if you want to try some video models by yourself ~ It really rare for people to deal with it that way, so it may be a good direction~

It doesn't look like this dataset contains a list of mosei files, do you know where it is please, thanks! @Vanvan2017

[Vanvan2017]

Hi, @TianhaoFu, did you mean the videos? It should be in the zip file here. You can download and unzip it and then use a video processing tool to get the raw frames~ As for the labels, you can check the labels.

[TianhaoFu]

Thanks for your reply. I mean I want a file contains the video files name. such as

column1
7155.mp4
7156.mp4
7157.mp4
...

do you know where can I such file? thanks!

By the way, It seems that the labels file you provided is .csd file, could tell me how to open it?
Thanks very much!

@Vanvan2017

[Vanvan2017]

Hi, @TianhaoFu, I think you can open it with h5py, and the names of the files are in it, you can add .mp4 after the names~ For more details of the .csd file, you can refer to the multimodal SDK, but in a word, .csd just means the .h5 file~ So h5py is okay for it.