IndexError when appending in cur_image_features by image_token_indices.

[yujiangpu20]

I tried my own video and IndexError triggered at line 277, arch.py: index xx is out of bounds for dimension 0 with size xx. This seems to be because the input video with variable length does not match the MAX_IMAGE_LENGTH (64). I modified the code as follows and it worked:

for _ in i:
    if cur_image_idx < image_features.shape[0]:
        cur_image_features.append(image_features[cur_image_idx])
        cur_image_idx += 1
    else:
        break

[jpthu17]

I suspect this might be a dataset-loading error. I didn't encounter this issue while training with the dataset I provided.

Do you use your own dataset for training? Could you share some examples of the datasets you utilized?

[yujiangpu20]

Not in the training phase, I just use the inference code you provided for video understanding, where different videos have different "max_frame".

[jpthu17]

I understand. What are the "max_frame" you use? Please allow me to try to reproduce this error.

[yujiangpu20]

I am using this video from Charades-STA for inference, and the max_frame is set to 902 (total number of frames) if I understood correctly.

[jpthu17]

I understand. I will address this issue promptly. Thank you for providing the information.

[jpthu17]

Sorry for replying so late.

I have fixed this bug. I also added a new parameter video_framerate (The number of frames retained per second in the video).

You can re-run our model with the updated command. Link

I tested this video and got the following results:

The video shows a man cleaning a window with a bottle of Windex and a cloth. He sprays the window with the Windex and then wipes it with the cloth to make it clean and clear.

[yujiangpu20]

Thanks for your kind reply. I tried the new version and it works great!