How to run inference for a new video?

[yonatanbitton]

Hello, i'm trying to run inference. I've installed the package. I've tried the code here.

I have a new directory of videos, called "videos".

(venv) (base) yonatan:~/VBench$ ls videos/
Iron_Man.mp4  birthday.mp4  lavie_human_action_full_info.json  skateboarding_dog.mp4

When I try to run

vbench evaluate --videos_path "videos" --dimension "human_action"

I receive an error indicating that no data is found.
When I try through the code, I also get a long list of missing videos:

>>> from vbench import VBench
>>> my_VBench = VBench('cuda', 'VBench_full_info.json', 'videos')
>>> my_VBench.evaluate(videos_path='videos', name='my_test', dimension_list=['human_action'])
WARNING!!! This required video is not found! Missing benchmark videos can lead to unfair evaluation result. The missing video is: A person is riding a bike-0.mp4
...
...

I guess that your code only supports your existing evaluated videos? And the instructions do not yet support inference on new videos?

The question is how can I run the evaluation for a new video + description. Thank you very much 🙏

[ziqihuangg]

Thanks for the question and suggestion! We will provide instructions for evaluating a new video soon.

Currently, we enforce evaluation on the standard VBench prompt lists to ensure fair comparison among different video generation models. That's also why we give warnings when a required video is not found. This is done via defining the set of prompts in: https://github.com/Vchitect/VBench/blob/master/VBench_full_info.json
A quick fix is to define your own VBench_full_info.json file: replacing the prompts in "prompt_en" by your video name.

[yonatanbitton]

Thank you very much for your great work! looking forward for instructions for evaluating a new video :)

[moclimb]

Thanks for the question and suggestion! We will provide instructions for evaluating a new video soon.

Currently, we enforce evaluation on the standard VBench prompt lists to ensure fair comparison among different video generation models. That's also why we give warnings when a required video is not found. This is done via defining the set of prompts in: https://github.com/Vchitect/VBench/blob/master/VBench_full_info.json A quick fix is to define your own VBench_full_info.json file: replacing the prompts in "prompt_en" by your video name.

If I use my video name replace the prompts in "prompt_en" , but I can't supply the "auxiliary_info", will this have any impact?
Looking forward for you replay.
Thanks!!

[yinanhe]

Thanks for the question and suggestion! We will provide instructions for evaluating a new video soon.
Currently, we enforce evaluation on the standard VBench prompt lists to ensure fair comparison among different video generation models. That's also why we give warnings when a required video is not found. This is done via defining the set of prompts in: https://github.com/Vchitect/VBench/blob/master/VBench_full_info.json A quick fix is to define your own VBench_full_info.json file: replacing the prompts in "prompt_en" by your video name.

If I use my video name replace the prompts in "prompt_en" , but I can't supply the "auxiliary_info", will this have any impact? Looking forward for you replay. Thanks!!

@moclimb Thank you for your interest in our work! In ['subject_consistency', 'background_consistency', 'temporal_flickering', 'motion_smoothness', 'dynamic_degree', 'aesthetic_quality', 'imaging_quality', 'temporal_style', 'overall_consistency'], you don't need to fill in any auxiliary_info, you just need to fill in the dimension corresponding to your prompt

In ['object_class', 'multiple_objects', 'human_action', 'color', 'spatial_relationship', 'scene', 'appearance_style'], you need to fill in auxiliary_info in accordance with the corresponding rules. You must note that the object class and multiple objects in the tuple must be an object of COCO. The human action must be one action of Kinetics-400.

[ziqihuangg]

Hello, i'm trying to run inference. I've installed the package. I've tried the code here.

I have a new directory of videos, called "videos".

(venv) (base) yonatan:~/VBench$ ls videos/
Iron_Man.mp4  birthday.mp4  lavie_human_action_full_info.json  skateboarding_dog.mp4

When I try to run

vbench evaluate --videos_path "videos" --dimension "human_action"

I receive an error indicating that no data is found. When I try through the code, I also get a long list of missing videos:

>>> from vbench import VBench
>>> my_VBench = VBench('cuda', 'VBench_full_info.json', 'videos')
>>> my_VBench.evaluate(videos_path='videos', name='my_test', dimension_list=['human_action'])
WARNING!!! This required video is not found! Missing benchmark videos can lead to unfair evaluation result. The missing video is: A person is riding a bike-0.mp4
...
...

I guess that your code only supports your existing evaluated videos? And the instructions do not yet support inference on new videos?

The question is how can I run the evaluation for a new video + description. Thank you very much 🙏

Hi Yonatan, we now support evaluating user-customised videos: https://github.com/Vchitect/VBench?tab=readme-ov-file#new-evaluate-your-own-videos. You can do this by simply turning the custom_input flag on.

This new feature now supports those "video quality" dimensions, like temporal consistency, and frame-wise quality. For dimensions that heavily rely on the text prompt, sometimes VBench utilize pre-defined metadata, like the color labels for objects associated with our pre-defined prompt list. Therefore, evaluating videos from user-customised prompts is not yet supported for these prompt-dependent dimensions, but we are working on it. We will gradually enable customised evaluation of more "video-text alignment" dimensions.

[ziqihuangg]

I'm closing the issue, feel free to re-open it if you still encountered issues when evaluating new videos.