Release of the code for evaluating on long video benchmark

[anhdao69]

Thank you for this interesting work. Could you share when you plan to release the code for evaluation on long video benchmarks? It would be very helpful for us to reproduce and further build upon your great work.

Thanks!

[anonymous626]

hey @Johnx69 thanks for your interest. I think current release is pretty close to the benchmark evaluation. We also provide the link in README to download the database we built for benchmarks. If it still blocks you, I will try to clean that code and release it next week (a few days after ICLR submission DDL I think).

[anhdao69]

@anonymous626 Thanks a lot! I’m really looking forward to it—I’d love to reproduce the results.

[anhdao69]

@anonymous626 One more thing, what is the cost of API Keys when you evaluated on these dataset?

[anonymous626]

Sorry @Johnx69 for the late reply. I've refactored related code and upload full database for reproducing performance. Check out REPRODUCE.md if you get any questions.

[echo-bjn]

Sorry @Johnx69 for the late reply. I've refactored related code and upload full database for reproducing performance. Check out REPRODUCE.md if you get any questions.很抱歉回复晚了。我已经重构了相关代码并上传了完整的数据库以重现性能。如果您有任何问题，请查看 REPRODUCE.md。

I can't seem to find the file “video_info.meta.jsonl”. Can you provide this file? Thanks！！！

[anonymous626]

@echo-bjn You can find it in the original LVBench dataset here.

[echo-bjn]

@echo-bjn You can find it in the original LVBench dataset here.

Thanks！

[anonymous626]

@anonymous626 One more thing, what is the cost of API Keys when you evaluated on these dataset?

hey @Johnx69 I recompute the cost today. According to current O3 price, it takes $0.1 to $0.3 per questions depending on the cache utilization. According to OpenAI, using this response API will give better cache hit rate.

You can also try to remove or compress historical tool use message (e.g. direct ClipSearch will return massive tokens) to reduce the token number.

[anonymous626]

I am closing this issue. Feel free to reopen it if there are more issues then I will get notified.