[Benchmarks] Add MMVU video dataset support and clean up deprecated datasets

[Isotr0py]

Purpose

• Add https://huggingface.co/datasets/yale-nlp/MMVU video dataset support for out-of-box video benchmarking
• Remove benchmarks/benchmark_dataset.py as benchmarks/benchmark_serving.py etc have been deprecated by vllm bench.

Test Plan

Test Result

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

[gemini-code-assist]

Code Review

This pull request adds support for the MMVU video dataset for benchmarking and removes the deprecated benchmark_dataset.py file. The changes are well-structured, but I've identified an issue in the new MMVUDataset implementation where the sample method is missing the no_oversample parameter. This would cause the --no-oversample CLI flag to be ignored for this dataset. I've provided a code suggestion to fix this inconsistency.

[DarkLight1337]

Can you post the result of benchmarking some models on this dataset?

[tjtanaa]

@Isotr0py amazing feature. Thank you for this PR. Need this.

[david6666666]

Nice one, We really need a video benchmark dataset.

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @Isotr0py.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[Isotr0py]

Can you post the result of benchmarking some models on this dataset?

Test suite: T4 GPU x 2

vllm serve Qwen/Qwen2-VL-2B-Instruct -tp 2 --max-model-len 32768 --enforce-eager --max-num-seqs 5

Benchmark commands:

vllm bench serve \
  --backend openai-chat \
  --endpoint-type openai-chat \
  --model Qwen/Qwen2-VL-2B-Instruct \
  --endpoint /v1/chat/completions \
  --dataset-name hf \
  --dataset-path yale-nlp/MMVU \
  --hf-split validation \
  --num-prompts 10

Results:

Initial test run completed. Starting main benchmark run...
Traffic request rate: inf
Burstiness factor: 1.0 (Poisson process)
Maximum request concurrency: None
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [02:10<00:00, 13.00s/it]
============ Serving Benchmark Result ============
Successful requests:                     10        
Benchmark duration (s):                  130.04    
Total input tokens:                      578       
Total generated tokens:                  605       
Request throughput (req/s):              0.08      
Output token throughput (tok/s):         4.65      
Total Token throughput (tok/s):          9.10      
---------------Time to First Token----------------
Mean TTFT (ms):                          64757.39  
Median TTFT (ms):                        58903.87  
P99 TTFT (ms):                           112885.96 
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          837.82    
Median TPOT (ms):                        771.18    
P99 TPOT (ms):                           1754.11   
---------------Inter-token Latency----------------
Mean ITL (ms):                           632.08    
Median ITL (ms):                         316.43    
P99 ITL (ms):                            3830.31   
==================================================