Inquiry regarding Table 2 Reproduction and Evaluation Prompts

[chilouis]

Hello,

I am currently working on reproducing the results presented in Table 2 of the DriveBench paper. I have two specific questions regarding the experimental setup:

1. Data Specification for Inference

Could you clarify whether the inference results for all models were obtained using drivebench-test.json or drivebench-test-final.json?

Additionally, I would appreciate it if you could explain the motivation behind adding the test-final version, particularly for handling single-image cases.

2. Evaluation Prompt Consistency

I noticed a potential discrepancy between the PERCEPTION_VQA_PROMPT in the repository and the version described in Figure 23 of the paper. Could you please verify this?

Since the paper mentions various prompt types (e.g., rubric-aware, context-aware), could you specify which evaluation prompt was used to generate the results in Table 2?

Thank you for your time and for sharing this valuable research.

[Daniel-xsy]

Hi @chilouis, thanks for your interest in our work. Regarding your question:

1. The model is evaluated in a single view if the question is only referring to a single-view camera. The motivation is the observation that VLMs are not good at multi-view reasoning now [Related Research]. Providing multi-view images to a single-view question might degrade the performance, which we try to avoid in this research.

2. Please use the version we provided in the codebase. We use different prompts for each question type (e.g., MCQ, open-ended) and each task (e.g., perception, prediction, etc). Figure 23 in the paper is a simplified demonstration.

Please let me know if you have further questions.

[chilouis]

Thank you for the kind and detailed response!

After taking a closer look, I noticed what appears to be a discrepancy between the paper and the codebase that I'd like to clarify.

In evaluate/prompts.py (line 48), the PERCEPTION_VQA_PROMPT is formatted with three fields:

Here is the question: "{QUESTION}"
Here is the predicted answer: "{PRED}"
Here is the correct answer: "{GT}"

However, in Figure 23 of the paper (GPT evaluation prompt for Open-Ended Questions), only two fields are shown:

Here is the predicted answer: PRED
Here is the correct answer: GT

The {QUESTION} field is not present in the figure. Since the six scoring criteria (Action Alignment, Motion Precision, etc.) are identical in both, I assume they refer to the same prompt — but the inputs differ.

Could you confirm which version was actually used to produce the reported results? In other words, was the question included when prompting the GPT evaluator, or not? This matters for reproducibility, since including the question can meaningfully change the evaluator's judgment.

Thanks again for your help!

[Daniel-xsy]

Thanks for bringing this up. We always use {QUESTION} in our evaluation. The paper template should be a typo. Sorry for the confusion.