Did the Evaluation of DriveLM Use the Open-Source Version?

[RiverCrosser-White]

Hello, thank you for your great work! I’m very interested in your benchmark. I have a quick question regarding the evaluation of DriveLM in your experiments: did you use the open-source implementation of DriveLM based on LLaMA-Adapter for your evaluation?

[ldkong1205]

Hi @RiverCrosser-White, thanks for your interest in this work!

Your understanding is correct. The DriveLM-Agent in our benchmark is based on LLaMA-Adapter-V2, the same configuration in the original DriveLM paper.

[Daniel-xsy]

Hi @RiverCrosser-White, thanks for your interest, and thanks @ldkong1205 for the clarification.

To add a few more information: since our DriveBench is developed from the train split of DriveLM, the original checkpoint for the LLaMA-Adapter-V2 model from DriveLM might not be applicable due to information leakage. Therefore, we re-train the model with the same setups but exclude the data in DriveBench. Hope it can help!

[RiverCrosser-White]

Thank you for your timely answers. It really helped me a lot. Thank you very much

[Melonehhh]

Hi @Daniel-xsy , could you share the model from DriveLM which was trained without the data in DriveBench? It would help a lot! :)

[Daniel-xsy]

Hi @Melonehhh, sure, I can upload the trained model this weekend. Sorry for the delay as I'm blocked by other tasks.

[RiverCrosser-White]

Hi @Daniel-xsy , thank you for your previous response and for confirming that the model will be shared.
Just checking in to see if there’s an expected release location (e.g., Hugging Face, Google Drive, etc.) or timeline?
Looking forward to the release—thanks again for your great work!

[Daniel-xsy]

Hi @RiverCrosser-White, sorry for the late response. I uploaded the checkpoitn on HuggingFace, you can follow the instruction in DriveLM to do the inference.

[RiverCrosser-White]

Thank you for your reply. It helped me a lot

[RDXSun]

Hi @RiverCrosser-White, sorry for the late response. I uploaded the checkpoitn on HuggingFace, you can follow the instruction in DriveLM to do the inference.

I'm quite new to this, and I noticed the uploaded files look a bit different from typical HuggingFace model formats (e.g., there's no config file). Could you kindly let me know what code I should use to load the weights for inference? I checked the DriveLM instructions but still feel a bit unsure how to proceed with this specific checkpoint.

Thanks in advance for your help!

[Daniel-xsy]

Hi @RDXSun, thanks for your interest.

TL;DR: directly download the model and specify it in the inference code.

Your observation is correct. The official DriveLM code uses LLaMA-Adapter-V2 as the DriveLM-Agent. Thus, the model weight loading and inference process follow the command here since our uploaded model is the direct saved checkpoint.