CollabBench: Benchmarking and Unleashing Collaborative Ability of LLMs with Diverse Players via Proactive Engagement

ICML 2026

Hong Qian, Yuanhao Liu, Zihan Zhou, Zongbao Zhang, Hanjie Ge, Haotian Shi, Liang Dou, Xiangfeng Wang, Jingwen Yang*, and Aimin Zhou

East China Normal University
Shanghai Innovation Institute
Tencent Inc.

---

Overview

We propose CollabBench, a benchmark for systematically evaluating and training LLM-based agents to proactively collaborate with diverse players. CollabBench focuses on collaborative agent research, aiming to facilitate research on LLM-based agents in efficient and affective interactions.

Table of Contents

• Diverse Player Profiles Simulation
• Collaborative Agentic Training
• Evaluation
• Player Trajectory Demonstration
• Citation

---

1️⃣ Diverse Player Profiles Simulation

cd Anthropomorphic

This section focuses on modeling diverse player profiles from trajectory data.

📄 Details: Anthropomorphic

---

2️⃣ Collaborative Agentic Training

This section describe the training of the collaborative agents for the two multi-player game environments.

cd Training

🎮 CWAH-MultiPlayer

cd CWAH-MultiPlayer

📄 Details: CWAH-MultiPlayer

🎮 Cook-MultiPlayer

cd Cook-MultiPlayer

📄 Details: Cook-MultiPlayer

---

3️⃣ Evaluation

This section describes the trajectory data collection and affective LLM judge used in CollabBench for the two multi-player game environments.

cd Evaluation

Trajectory Data Collection

cd Running

🎮 CWAH-MultiPlayer

cd CWAH-MultiPlayer

📄 Details: CWAH-MultiPlayer

🎮 Cook-MultiPlayer

cd Cook-MultiPlayer

📄 Details: Cook-MultiPlayer

Affective LLM Judge

cd Judge

📄 Details: Evaluation

---

4️⃣ Player Trajectory Demonstration

We visualize representative trajectories for five typical player types (GIF format) to illustrate their collaboration behaviors.

❶ Efficient Collaboration Expert

❷ Hesitant Laggard

❸ Anxious Doubter

❹ Proactive Leader

❺ Independent Loner

---

💭 Citation

If you find this repository useful in your research, please cite:

@inproceedings{CollabBench2026,
  author = {Hong Qian and Yuanhao Liu and Zihan Zhou and Zongbao Zhang and Hanjie Ge and Haotian Shi and Liang Dou and Xiangfeng Wang and Jingwen Yang and Aimin Zhou},
  title = {CollabBench: Benchmarking and Unleashing the Collaborative Ability of LLMs with Diverse Players via Proactive Engagement},
  booktitle = {Proceedings of the 43rd International Conference on Machine Learning (ICML)},
  year = {2026},
  address = {Seoul, South Korea}
}

Reference:

Hong Qian, Yuanhao Liu, Zihan Zhou, Zongbao Zhang, Hanjie Ge, Haotian Shi, Liang Dou, Xiangfeng Wang, Jingwen Yang, and Aimin Zhou. CollabBench: Benchmarking and Unleashing the Collaborative Ability of LLMs with Diverse Players via Proactive Engagement. In Proceedings of the 43rd International Conference on Machine Learning (ICML), 2026.