Loong: Benchmarking Long-Context LLMs with Extended Multi-Doc QA

📃 • 🔔

👀Overview

This repository contains code for our paper Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QA. We propose a novel long-context benchmark, 🐉 Loong, aligning with realistic scenarios through extended multi-document question answering (QA). Loong typically consists of 11 documents per test instance on average, spanning three real-world scenarios in English and Chinese: (1) Financial Reports, (2) Legal Cases, and (3) Academic Papers. Meanwhile, Loong introduces new evaluation tasks from the perspectives of Spotlight Locating, Comparison, Clustering, and Chain of Reasoning, to facilitate a more realistic and comprehensive evaluation of long-context understanding. Furthermore, Loong features inputs of varying lengths (e.g., 10K-50K, 50K-100K, 100K-200K, beyond 200K) and evaluation tasks of diverse difficulty, enabling fine-grained assessment of LLMs across different context lengths and task complexities.

Please find more details of this work in our paper.

Showcase of four evaluation tasks in Loong (<di>...</di> marks the content of the i-th document). (a) Spotlight Locating: Locate the evidence. (b) Comparison: Locate and compare the evidence. (c) Clustering: Locate and cluster the evidence into groups. (d) Chain of Reasoning: Locate and reasoning along a logical chain.

📰News

[2024-07-30] 🤖The performance of phi-3, llama-3.1-8B, gpt-4o-mini on Loong are updated.

[2024-07-03] 🔥The code and benchmark are releasing. If you encounter any issues, please feel free to contact us.

[2024-06-25] 👨‍💻The code is currently being refined, and we plan to release the evaluation code and benchmark within the next one or two weeks. If you encounter any issues, please feel free to contact me at wangminzheng2023@ia.ac.cn.

🏆Leaderboard

Models	Claimed Length	Spotlight Locating		Comparison		Clustering		Chain of Reason		Overall
Gemini-1.5-pro	1000K	75.02	0.56	49.94	0.27	44.10	0.09	64.97	0.37	55.37	0.27
GPT-4o	128K	73.95	0.62	50.50	0.28	44.29	0.09	57.95	0.28	53.47	0.26
Claude3.5-Sonnet	200K	58.45	0.49	54.21	0.35	45.77	0.07	43.92	0.25	48.85	0.23
Claude3-Haiku	200K	68.68	0.59	42.10	0.21	35.04	0.02	47.59	0.17	44.88	0.19
Qwen2-72B-Instruct	128K	54.17	0.36	42.38	0.20	36.71	0.04	47.76	0.18	43.29	0.15
GPT-4o-mini	128K	53.12	0.41	44.27	0.20	32.58	0.04	52.34	0.23	42.95	0.18
GLM4-9B-Chat	1000K	57.35	0.47	40.38	0.20	28.52	0.02	39.94	0.16	38.31	0.16
Kimi-Chat	200K	60.98	0.50	34.74	0.13	28.76	0.04	38.52	0.15	37.49	0.16
Llama-3.1-8B-Instruct	128K	59.96	0.46	35.73	0.18	27.83	0.01	35.59	0.14	36.31	0.14
Phi-3-small	128K	29.23	0.10	20.12	0.06	17.53	0.00	14.36	0.01	19.03	0.03
Phi-3-mini	128K	25.65	0.15	13.34	0.04	12.00	0.00	12.61	0.01	14.54	0.04

Overall results on four evaluation tasks. For each task, the indicator on the left represents the Avg Scores(0～100), while the right one represents the Perfect Rate(0~1).

Model	Claimed Length	Spotlight Locating		Comparison		Clustering		Chain of Reasoning		Overall
Set1 (10K-50K)
GPT-4o	128K	85.67	0.81	64.27	0.33	57.01	0.24	81.58	0.55	70.40	0.44
Claude3.5-Sonnet	200K	60.85	0.55	69.07	0.47	58.63	0.13	68.57	0.50	63.69	0.37
Gemini-1.5-pro	1000K	75.00	0.60	54.88	0.28	56.15	0.23	70.64	0.37	63.36	0.34
GPT-4o-mini	128K	62.49	0.56	65.48	0.40	45.81	0.12	79.85	0.55	62.42	0.36
Qwen2-72B-Instruct	200K	68.49	0.55	60.60	0.37	47.08	0.08	70.39	0.36	60.11	0.29
Claude3-Haiku	200K	60.94	0.55	59.97	0.40	45.53	0.04	66.85	0.34	57.14	0.28
Kimi-Chat	200K	81.11	0.74	46.70	0.20	47.84	0.07	53.77	0.17	55.02	0.24
GLM4-9B-Chat	1000K	63.11	0.53	54.10	0.27	39.50	0.08	56.32	0.28	51.43	0.25
Llama-3.1-8B-Instruct	128K	67.91	0.57	41.62	0.20	36.55	0.04	54.74	0.34	48.10	0.24
Phi-3-mini	128K	46.13	0.30	22.18	0.05	19.30	0.02	20.44	0.03	24.58	0.07
Phi-3-small	128K	35.00	0.15	26.83	0.12	17.01	0.00	15.87	0.00	21.44	0.05
Set2 (50K-100K)
GPT-4o	128K	86.76	0.72	59.81	0.40	47.83	0.11	62.09	0.34	58.38	0.29
Gemini-1.5-pro	1000K	76.50	0.57	54.51	0.34	44.58	0.09	64.87	0.34	55.56	0.26
Claude3.5-Sonnet	200K	63.83	0.53	58.90	0.39	50.96	0.10	46.09	0.26	52.73	0.24
GPT-4o-mini	128K	63.54	0.46	51.48	0.26	36.56	0.04	56.51	0.25	47.74	0.19
Qwen2-72B-Instruct	128K	64.53	0.43	42.60	0.21	38.52	0.05	51.18	0.20	45.71	0.17
Claude3-Haiku	200K	73.71	0.66	41.90	0.22	36.18	0.02	50.20	0.15	45.45	0.17
Kimi-Chat	200K	72.82	0.52	46.77	0.21	33.46	0.06	40.51	0.15	42.40	0.16
Llama-3.1-8B-Instruct	128K	72.79	0.59	44.51	0.27	32.98	0.01	40.53	0.15	41.98	0.16
GLM4-9B-Chat	1000K	65.04	0.54	41.80	0.23	30.72	0.02	42.34	0.17	40.19	0.17
Phi-3-small	128K	34.17	0.16	22.08	0.08	20.51	0.01	16.20	0.01	21.40	0.04
Phi-3-mini	128K	44.71	0.29	22.81	0.09	16.37	0.00	15.39	0.01	20.84	0.05
Set3 (100K-200K)
Gemini-1.5-pro	1000K	81.25	0.56	44.66	0.20	39.90	0.05	58.38	0.36	52.05	0.24
GPT-4o	128K	74.84	0.65	42.40	0.21	38.70	0.04	45.06	0.09	46.95	0.19
Claude3.5-Sonnet	200K	65.36	0.56	50.32	0.34	37.79	0.03	25.95	0.11	42.06	0.19
Claude3-Haiku	200K	77.81	0.67	37.07	0.17	30.94	0.01	36.87	0.12	41.41	0.18
GPT-4o-mini	128K	58.27	0.49	33.46	0.09	27.33	0.01	35.67	0.04	35.63	0.11
Qwen2-72B-Instruct	128K	46.99	0.27	37.06	0.13	31.50	0.02	35.01	0.07	35.94	0.09
GLM4-9B-Chat	1000K	69.19	0.56	37.99	0.18	26.63	0.01	32.30	0.09	37.36	0.16
Kimi-Chat	200K	62.13	0.54	24.20	0.05	21.98	0.01	31.02	0.14	31.37	0.14
Llama-3.1-8B-Instruct	128K	60.05	0.46	25.86	0.11	21.96	0.00	19.14	0.02	28.41	0.10
Phi-3-small	128K	25.12	0.06	15.26	0.01	16.80	0.00	12.75	0.01	16.94	0.01
Phi-3-mini	128K	7.40	0.03	1.97	0.00	6.07	0.00	7.38	0.01	5.79	0.01
Set4 (200K-250K)
Gemini-1.5-pro	1000K	62.23	0.49	43.08	0.20	36.48	0.00	68.51	0.49	50.70	0.25
Claude3-Haiku	200K	53.26	0.40	27.00	0.03	25.36	0.00	28.11	0.05	32.15	0.10
GPT-4o	128K	36.79	0.19	23.97	0.08	30.40	0.00	32.89	0.07	31.11	0.07
Claude3.5-Sonnet	200K	36.91	0.24	28.82	0.05	28.68	0.00	28.77	0.08	30.51	0.08
Qwen2-72B-Instruct	128K	33.18	0.16	26.59	0.08	29.84	0.01	25.81	0.04	28.92	0.06
Llama-3.1-8B-Instruct	128K	31.72	0.13	27.27	0.10	15.17	0.00	22.89	0.02	22.51	0.05
GPT-4o-mini	128K	20.66	0.09	19.18	0.03	16.03	0.00	27.81	0.00	20.41	0.02
GLM4-9B-Chat	1000K	15.67	0.12	21.33	0.05	12.35	0.00	21.04	0.05	16.84	0.05
Kimi-Chat	200K	20.17	0.12	9.17	0.00	5.65	0.00	22.61	0.11	13.50	0.05
Phi-3-small	128K	22.36	0.02	16.43	0.05	11.50	0.00	10.35	0.00	14.27	0.01
Phi-3-mini	128K	5.21	0.00	2.20	0.00	3.45	0.00	2.58	0.00	3.38	0.00

The performance of LLMs on four evaluation tasks with different length sets. For each task, the indicator on the left represents the Avg Scores(0～100), while the right one represents the Perfect Rate(0~1).

• Following previous work, we prompt GPT-4 as a judge to evaluate the model's output based on the golden answer and the question's requirements from three aspects: Accuracy, Hallucinations, and Completeness, scoring from 0 to 100. For a detailed prompt, please refer to our paper.
• We design two indicators: (1) Avg Scores: the average value of scores given by GPT-4 for all questions; (2) Perfect Rate: the proportion of cases scoring 100 out of the total cases. The latter is a more stringent evaluation metric compared to the former.
• We set temperature = 0 to eliminate randomness and keep other hyper-parameters default. For API-Based LLMs, we directly utilize the official API for testing. Since the Kimi-Chat-200k currently does not provide an interface, we manually input content on the web. As for open-source models, we conduct experiments on a server with 8 $\times$ A100 80GB.

🔧Evaluate long-context LLMs

Step1 Download Loong benchmark

git clone https://github.com/MozerWang/Loong.git
cd Loong

Step2 Create a conda environment and Install other dependencies.

conda create --name loong python=3.9 -y
conda activate loong
pip install -r requirements.txt

Step3 Preparing the Model

1. (Must) Set up your OPENAI key in config/models/gpt4.yaml

api_key: "Your OPENAI key"

2. If you are using API-based LLM

# Firstly, Set up your key in config/models/*.yaml
api_key: "Your API key"

3. If you are using Open-sourced LLM

# We recommend using vLLM. And we use HTTP server that implements OpenAI’s Completions and Chat API.
# We have provided using example for Qwen2 and GLM4. See details in Loong/src/vllm_eample.sh
cd src
sh vllm_example.sh

Step4 Evaluate

cd src
sh run.sh

Things To Know

• We provide a complete evaluation process:
  step1_load_data.py # Data loading
  step2_model_generate.py # Model generation
  step3_model_evaluate.py # GPT-4 evaluation
  step4_cal_metric.py # Result statistics

• For step2_model_generate.py, you can design the model generation part yourself, modifying it to use your own model's inference method. Just make sure the input and output interfaces in src/utils/generate.py remain consistent:

# Input
generate(prompts, config, output_path, process_num, tag)

# Output
result = prompt.copy() # for prompt in prompts
result[tag] = response_content # Your LLM's response
with open(output_path, 'a', encoding='utf-8') as fw:
    fw.write(json.dumps(result, ensure_ascii=False) + '\n')

• In data/loong.jsonl, the level key means task:
  level1 # Spotlight Locating
  level2 # Comparison
  level3 # Clustering
  level4 # Chain of Reasoning

Citation

@article{wang2024loong,
  title={Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QA},
  author={Minzheng Wang, Longze Chen, Cheng Fu, Shengyi Liao, Xinghua Zhang, Bingli Wu, Haiyang Yu, Nan Xu, Lei Zhang, Run Luo, Yunshui Li, Min Yang, Fei Huang, Yongbin Li},
  year={2024}
  journal={arXiv preprint arXiv:2406.17419},
}