Benchmark methodology and reproduction: LLMSlim vs LLMLingua-2 and LongLLMLingua

[Thanatos9404]

This thread is the place to question, reproduce or extend the benchmarks in benchmarks/live/. Criticism of the method is welcome. If something is unfair to a baseline, I'll fix it and republish, whichever way the result moves.

What was measured

• Reader: claude-haiku-5-5 at effort low. It is the same reader, prompt template and scoring for every arm; only the context changes.
• Head-to-head (NaturalQuestions, 20 documents, "lost in the middle"): 300 cases. That is 100 distinct questions, each asked with the gold passage 1st, 10th and 20th. 50 questions come from the originally reported sample (test_a) and 50 from a fresh sample (test_b) drawn after the code was frozen and run once.
• Budget: every arm is held to about 25% of the context's cl100k tokens.

Arm	Accuracy	Mean context tokens	Median compression time per question
Full context	89.0%	2,889	n/a
LLMSlim 0.8 + reranker (cross-encoder/ms-marco-MiniLM-L6-v2, 22.7M)	78.7%	668	0.09 s (GPU)
LongLLMLingua (LLaMA-2-7B-chat, 4-bit NF4)	78.0%	630	9.3 s (GPU)
LLMLingua-2 (xlm-roberta-large-meetingbank, held to the same token count)	69.0%	680	2.3 s (GPU)
LLMSlim 0.8, lexical (no model)	68.7%	658	25 ms (CPU)

Paired differences, 95% bootstrap intervals:

Comparison	Difference	Interval, resampling cases	Interval, resampling whole questions
Reranker vs LLMLingua-2	+9.7	+3.7 to +15.7	about 0.0 to +19.3
Reranker vs LongLLMLingua	+0.7	−4.7 to +6.0	−7.7 to +9.3

The reranker and LongLLMLingua are level. LongLLMLingua won test_a by 6.0 points and LLMSlim won test_b by 7.3.

Known limitations

• There is only one reader, and only one dataset family for the competitor comparison.
• The baselines were run by me. LongLLMLingua was loaded in 4-bit to fit a 6 GB GPU. Its paper does not state precision.
• Without the reranker, LLMSlim is level with LLMLingua-2 (−0.3), not ahead.
• Full context still scores higher on NQ, HotpotQA and LongBench v2.
• Times are wall-clock medians, not FLOPs, and model loading is excluded.
• SWE-bench resolved counts are not yet backed by committed harness reports.

Reproduce

See the commands in the benchmark README: runs.sh for the LLMSlim arms and competitors.sh for LLMLingua-2 and LongLLMLingua. Per-question results are in results/*.jsonl.gz. Total API spend for every suite was $29.73 (results/spend-*.json).

How to contribute

• Report a methodology problem: describe it here, with the file and line if you can.
• Submit alternative results: post your config, the reader you used and the per-question output.
• Add a workload or a compressor: open a discussion first so we can agree on the budget and scoring.

The papers for the two baselines are LLMLingua-2 (arXiv 2403.12968) and LongLLMLingua (arXiv 2310.06839), with code at microsoft/LLMLingua. This benchmark is my own work; it is not affiliated with or endorsed by Microsoft.

– Yashvardhan (maintainer)