LLM4IR-Survey

This is the collection of papers related to large language models for information retrieval. These papers are organized according to our survey paper Large Language Models for Information Retrieval: A Survey.

Feel free to contact us if you find a mistake or have any advice. Email: yutaozhu94@gmail.com and dou@ruc.edu.cn.

🌟 Citation

Please kindly cite our paper if helps your research:

@article{LLM4IRSurvey,
    author={Yutao Zhu and
            Huaying Yuan and
            Shuting Wang and
            Jiongnan Liu and
            Wenhan Liu and
            Chenlong Deng and
            Haonan Chen and
            Zhicheng Dou and
            Ji-Rong Wen},
    title={Large Language Models for Information Retrieval: A Survey},
    journal={CoRR},
    volume={abs/2308.07107},
    year={2023},
    url={https://arxiv.org/abs/2308.07107},
    eprinttype={arXiv},
    eprint={2306.07401}
}

🚀 Update Log

• Version 2 [2024-01-19]
  • We added a new section to introduce search agents, which represent an innovative approach to integrating LLMs with IR systems.
  • Rewriter: We added recent works on LLM-based query rewriting, most of which focus on conversational search.
  • Retriever: We added the latest techniques that leverage LLMs to expand the training corpus for retrievers or to enhance retrievers' architectures.
  • Reranker: We added recent LLM-based ranking works to each of the three part: Utilizing LLMs as Supervised Rerankers, Utilizing LLMs as Unsupervised Rerankers, and Utilizing LLMs for Training Data Augmentation.
  • Reader: We added the latest studies in LLM-enhanced reader area, including a section introducing the reference compression technique, a section discussing the applications of LLM-enhanced readers, and a section analyzing the characteristics of LLM-enhanced readers.
  • Future Direction: We added a section about search agents and a section discussing the bias caused by leveraging LLMs into IR systems.

📋 Table of Content

• Query Rewriter
  • Prompting Methods
  • Fine-tuning Methods
  • Knowledge Distillation Methods
• Retriever
  • Leveraging LLMs to Generate Search Data
  • Employing LLMs to Enhance Model Architecture
• Re-ranker
  • Utilizing LLMs as Supervised Rerankers
  • Utilizing LLMs as Unsupervised Rerankers
  • Utilizing LLMs for Training Data Augmentation
• Reader
  • Passive Reader
  • Active Reader
  • Compressor
  • Analysis
  • Applications
• Search Agent
  • Static Agent
  • Dynamic Agent
• Other Resources

📄 Paper List

Query Rewriter

Prompting Methods

1. Query2doc: Query Expansion with Large Language Models, Wang et al., arXiv 2023. [Paper]
2. Generative and Pseudo-Relevant Feedback for Sparse, Dense and Learned Sparse Retrieval, Mackie et al., arXiv 2023. [Paper]
3. Generative Relevance Feedback with Large Language Models, Mackie et al., SIGIR 2023 (short paper). [Paper]
4. GRM: Generative Relevance Modeling Using Relevance-Aware Sample Estimation for Document Retrieval, Mackie et al., arXiv 2023. [Paper]
5. Large Language Models Know Your Contextual Search Intent: A Prompting Framework for Conversational Search, Mao et al., arXiv 2023. [Paper]
6. Precise Zero-Shot Dense Retrieval without Relevance Labels, Gao et al., ACL 2023. [Paper]
7. Query Expansion by Prompting Large Language Models, Jagerman et al., arXiv 2023. [Paper]
8. Large Language Models are Strong Zero-Shot Retriever, Shen et al., arXiv 2023. [Paper]
9. Enhancing Conversational Search: Large Language Model-Aided Informative Query Rewriting, Ye et al., EMNLP 2023 (Findings). [Paper]
10. Can generative llms create query variants for test collections? an exploratory study, M. Alaofi et al., SIGIR 2023 (short paper). [Paper]

Fine-tuning Methods

1. QUILL: Query Intent with Large Language Models using Retrieval Augmentation and Multi-stage Distillation, Srinivasan et al., EMNLP 2022 (Industry). [Paper] (This paper explore fine-tuning methods in baseline experiments.)

Knowledge Distillation Methods

1. QUILL: Query Intent with Large Language Models using Retrieval Augmentation and Multi-stage Distillation, Srinivasan et al., EMNLP 2022 (Industry). [Paper]
2. Knowledge Refinement via Interaction Between Search Engines and Large Language Models, Feng et al., arXiv 2023. [Paper]
3. Query Rewriting for Retrieval-Augmented Large Language Models, Ma et al., arXiv 2023. [Paper]

Retriever

Leveraging LLMs to Generate Search Data

1. InPars: Data Augmentation for Information Retrieval using Large Language Models, Bonifacio et al., arXiv 2022. [Paper]
2. Pre-training with Large Language Model-based Document Expansion for Dense Passage Retrieval, Ma et al., arXiv 2023. [Paper]
3. InPars-v2: Large Language Models as Efficient Dataset Generators for Information Retrieval, Jeronymo et al., arXiv 2023. [Paper]
4. Promptagator: Few-shot Dense Retrieval From 8 Examples, Dai et al., ICLR 2023. [Paper]
5. AugTriever: Unsupervised Dense Retrieval by Scalable Data Augmentation, Meng et al., arXiv 2023. [Paper]
6. UDAPDR: Unsupervised Domain Adaptation via LLM Prompting and Distillation of Rerankers, Saad-Falco et al., arXiv 2023. [Paper]
7. Soft Prompt Tuning for Augmenting Dense Retrieval with Large Language Models, Peng et al., arXiv 2023. [Paper]
8. CONVERSER: Few-shot Conversational Dense Retrieval with Synthetic Data Generation, Huang et al., ACL 2023. [Paper]
9. Leveraging LLMs for Synthesizing Training Data Across Many Languages in Multilingual Dense Retrieval, Thakur et al., arXiv 2023. [Paper]
10. Questions Are All You Need to Train a Dense Passage Retriever, Sachan et al., ACL 2023. [Paper]
11. Beyond Factuality: A Comprehensive Evaluation of Large Language Models as Knowledge Generators, Chen et al., EMNLP 2023. [Paper]

Employing LLMs to Enhance Model Architecture

1. Text and Code Embeddings by Contrastive Pre-Training, Neelakantan et al., arXiv 2022. [Paper]
2. Fine-Tuning LLaMA for Multi-Stage Text Retrieval, Ma et al., arXiv 2023. [Paper]
3. Large Dual Encoders Are Generalizable Retrievers, Ni et al., EMNLP 2022. [Paper]
4. Task-aware Retrieval with Instructions, Asai et al., ACL 2023 (Findings). [Paper]
5. Transformer memory as a differentiable search index, Tay et al., NeurIPS 2022. [Paper]
6. Large Language Models are Built-in Autoregressive Search Engines, Ziems et al., ACL 2023 (Findings). [Paper]

Reranker

Utilizing LLMs as Supervised Rerankers

1. Multi-Stage Document Ranking with BERT, Nogueira et al., arXiv 2019. [Paper]
2. Document Ranking with a Pretrained Sequence-to-Sequence Model, Nogueira et al., EMNLP 2020 (Findings). [Paper]
3. Text-to-Text Multi-view Learning for Passage Re-ranking, Ju et al., SIGIR 2021 (Short Paper). [Paper]
4. The Expando-Mono-Duo Design Pattern for Text Ranking with Pretrained Sequence-to-Sequence Models, Pradeep et al., arXiv 2021. [Paper]
5. RankT5: Fine-Tuning T5 for Text Ranking with Ranking Losses, Zhuang et al., SIGIR 2023 (Short Paper). [Paper]
6. Fine-Tuning LLaMA for Multi-Stage Text Retrieval, Ma et al., arXiv 2023. [Paper]
7. RankingGPT: Empowering Large Language Models in Text Ranking with Progressive Enhancement, Zhang et al., arXiv 2023. [Paper]
8. Rank-without-GPT: Building GPT-Independent Listwise Rerankers on Open-Source Large Language Models, Zhang et al., arXiv 2023. [Paper]

Utilizing LLMs as Unsupervised Rerankers

1. Holistic Evaluation of Language Models, Liang et al., arXiv 2022. [Paper]
2. Improving Passage Retrieval with Zero-Shot Question Generation, Sachan et al., EMNLP 2022. [Paper]
3. Discrete Prompt Optimization via Constrained Generation for Zero-shot Re-ranker, Cho et al., ACL 2023 (Findings). [Paper]
4. Open-source Large Language Models are Strong Zero-shot Query Likelihood Models for Document Ranking, Zhuang et al., EMNLP 2023 (Findings). [Paper]
5. PaRaDe: Passage Ranking using Demonstrations with Large Language Models, Drozdov et al., EMNLP 2023 (Findings). [Paper]
6. Beyond Yes and No: Improving Zero-Shot LLM Rankers via Scoring Fine-Grained Relevance Labels, Zhuang et al., arXiv 2023. [Paper]
7. Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agent, Sun et al., EMNLP 2023. [Paper]
8. Zero-Shot Listwise Document Reranking with a Large Language Model, Ma et al., arXiv 2023. [Paper]
9. Found in the Middle: Permutation Self-Consistency Improves Listwise Ranking in Large Language Models, Tang et al., arXiv 2023. [Paper]
10. Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting, Qin et al., arXiv 2023. [Paper]
11. A Setwise Approach for Effective and Highly Efficient Zero-shot Ranking with Large Language Models, Zhuang et al., arXiv 2023. [Paper]

Utilizing LLMs for Training Data Augmentation

1. ExaRanker: Explanation-Augmented Neural Ranker, Ferraretto et al., SIGIR 2023 (Short Paper). [Paper]
2. InPars-Light: Cost-Effective Unsupervised Training of Efficient Rankers, Boytsov et al., arXiv 2023. [Paper]
3. Generating Synthetic Documents for Cross-Encoder Re-Rankers, Askari et al., arXiv 2023. [Paper]
4. Instruction Distillation Makes Large Language Models Efficient Zero-shot Rankers, Sun et al., arXiv 2023. [Paper]
5. RankVicuna: Zero-Shot Listwise Document Reranking with Open-Source Large Language Models, Pradeep et al., arXiv 2023. [Paper]
6. RankZephyr: Effective and Robust Zero-Shot Listwise Reranking is a Breeze!, Pradeep et al., arXiv 2023. [Paper]

Reader

Passive Reader

1. REALM: Retrieval-Augmented Language Model Pre-Training, Guu et al., ICML 2020. [Paper]
2. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, Lewis et al., NeurIPS 2020. [Paper]
3. REPLUG: Retrieval-Augmented Black-Box Language Models, Shi et al., arXiv 2023. [Paper]
4. Atlas: Few-shot Learning with Retrieval Augmented Language Models, Izacard et al., JMLR 2023. [Paper]
5. Internet-augmented Language Models through Few-shot Prompting for Open-domain Question Answering, Lazaridou et al., arXiv 2022. [Paper]
6. Rethinking with Retrieval: Faithful Large Language Model Inference, He et al., arXiv 2023. [Paper]
7. FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation, Vu et al., arxiv 2023. [Paper]
8. Enabling Large Language Models to Generate Text with Citations, Gao et al., EMNLP 2023. [Paper]
9. Chain-of-Note: Enhancing Robustness in Retrieval-Augmented Language Models, Yu et al., arxiv 2023. [Paper]
10. Improving Retrieval-Augmented Large Language Models via Data Importance Learning, Lyu et al., arXiv 2023. [Paper]
11. Search Augmented Instruction Learning, Luo et al., EMNLP 2023 (Findings). [Paper]
12. RADIT: Retrieval-Augmented Dual Instruction Tuning, Lin et al., arXiv 2023. [Paper]
13. Improving Language Models by Retrieving from Trillions of Tokens, Borgeaud et al., ICML 2022. [Paper]
14. In-Context Retrieval-Augmented Language Models, Ram et al., arXiv 2023. [Paper]
15. Interleaving Retrieval with Chain-of-thought Reasoning for Knowledge-intensive Multi-step Questions, Trivedi et al., ACL 2023. [Paper]
16. Improving Language Models via Plug-and-Play Retrieval Feedback, Yu et al., arXiv 2023. [Paper]
17. Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergy, Shao et al., EMNLP 2023 (Findings). [Paper]
18. Retrieval-Generation Synergy Augmented Large Language Models, Feng et al., arXiv 2023. [Paper]
19. Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection, Asai et al., arXiv 2023. [Paper]
20. Active Retrieval Augmented Generation, Jiang et al., EMNLP 2023. [Paper]

Active Reader

1. Measuring and Narrowing the Compositionality Gap in Language Models, Press et al., arXiv 2022. [Paper]
2. DEMONSTRATE–SEARCH–PREDICT: Composing Retrieval and Language Models for Knowledge-intensive NLP, Khattab et al., arXiv 2022. [Paper]
3. Answering Questions by Meta-Reasoning over Multiple Chains of Thought, Yoran et al., arXiv 2023. [Paper]

Compressor

1. LeanContext: Cost-Efficient Domain-Specific Question Answering Using LLMs, Arefeen et al., arXiv 2023. [Paper]
2. RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective Augmentation, Xu et al., arXiv 2023. [Paper]
3. TCRA-LLM: Token Compression Retrieval Augmented Large Language Model for Inference Cost Reduction, Liu et al., EMNLP 2023 (Findings). [Paper]
4. Learning to Filter Context for Retrieval-Augmented Generation, Wang et al., arXiv 2023. [Paper]

Analysis

1. Lost in the Middle: How Language Models Use Long Contexts, Liu et al., arXiv 2023. [Paper]
2. Investigating the Factual Knowledge Boundary of Large Language Models with Retrieval Augmentation, Ren et al., arXiv 2023. [Paper]
3. Exploring the Integration Strategies of Retriever and Large Language Models, Liu et al., arXiv 2023. [Paper]
4. Characterizing Attribution and Fluency Tradeoffs for Retrieval-Augmented Large Language Models, Aksitov et al., arXiv 2023. [Paper]
5. When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories, Mallen et al., ACL 2023. [Paper]

Applications

1. Augmenting Black-box LLMs with Medical Textbooks for Clinical Question Answering, Wang et al., arXiv 2023. [Paper]
2. ATLANTIC: Structure-Aware Retrieval-Augmented Language Model for Interdisciplinary Science, Munikoti et al., arXiv 2023. [Paper]
3. Crosslingual Retrieval Augmented In-context Learning for Bangla, Li et al., arXiv 2023. [Paper]
4. Clinfo.ai: An Open-Source Retrieval-Augmented Large Language Model System for Answering Medical Questions using Scientific Literature, Lozano et al., arXiv 2023. [Paper]
5. Enhancing Financial Sentiment Analysis via Retrieval Augmented Large Language Models, Zhang et al., ICAIF 2023. [Paper]
6. Interpretable Long-Form Legal Question Answering with Retrieval-Augmented Large Language Models, Louis et al., arXiv 2023. [Paper]
7. RETA-LLM: A Retrieval-Augmented Large Language Model Toolkit, Liu et al., arXiv 2023. [Paper]
8. Chameleon: a Heterogeneous and Disaggregated Accelerator System for Retrieval-Augmented Language Models, Jiang et al., arXiv 2023. [Paper]
9. RaLLe: A Framework for Developing and Evaluating Retrieval-Augmented Large Language Models, Hoshi et al., EMNLP 2023. [Paper]
10. Don't forget private retrieval: distributed private similarity search for large language models, Zyskind et al., arXiv 2023. [Paper]

Search Agent

Static Agent

1. LaMDA: Language Models for Dialog Applications, Thoppilan et al., arXiv 2022. [Paper]
2. Language Models that Seek for Knowledge: Modular Search & Generation for Dialogue and Prompt Completion, Shuster et al., EMNLP 2022 (Findings). [Paper]
3. Teaching language models to support answers with verified quotes, Menick et al., arXiv 2022. [Paper]
4. WebGLM: Towards An Efficient Web-Enhanced Question Answering System with Human Preferences, Liu et al., KDD 2023. [Paper]
5. A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis, Gur et al., arXiv 2023. [Paper]
6. Know Where to Go: Make LLM a Relevant, Responsible, and Trustworthy Searcher, Shi et al., arXiv 2023. [Paper]

Dynamic Agent

1. WebGPT: Browser-assisted question-answering with human feedback, Nakano et al., arXiv 2021. [Paper]
2. WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents, Yao et al., arXiv 2022. [Paper]
3. WebCPM: Interactive Web Search for Chinese Long-form Question Answering, Qin et al., ACL 2023. [Paper]
4. Mind2Web: Towards a Generalist Agent for the Web, Deng et al., arXiv 2023. [Paper]
5. WebArena: A Realistic Web Environment for Building Autonomous Agents, Zhou et al., arXiv 2023. [Paper]
6. Hierarchical Prompting Assists Large Language Model on Web Navigation, Sridhar et al., EMNLP 2023 (Findings). [Paper]

Other Resources

1. ACL 2023 Tutorial: Retrieval-based Language Models and Applications, Asai et al., ACL 2023. [Link]
2. A Survey of Large Language Models, Zhao et al., arXiv 2023. [Paper]
3. Information Retrieval Meets Large Language Models: A Strategic Report from Chinese IR Community, Ai et al., arXiv 2023. [Paper]