Awesome RL-based Agentic Search Papers

This repository summarizes recent research on reinforcement‑learning (RL)‑based agentic search systems. These systems treat information‑seeking as a decision process: when a large language model (LLM) faces a complex question, it can plan and act by issuing search queries, revising those queries, and integrating evidence into its reasoning. RL techniques allow these agents to learn when to search, how intensively to search and how to integrate retrieved evidence into reasoning.

For more details, please check out our survey paper: A Comprehensive Survey on Reinforcement Learning-based Agentic Search: Foundations, Roles, Optimizations, Evaluations, and Applications. If you find this repository helpful, please cite our survey paper.

@article{lin2025comprehensive,
  title={A Comprehensive Survey on Reinforcement Learning-based Agentic Search: Foundations, Roles, Optimizations, Evaluations, and Applications},
  author={Minhua Lin, Zongyu Wu, Zhichao Xu, Hui Liu, Xianfeng Tang, Qi He, Charu Aggarwal, Hui Liu, Xiang Zhang, Suhang Wang},
  journal={arXiv preprint arXiv:2510.16724},
  year={2025}
}

We are actively maintaining this repository!

Contents

• Overview of RL-based Agentic Search
• Illustrative Framework of RL-based Agentic Search
• Representative Survey
• Method
• Evaluation

Overview of RL-based Agentic Search

Illustrative Framework of RL-based Agentic Search

Representative Survey

Time	Paper Title	Venue
2026.3	SoK: Agentic Retrieval-Augmented Generation (RAG): Taxonomy, Architectures, Evaluation, and Research Directions	arXiv
2026.3	Deep Research of Deep Research: From Transformer to Agent, From AI to AI for Science	arXiv
2026.1	Agentic Reasoning for Large Language Models	arXiv
2025.12	Deep Research: A Systematic Survey	arXiv
2025.10	A Comprehensive Survey on Reinforcement Learning-based Agentic Search: Foundations, Roles, Optimizations, Evaluations, and Applications	arXiv
2025.9	Reinforcement Learning Foundations for Deep Research Systems: A Survey	arXiv
2025.9	The Landscape of Agentic Reinforcement Learning for LLMs: A Survey	TMLR
2025.8	Deep Research: A Survey of Autonomous Research Agents	arXiv
2025.8	A Survey of LLM-based Deep Search Agents: Paradigm, Optimization, Evaluation, and Challenges	arXiv
2025.6	Deep Research Agents: A Systematic Examination And Roadmap	arXiv
2025.6	From Web Search towards Agentic Deep Research: Incentivizing Search with Reasoning Agents	arXiv
2025.6	Reasoning RAG via System 1 or System 2: A Survey on Reasoning Agentic Retrieval-Augmented Generation for Industry Challenges	arXiv
2025.4	Synergizing RAG and Reasoning: A Systematic Review	arXiv
2025.1	Agentic Retrieval-Augmented Generation: A Survey on Agentic RAG	arXiv

---

Method

How RL is Used: Optimization Strategies

The below table summarizes representative works with corresponding optimization strategies. Specifically, ORM and PRM denote the Outcome Reward Model and the Process Reward Model, respectively. “Rule-based” indicates that the reward function is entirely computed from predefined rules; otherwise, an LLM is involved as a reward judge.

📊 Click to expand long table (scrollable).

Method	RL Func. Role	Cold Start?	Training Env.	RL Alg.	Reward Type	Reward Func.	Opt. Scope	Dataset
FA-SD	R–S Inter. Training Stability	-	Real-world Retrieval-Interleaved Search	Self-Distillation EMA Teacher	-	KL Self-Distillation Feedback-Augmented Supervision	Single-agent	Retrieval-interleaved search tasks
PCTD	R-Aware Opt. Multi-tool	-	Tool Retrieval Mobile Multi-Turn	RL	-	Counterfactual Retrieval Gain Preference Reward	Module-level	MTDTool
VideoSearcher (Code)	Multi-modal Multi-tool R–S Inter.	-	Real-world Video Deep Research	BiSPO	-	Tool-Invocation Objective Answer-Accuracy Objective	Single-agent	VideoSearch-QA
DeepSearch-Evolve	Adapt-Search Ctx-Mem. Task / Data Synthesis	-	Simulated Verifiable	Self-Distillation Fine-tuning	-	Trajectory Filtering Data Mixing	Single-agent	DeepSearch-World (420K tasks)
HOTE	Cooperative Multi-Agent Systems Self-Evolving	-	Real-world Open-Ended Research	Hybrid-Mode RL	-	Proposer–Solver–Judge Co-Evolution	Multi-agent	Three long-form deep-research benchmarks
SearchSwarm	P–E Orches. Ctx-Mem.	-	Real-world Harness-Guided	SFT	-	Delegation Trajectory Supervision	Multi-agent	BrowseComp, BrowseComp-ZH
LAPO	R–S Inter. Step-level	-	Local Retrieval	RL	Self-generated PRM	Answer-Likelihood Gain Sign-Consistency Gating	Step-level	Seven knowledge-intensive QA datasets
AWA-RL (Code)	Adapt-Search Reliability	-	Local Retrieval	RL	ORM + Abstention Reward	Dynamic Abstention Reward Answer Reward	Single-agent	[MuSiQue, HotpotQA, 2WikiMultiHopQA]
GRASP	R-Aware Opt. Search Efficiency Multi-tool	-	Local Retrieval	RL	ORM+PRM	Answer Accuracy Grounded Reading Complementary Search Turn Efficiency	Single-agent	Multi-hop reasoning benchmarks
DeepRubric	R–S Inter. Task / Data Synthesis	-	Synthetic Real-world Evaluation	GRPO	Rubric-based ORM	Evidence-Tree Rubrics Report Quality	Single-agent	9K query–rubric pairs; three deep-research benchmarks
Libra (Code, Data)	R-Aware Opt. Ctx-Mem. Struct-Nav.	✓	Simulated Repository Env.	Environment Optimization	-	Catalog Healing Localization Feedback	Environment-level	SWE-bench Lite
PaperPilot	Conv-Reform. Struct-Nav. R–S Inter.	✓	Scientific Literature	SFT Preference Optimization	-	Workflow Imitation Preference over Workflow Corruptions	Single-agent	Scientific literature search
Maven	Ctx-Mem. R–S Inter.	-	Long-context	GRPO	PRM	Evidence-State Rewards Action-level Transition Reward	Step-level	LongBench v2, LongReason, RULER
PlanRAG	Conv-Reform. R-Aware Opt. Search Efficiency	-	Real-world Query Planning	Dynamic Programming Cost Model	-	Logical Query Tree Cost	Module-level	WikiWeb-ERP
BaRA (Code)	Multi-tool Search Efficiency	-	Real-world Web	BFS Self-reflection	Rule-based	Liveness Verification Provenance Checks	Single-agent	Synthetic and real websites
Harness-1 (Code, Model)	Adapt-Search Ctx-Mem. Search Efficiency	-	Real-world Stateful Harness	RL	-	-	Single-agent	[BrowseComp-Plus, web, finance, patents, multi-hop QA]
Query Recycling	Search Efficiency Training Stability	✓	Simulated Sandboxed	GRPO	Rule-based ORM	Zero-Variance Query Recycling Answer Reward	Single-agent	Multi-hop QA
SAAS (Code)	Search Efficiency Adapt-Search	-	Real-world	GRPO	Rule-based ORM	Boundary-aware search penalty Answer reward	Single-agent	-
SD-Search	R–S Inter. R-Aware Opt.	-	Real-world	GRPO Self-Distillation	PRM	Hindsight query distillation Answer reward	Step-level	-
Search-E1	R–S Inter. Search Efficiency	-	Real-world	GRPO OPSD	Self-distillation	Answer EM Token-level KL	Step-level	QA benchmarks
GrepSeek	R-Aware Opt. Struct-Nav.	✓	Real-world Direct Corpus Interaction	GRPO	ORM	Answer EM/F1	Single-agent	-
QUEST	Adapt-Search Ctx-Mem.	✓	Synthetic Real-world	SFT RL	Rule-based ORM	Verifiable rubric-tree reward	Single-agent	-
SciResearcher	Adapt-Search Multi-tool Domain-Specific	✓	Synthetic Scientific	SFT Agentic RL	ORM	Scientific reasoning reward	Single-agent	[HLE-Bio/Chem-Gold, SuperGPQA-Hard-Biology, TRQA-Literature]
OThink-SRR1	R–S Inter. Search Efficiency	-	Real-world	GRPO-IR	Rule-based ORM+PRM	Format Retrieval hit Refinement hit Retrieval penalty	Single-agent	Multi-hop QA
CalibAdv (Code)	Training Stability R–S Inter.	-	Real-world	GRPO	ORM	Advantage Calibration	Step-level	-
O-Researcher	P–E Orches. Multi-agent	✓	Synthetic	SFT Agentic RL	-	-	Single-agent	Deep Research Benchmark
RAGShaper	Task / Data Synthesis R–S Inter.	✓	Synthetic	SFT	-	Robust trajectory synthesis	Single-agent	-
MTA-Agent	Multi-modal Multi-tool	✓	Synthetic Cached Tool Replay	SFT	-	Verified multi-hop trajectory synthesis	Single-agent	MTA-Vision-DeepSearch
On-Policy Data Evolution	Multi-modal Task / Data Synthesis	✓	Real-world On-policy	SFT RL	ORM	Rollout-feedback data evolution	Single-agent	-
Argus	P–E Orches. Ctx-Mem.	-	Real-world	RL	-	Evidence Assembly Source-traced Answer	Multi-agent	[BrowseComp-en]
CuSearch (Code)	Search Efficiency R-Aware Opt.	-	Real-world Simulated	GRPO RLVR	Verifiable Reward	Search-Depth Curriculum SDGA	Single-agent	[NQ, TriviaQA, PopQA, HotpotQA, 2WikiMultiHopQA, MuSiQue, Bamboogle]
HyperEyes (Code)	Multi-modal Multi-tool Search Efficiency	✓	Real-world	SFT RL	ORM+PRM	TRACE On-Policy Distillation	Single-agent	[IMEB, MMSearch, LiveVQA, InfoSeek]
VISOR	Multi-modal R–S Inter. Search Efficiency	✓	Real-world Visual Retrieval	GRPO	ORM+PRM	Visual Action Evaluation Credit Assignment Search Drift Control	Single-agent	[ViDoSeek, SlideVQA, MMLongBench]
RubricEM	Ctx-Mem. R–S Inter.	-	Real-world	Stage-Structured GRPO Meta-RL	PRM	Rubric Judgments Reflection	Single-agent	Long-form research benchmarks
DecEx-RAG (Code)	Adapt-Search R–S Inter.	-	Real-world	Process-level Policy Optimization	PRM	Decision Optimization Execution Optimization	Single-agent	Six datasets
OpenSearch-VL (Code)	Multi-modal Multi-tool Adapt-Search	✓	Real-world	Fatal-aware GRPO	ORM+PRM	Answer Acc Query Quality Format	Single-agent	[SearchVL-SFT-36k, SearchVL-RL-8k, SimpleVQA, VDR, MMSearch, LiveVQA, BrowseComp-VL, FVQA, InfoSeek]
OpenSeeker-v2	Adapt-Search	✓	Real-world	SFT	-	-	Single-agent	[BrowseComp-en, BrowseComp-zh, HLE, XBench]
LongSeeker	Ctx-Mem. Search Efficiency	✓	Real-world	SFT	-	Context Orchestration	Single-agent	[BrowseComp-en, BrowseComp-zh]
Search, Do not Guess	Adapt-Search Search Efficiency	✓	Real-world	Fine-tuning	-	Evidence-grounded Search Use	Single-agent	[Bamboogle, HotpotQA]
InfoTree	Search Efficiency Multi-tool	-	Tree-search Tool-use	GRPO	-	Rollout Informativeness UUCB	Step-level	[GAIA, HLE-100, BrowseComp-lite, APPS-verified, AgentBench-OS]
A$^2$TGPO	Multi-tool Step-level	-	Real-world	A$^2$TGPO	PRM	Information Gain Adaptive Turn-level Clipping	Step-level	-
IG-Search	R–S Inter. R-Aware Opt.	-	Real-world	GRPO	Rule-based PRM	Information Gain	Step-level	[NQ, TriviaQA, PopQA, HotpotQA, 2WikiMultiHopQA, MuSiQue, Bamboogle]
CW-GRPO	R-Aware Opt. R–S Inter.	-	Real-world	CW-GRPO	ORM+PRM	Retrieval Utility Reasoning Correctness	Step-level	[NQ, TriviaQA, PopQA, HotpotQA, 2WikiMultiHopQA, MuSiQue, Bamboogle]
Cycle-Consistent Search	Adapt-Search	-	Real-world	GRPO	PRM	Question Reconstructability	Single-agent	[NQ, TriviaQA, PopQA, HotpotQA, 2WikiMultiHopQA, MuSiQue, Bamboogle]
APEX-Searcher	Adapt-Search P–E Orches.	-	Real-world	GRPO	Rule-based ORM	Decomposition F1	Module-level	[HotpotQA, 2WikiMultiHopQA, MuSiQue, Bamboogle]
Search More, Think Less	Search Efficiency	✓	Real-world	Modified RLOO	ORM	LLM-Judge Format / Tool-use	Single-agent	[BrowseComp-en, BrowseComp-zh, GAIA, XBench, DeepResearchBench, WebWalkerQA]
To Search or Not to Search	Search Efficiency	-	Real-world	DPO	-	Decision Boundary Alignment	Step-level	[NQ, HotpotQA]
A-RAG (Code)	R-Aware Opt. R–S Inter.	-	Real-world Hierarchical Retrieval	-	-	Hierarchical Retrieval Interfaces Test-time Scaling	Single-agent	Multi-hop QA
SmartSearch	Conv-Reform. R-Aware Opt.	✓	Real-world	GRPO	PRM	Query Novelty Query Usefulness	Step-level	[HotpotQA, 2WikiMultiHopQA, MuSiQue, Bamboogle]
Evaluate-as-Action	R–S Inter.	-	Real-world	GRPO	PRM	Self-Evaluation Reward	Step-level	[NQ, TriviaQA, PopQA, HotpotQA, 2WikiMultiHopQA, MuSiQue, Bamboogle]
SE-Search	R–S Inter. Ctx-Mem.	-	Real-world	GRPO	ORM+PRM	Query Reward Memory Reward Answer Reward	Step-level	[NQ, TriviaQA, PopQA, HotpotQA, 2WikiMultiHopQA, MuSiQue, Bamboogle]
Search-P1	R–S Inter. Search Efficiency	-	Real-world	GRPO PPO	ORM+PRM	Path Reward Answer Reward	Step-level	[NQ, TriviaQA, PopQA, HotpotQA, 2WikiMultiHopQA, MuSiQue, Bamboogle, AD-QA]
SynPlanResearch-R1	Adapt-Search Multi-tool	✓	Real-world	GRPO	-	-	Single-agent	[HotpotQA, 2WikiMultiHopQA, MuSiQue, Bamboogle, GAIA, WebWalkerQA, GPQA]
RAPO	R-Aware Opt. Multi-tool	-	Real-world	RAPO	Retrieval-aware	Retrieval Reward Importance Shaping	Step-level	Agentic reasoning tasks
KARL	Adapt-Search Search Efficiency Ctx-Mem.	-	Real-world	OAPL	-	-	Single-agent	[BrowseComp-Plus, TREC-Biogen, FinanceBench, QAMPARI, Freshstack, PMBench]
GraphSearch	Struct-Nav. R-Aware Opt.	-	Graph Env.	-	-	Graph-aware Query Planning Hybrid Graph Retrieval	Single-agent	Node classification, link prediction
LiteResearcher (Code)	Adapt-Search Search Efficiency	-	Simulated Lite Virtual World	GRPO	ORM	Answer Correctness (LLM-Judge)	Single-agent	[GAIA, FRAMES, XBench]
CoSearch (Code)	R-Aware Opt. R–S Inter.	-	Real-world	GRPO	Rule-based ORM+PRM	Answer F1 Relevance Reward	Module-level	[NQ, TriviaQA, PopQA, HotpotQA, 2WikiMultiHopQA, MuSiQue, Bamboogle]
AutoSearch (Code)	Search Efficiency Search Intensity	-	Real-world	PPO GRPO	Rule-based ORM+PRM	Format Outcome EM Search Efficiency Search Quality F1	Step-level	[NQ, TriviaQA, PopQA, HotpotQA, 2WikiMultiHopQA, Bamboogle]
DR-Venus (Code, Models)	Adapt-Search Ctx-Mem.	✓	Real-world	IGPO-based Agentic RL	ORM+Turn-level Reward	Information Gain Format Regularization	Single-agent	[GAIA, BrowseComp-en, FRAMES]
ProMMSearchAgent	Multi-modal Adapt-Search	-	Simulated Sim-to-Real	RL with Process-oriented Reward	PRM	Knowledge-Boundary / Search-Decision Reward	Single-agent	[FVQA, InfoSeek, MMSearch]
IGPO	Adapt-Search R–S Inter.	-	Real-world	IGPO	Rule-based ORM+PRM	Answer F1 Information Gain	Step-level	[NQ, TriviaQA, PopQA, HotpotQA, 2WikiMultiHopQA, MuSiQue, Bamboogle]
SearchGym (Code)	Adapt-Search	-	Simulated	SearchGym-RL	-	Purified Feedback Curriculum	Single-agent	-
SIGHT	Adapt-Search R–S Inter. Search Efficiency	-	Real-world	GRPO	-	SES Correctness Information Gain	Single-agent	-
Search-R2	R–S Inter.	-	Real-world	-	PRM	Dense Process Reward	Single-agent	-
DeepControl	Search Efficiency Search Intensity	-	Real-world	RL	Utility-based control	Information utility Retrieval continuation Granularity control	Step-level	7 QA benchmarks
WideSeek-R1	Search Efficiency Cooperative Multi-Agent Systems	-	Real-world	MARL	-	-	Multi-agent	WideSearch
M-ASK (Code)	P–E Orches. Ctx-Mem.	-	Real-world	-	-	Turn-level Reward	Multi-agent	-
PRAISE	Search Efficiency Step-level	-	Real-world	-	PRM	Intermediate Step Reward	Step-level	-
Behavior Priming (Code)	Adapt-Search Ctx-Mem.	✓	Real-world Behavior-Primed Trajectories	SFT RL	ORM	Behavior-Primed Trajectory Reward	Single-agent	[GAIA, Web benchmarks, multi-hop QA]
Fathom-DeepResearch	Adapt-Search Ctx-Mem. Multi-tool	✓	Real-world Live Web	RAPO	PRM+ORM	Step-level Utility Reward Curriculum Pruning	Single-agent	[SimpleQA, FRAMES, WebWalker, Seal0, MuSiQue, HLE, GPQA, MedQA]
Search-R1	Adapt-Search	✗	Real-world	PPO GRPO	Rule-based ORM	Answer EM	Single-agent	[NQ, TriviaQA, PopQA, HotpotQA, MuSiQue, 2WikiMultiHopQA, Bamboogle]
ReSearch	Adapt-Search	✗	Real-world	GRPO	Rule-based ORM	Format Answer F1	Single-agent	[HotpotQA, 2WikiMultiHopQA, MuSiQue, Bamboogle]
AutoCoA	Adapt-Search	✓	Real-world	GRPO	Rule-based ORM	Format Answer EM	Single-agent	[NQ, TriviaQA, PopQA, HotpotQA, MuSiQue, 2WikiMultiHopQA, Bamboogle]
SimpleDeepSearcher	Adapt-Search	✓	Real-world	SFT	-	-	Single-agent	[2WikiMultiHopQA, MuSiQue, Bamboogle, FRAMES, GAIA]
ExSearch	Adapt-Search	✗	Real-world	GEM	PRM	Trajectory Quality	Single-agent	[NQ, HotpotQA, MuSiQue]
IKEA	Search Efficiency	✗	Real-world	GRPO	Rule-based ORM	Format Answer EM Knowledge-boundary	Step-level	[NQ, PopQA, HotpotQA, 2WikiMultiHopQA]
R1-Searcher	Adapt-Search	✓	Real-world	GRPO Reinforce++	Rule-based ORM	Format Answer F1	Single-agent	[HotpotQA, 2WikiMultiHopQA, Bamboogle, MuSiQue]
R1-Searcher++	Search Efficiency	✓	Real-world	GRPO Reinforce++	Rule-based ORM	Format Answer EM Std of Search Calls	Single-agent	[HotpotQA, 2WikiMultiHopQA, Bamboogle, MuSiQue]
DeepRAG	Adapt-Search Search Efficiency	✓	Real-world	GRPO	Rule-based ORM	Answer EM Retrieval Cost	Single-agent	[HotpotQA, 2WikiMultiHopQA, CAG, PopQA, WebQuestions, MuSiQue]
UR²	Adapt-Search	✓	Real-world Curriculum	Reinforce++	Rule-based ORM	Format Answer EM Fallback Penalty	Single-agent	[MATH, Minerva, MedQA, MMLU-Pro, HotpotQA, Bamboogle, 2WikiMultiHopQA, MuSiQue]
SSRL	Adapt-Search	✓	Simulated Self-Search	GRPO	Rule-based ORM	Format Answer EM	Single-agent	[NQ, TriviaQA, HotpotQA, MuSiQue, 2WikiMultiHopQA, Bamboogle]
Pangu DeepDiver	Adapt-Search Search Intensity	✓	Real-world	GRPO	Rule-based ORM	Format Answer EM Extra Search	Single-agent	[Pangu, Bamboogle, WebWalkerQA, FRAMES]
ReZero	Search Intensity	✗	Real-world	GRPO	ORM+PRM	Format Answer LLM-Judge Retry	Step-level	[Apollo-3]
StepSearch	Adapt-Search Search Intensity	✗	Real-world	PPO	Rule-based ORM+PRM	Format Answer F1 Search Key Information Gain Redundancy Penalty	Step-level	[HotpotQA, MuSiQue, 2WikiMultiHopQA, Bamboogle]
VERITAS	Adapt-Search R-Aware Opt.	✗	Real-world	PPO	ORM+PRM	Answer EM Enhancing Faithfulness	Step-level	[NQ, TriviaQA, PopQA, HotpotQA, MuSiQue, 2WikiMultiHopQA, Bamboogle]
ReasonRAG	Search Efficiency R-S Inter.	✗	Real-world MCTS	DPO	PRM	Shortest Path	Step-level	[PopQA, HotpotQA, 2WikiMultiHopQA, Bamboogle, MuSiQue]
Web-Sailor	Adapt-Search Ctx-Mem.	✓	Real-world	DUPO	ORM	Format Answer F1	Single-agent	[li2025sailorfogqa, BrowseComp-en, BrowseComp-zh, GAIA, XBench]
WebSailor-V2	Multi-tool Ctx-Mem.	✓	Real-world	GRPO	Rule-based ORM	Format Answer F1	Single-agent	[li2025sailorfogqav2, BrowseComp-en, BrowseComp-zh, GAIA, XBench, HLE, DeepResearchBench]
Search Wisely	Search Efficiency	✗	Real-world	β-GRPO	Rule-based ORM	Confidence-based Answer EM	Single-agent	[NQ, HotpotQA, TriviaQA, 2WikiMultiHopQA, Bamboogle, MuSiQue]
ZeroSearch	Search Efficiency	✓	Simulated Curriculum	PPO GRPO Reinforce	Rule-based ORM	Answer F1	Single-agent	[NQ, TriviaQA, PopQA, HotpotQA, 2WikiMultiHopQA, MuSiQue, Bamboogle]
ParallelSearch	Search Efficiency	✓	Real-world	GRPO	Rule-based ORM	Format Answer EM Query Decomposition Search count	Single-agent	[NQ, TriviaQA, PopQA, HotpotQA, 2WikiMultiHopQA, MuSiQue, Bamboogle]
RAG-R1	Search Efficiency Conv-Reform.	✓	Real-world	PPO	ORM	Answer EM	Single-agent	[NQ, PopQA, TriviaQA, HotpotQA, 2WikiMultiHopQA, MuSiQue, Bamboogle]
RL-QR	Conv-Reform. R-Aware Opt.	✓	Synthetic	GRPO	Rule-based ORM	Verifiable Search Reward	Module-level	MTEB VIDORE V2, MS MARCO v2.1
ConvSearch-R1	Conv-Reform.	✓	Real-world	GRPO	ORM	Format Rank-Incentive	Step-level	[TopiOCQA], [QReCC]
MaskSearch	Conv-Reform. R–S Inter.	✓	Real-world Curriculum RAMP	DAPO	Rule-based ORM	Format Answer Recall Length penalty	Single-agent	[HotpotQA, FANOUTQA, MuSiQue, 2WikiMultiHopQA, Bamboogle, FreshQA]
DeepRetrieval	R-Aware Opt.	✓	Simulated	PPO	ORM	Format Answer Recall	Single-level	[NQ, TriviaQA, SQuAD, FEVER, FactoidQA]
WebThinker	Search Efficiency	✗	Real-world	DPO	PRM	Answer EM Tool Calls Length penalty	Single-agent	[GPQA], GAIA, WebWalkerQA, HLE, SuperGPQA, OpenThoughts, NaturalReasoning, NuminaMath
s3	R-Aware Opt.	✓	Simulated	PPO	Rule-based ORM	Gain Beyond RAG	Module-level	[NQ, TriviaQA, PopQA, HotpotQA, 2WikiMultiHopQA, MuSiQue]
R-Search	R–S Inter.	✗	Real-world	PPO GRPO	Rule-based ORM+PRM	Format Answer F1 Evidence Quality	Single-agent	[HotpotQA, 2WikiMultiHopQA, MuSiQue, Bamboogle]
AutoRefine	R–S Inter.	✗	Real-world	GRPO	ORM+PRM	Answer F1 Retrieval Reward	Step-level	[NQ, TriviaQA, HotpotQA, 2WikiMultiHopQA, MuSiQue, Bamboogle, PopQA]
EvolveSearch	R–S Inter.	✓	Real-world Self-evolving	GRPO	ORM	Format Answer LLM-Judge	Single-agent	[NQ, TriviaQA, HotpotQA, 2WikiMultiHopQA, MuSiQue, Bamboogle, PopQA]
O²-Searcher	R–S Inter.	✓	Simulated	GRPO	Rule-based ORM	Format Diversity reward Factual reward	Single-agent	[NQ, HotpotQA, TriviaQA, PopQA, 2WikiMultiHopQA, MuSiQue, Bamboogle]
Atom-Searcher	R–S Inter.	✓	Real-world Curriculum	GRPO	PRM+Rule-based ORM	Format Answer F1 Atomic thought reward	Step-level	[NQ, TriviaQA, HotpotQA, 2WikiMultiHopQA, MuSiQue, Bamboogle, PopQA]
ReSum	Ctx-Mem.	✗	Real-world	Resume-GRPO	ORM	Answer LLM-Judge	Single-agent	[GAIA, BrowseComp-en, BrowseComp-zh, WebWalkerQA, XBench]
SFR-DeepResearch	Ctx-Mem. Multi-tool	✗	Real-world	REINFORCE	ORM	Answer LLM-Judge	Single-agent	[FRAMES, GAIA, HLE]
MAO-ARAG	P–E Orches.	✓	Real-world	PPO	ORM	Format Cost Penalty Answer F1	Multi-agent	[NQ, HotpotQA, 2WikiMultiHopQA, MuSiQue, Bamboogle, PopQA, AmbigQA]
OPERA	P–E Orches.	✓	Real-world	MAPGRPO	PRM+ORM	Answerer Reward Planner Reward Rewriter Reward	Multi-agent	[HotpotQA, 2WikiMultiHopQA, MuSiQue, NQ, Multihop-rag]
AI-SearchPlanner	P–E Orches.	✗	Real-world	PPO	ORM	Answer LLM-Judge Trajectory Rationality	Module-level	[NQ, TriviaQA, HotpotQA, 2WikiMultiHopQA, MuSiQue, Bamboogle, PopQA]
AgentFlow	P–E Orches.	✗	Real-world	Flow-GRPO	Rule-based ORM	EM LLM Judge	Module-level	[HotpotQA, 2WikiMultiHopQA, MuSiQue, Bamboogle, AIME24, AMC23, Gameof24, GPQA, MedQA]
SIRAG	Cooperative Multi-Agent Systems	✗	Real-world	PPO	PRM	Process LLM-Judge	Multi-agent	[2WikiMultiHopQA, HotpotQA, NQ, PopQA]
MMOA-RAG	Cooperative Multi-Agent Systems R-aware Opt.	✗	Real-world	MA-PPO	Rule-based ORM	Answer F1 Efficiency penalty	Multi-agent	[HotpotQA, 2WikiMultiHopQA, AmbigQA]
Tool-Star	Multi tool	✓	Real-world	REINFORCE++ GRPO DPO	Rule-based ORM	Format Answer EM	Single-agent	[LTMS, MATH, GSM8K, WebWalkerQA, HotpotQA, 2WikiMultiHopQA, MuSiQue, Bamboogle]
WebWatcher	Multi tool Multi-modal	✓	Real-world	GRPO	ORM	Format Answer LLM-Judge	Single-agent	[HLE, Wu2025mm, LiveVQA, MMSearch, SimpleVQA]
Visual-ARFT	Multi-modal Multi-tool Adapt-Search	✓	Real-world	GRPO	Rule-based ORM+PRM	Format Answer F1 Query Semantic Sim.	Single-agent	[MATSearch]
VRAG-RL	Multi-modal Search Efficiency	✓	Simulated	GRPO	ORM	Format Answer LLM-Judge Retrieval Efficiency	Single-agent	[SlideVQA, VidORAG, MMLongBench]
MMSearch-R1	Multi-modal Search Efficiency	✗	Real-world	GRPO	Rule-based ORM	Format Answer EM Search Penalty	Single-agent	[MMSearch, Chen2023can, MMSearch, SimpleVQA, LiveVQA]
GRAIL	Adapt-Search Struct-Nav.	✓	Real-world Graph Env.	GRPO	PRM	Process LLM-Judge	Single-agent	[WebQuestions, MetaQA, WebQSP]
DynaSearcher	Struct-Nav.	✗	Real-world Graph Env. KG+Doc Search	GRPO	Rule-based ORM	Format Answer F1 Information Gain Retrieval Penalty	Single-agent	[HotpotQA, 2WikiMultiHopQA, MuSiQue, Bamboogle, FRAMES]
HARIS	R-S Inter.	✗	Real-world	GRPO	Rule-based ORM	Format Answer Accuracy Decision Accuracy	Multi-agent	[M3, HoVer, CheckWhy]
DeepNote	Adapt-Search Conv-Reform.	✗	Real-world	DPO	-	-	Single-agent	[HotpotQA, 2WikiMultiHopQA, MuSiQue, StrategyQA, ASQA]
DeepResearcher	Adapt-Search Search Efficiency Ctx-Mem.	✗	Real-world	GRPO	Rule-based ORM	Format Answer F1	Module-level	[NQ, TriviaQA, HotpotQA, 2WikiMultiHopQA]
SWiRL	Adapt-Search R-S Inter.	✗	Simulated Offline	Policy Gradient	PRM	Step-wise LLM-Judge (Gemini 1.5 Pro)	Step-level	[HotpotQA, MuSiQue, CoFCA, GSM8K, Qasper, BeerQA]
WebDancer (Code)	Multi tool	✓	Real-world	DAPO	ORM	Answer EM	Single-agent	[GAIA, WebWalkerQA, BrowseComp-zh, BrowseComp-en]
MedResearcher-R1	Adapt-Search Multi-Tool	✓	Real-world Medical Tool	GRPO	ORM	Answer Acc Response Quality Efficiency penalty	Single-agent	[XBench, GAIA, MedBrowseComp]
Lucy	Search Efficiency R–S Inter.	✓	Real-world SLMs	DAPO	Rule-based ORM	Format/XML validity Answer EM Tool exec. success Visit/Search ratio Efficient thinking	Single-agent	[WebWalkerQA]
ASearcher	R-S Inter. Ctx-Mem. Multi-tool	✓	Real-world Browser Env. Asynchronous	GRPO	ORM	Answer LLM-Judge	Single-agent	[NQ, TriviaQA, PopQA, HotpotQA, 2WikiMultiHopQA, MuSiQue, Bamboogle, FRAMES, GAIA, XBench]
WebExplorer	Ctx-Mem. Conv-Reform.	✓	Real-world Curriculum	GRPO	Rule-based ORM	Format Answer EM	Single-agent	[BrowseComp-zh, BrowseComp-en, GAIA, WebWalkerQA, FRAMES, XBench, HLE]
WebResearcher	Multi-tool	✓	Real-world Curriculum	GSPO	Rule-based ORM	Answer EM	Single-agent	[HLE, GAIA, BrowseComp-en, BrowseComp-zh, XBench, FRAMES]
RECON	Ctx-Mem.	✓	Real-world	PPO	Rule-based ORM	Answer EM	Single-agent	[NQ, TriviaQA, Bamboogle, HotpotQA, 2WikiMultiHopQA, MuSiQue, PopQA]
AgentGym-RL	Cooperative Multi-Agent Systems Multi tool	-	-	-	-	-	Unified RL Agentic Framework	-
Chain-of-Agents	Cooperative Multi-Agent Systems Multi tool	-	-	-	-	-	Unified RL Agentic Framework	-
Verl	Multi tool	-	-	-	-	-	Unified RL Agentic Framework	-
VerlTool	Multi tool	-	-	-	-	-	Unified RL Agentic Framework	-

What RL is for: Functional Roles in Agentic Search

Retrieval Control

Time	Paper Title	Role	Venue	Code
2026.7	SearchOS-V1: Towards Robust Open-Domain Information-Seeking Agent Collaboration	Context & Memory Management; Search Efficiency; Multi-Agent Collaboration	arXiv	Code
2026.7	WebSwarm: Recursive Multi-Agent Orchestration for Deep-and-Wide Web Search	Adaptive Search Decision; Recursive Delegation; Search Coverage	arXiv
2026.7	DeepSearch-World: Self-Distillation for Deep Search Agents in a Verifiable Environment	Adaptive Search Decision; Self-Distillation; Verifiable Environment	arXiv
2026.7	VideoSearcher: Empowering Video Deep Research with Multi-Tool Agentic Reasoning via Reinforcement Learning	Adaptive Search Decision; Multi-Modal; Multi-Tool	arXiv	Code
2026.6	SearchSwarm: Towards Delegation Intelligence in Agentic LLMs for Long-Horizon Deep Research	Adaptive Search Decision; Delegation Intelligence; Context Management	arXiv
2026.7	To Answer or to Abstain: Mitigating Search-Agent Hallucinations via Abstention-Aware Reinforcement Learning	Adaptive Search Decision; Reliability / Abstention	arXiv	Code
2026.7	GRASP: GRanularity-Aware Search Policy for Agentic RAG	Adaptive Search Decision; Retrieval-Granularity Control; Multi-Tool	arXiv
2026.7	Libra: Training the Environment for Agentic Information Retrieval	Environment Optimization; Context & Memory Management	arXiv	Code Data
2026.7	Multi-Turn Agentic Scientific Literature Search via Workflow Induction	Adaptive Search Decision; Workflow Induction	arXiv
2026.7	BaRA: Budget-constrained and Reliable Web Data Collection Agent	Search Efficiency; Budget-constrained Web Data Collection	arXiv	Code
2026.6	Harness-1: Reinforcement Learning for Search Agents with State-Externalizing Harnesses	Adaptive Search Decision; Context & Memory Management; Search Efficiency	arXiv	Code Model
2026.6	Effective Reinforcement Learning for Agentic Search by Recycling Zero-Variance Queries During Training	Search Efficiency; Training Stability	arXiv
2026.5	SAAS: Self-Aware Reinforcement Learning for Over-Search Mitigation in Agentic Search	Search Efficiency; Adaptive Search Decision	arXiv	Code
2026.4	OThink-SRR1: Search, Refine and Reasoning with Reinforced Learning for Large Language Models	Adaptive Search Decision; Search Efficiency	arXiv
2026.4	VISOR: Agentic Visual Retrieval-Augmented Generation via Iterative Search and Over-horizon Reasoning	Adaptive Search Decision; Multi-modal Search Efficiency	arXiv
2026.5	QUEST: Training Frontier Deep Research Agents with Fully Synthetic Tasks	Adaptive Search Decision; Context & Memory Management	arXiv
2026.5	SciResearcher: Scaling Deep Research Agents for Frontier Scientific Reasoning	Adaptive Search Decision; Multi‑Tool; Domain-Specific	arXiv
2026.1	O-Researcher: An Open Ended Deep Research Model via Multi-Agent Distillation and Agentic RL	Adaptive Search Decision; Planner-Executor Orchestration	arXiv
2026.5	CuSearch: Curriculum Rollout Sampling via Search Depth for Agentic RAG	Search Efficiency; Curriculum Rollout Sampling	arXiv	Code
2026.3	Marco DeepResearch: Unlocking Efficient Deep Research Agents via Verification-Centric Design	Adaptive Search Decision; Verification-Centric Design	arXiv
2026.4	DR-Venus: Towards Frontier Edge-Scale Deep Research Agents with Only 10K Open Data	Adaptive Search Decision; Context & Memory Management; Edge-Scale Deep Research	arXiv	Code Models
2026.5	OpenSeeker-v2: Pushing the Limits of Search Agents with Informative and High-Difficulty Trajectories	Adaptive Search Decision; Search Efficiency	arXiv
2026.5	LongSeeker: Elastic Context Orchestration for Long-Horizon Search Agents	Context & Memory Management; Search Efficiency	arXiv
2026.4	Search, Do not Guess: Teaching Small Language Models to Be Effective Search Agents	Adaptive Search Decision; Search Efficiency	arXiv
2026.1	Deep Search with Hierarchical Meta-Cognitive Monitoring Inspired by Cognitive Neuroscience	Adaptive Search Decision; Context & Memory Management	arXiv
2026.3	APEX-Searcher: Augmenting LLMs' Search Capabilities through Agentic Planning and Execution	Adaptive Search Decision; Planner-Executor Orchestration	arXiv
2026.3	Meta-Reinforcement Learning with Self-Reflection for Agentic Search	Adaptive Search Decision; Self-Reflection	arXiv	Code
2026.2	Search More, Think Less: Rethinking Long-Horizon Agentic Search for Efficiency and Generalization	Adaptive Search Decision; Search Efficiency	arXiv	Code
2026.2	To Search or Not to Search: Aligning the Decision Boundary of Deep Search Agents via Causal Intervention	Adaptive Search Decision; Search Efficiency	WWW 2026
2026.3	KARL: Knowledge Agents via Reinforcement Learning	Adaptive Search Decision; Search Efficiency; Context & Memory Management	arXiv
2026.4	LiteResearcher: A Scalable Agentic RL Training Framework for Deep Research Agent	Adaptive Search Decision; Search Efficiency	arXiv	Code
2026.4	AutoSearch: Adaptive Search Depth for Efficient Agentic RAG via Reinforcement Learning	Search Efficiency; Search Intensity	arXiv	Code
2026.2	SIGHT: Reinforcement Learning with Self-Evidence and Information-Gain Diverse Branching for Search Agent	Adaptive Search Decision; Search Efficiency	arXiv
2026.2	Adaptive Information Control for Search-Augmented LLM Reasoning	Search Efficiency; Search Intensity / Information Control	arXiv
2026.1	SearchGym: Bootstrapping Real-World Search Agents via Cost-Effective and High-Fidelity Environment Simulation	Adaptive Search Decision; Sim-to-Real / Environment Simulation	arXiv	Code
2026.2	How to Train Your Deep Research Agent? Prompt, Reward, and Policy Optimization in Search-R1	Adaptive Search Decision; Search Efficiency	arXiv
2026.2	REDSearcher: A Scalable and Cost-Efficient Framework for Long-Horizon Search Agents	Adaptive Search Decision; Search Efficiency	arXiv
2026.2	Agent-Omit: Training Efficient LLM Agents for Adaptive Thought and Observation Omission via Agentic Reinforcement Learning	Search Efficiency	arXiv
2026.2	IntentRL: Training Proactive User-intent Agents for Open-ended Deep Research via Reinforcement Learning	Adaptive Search Decision; User-intent / Autonomy–Interaction	arXiv
2026.2	Training Multi-Turn Search Agent via Contrastive Dynamic Branch Sampling	Adaptive Search Decision; Search Efficiency	arXiv
2026.1	BAPO: Boundary-Aware Policy Optimization for Reliable Agentic Search	Adaptive Search Decision; Reliability / IDK Boundary	arXiv	Code
2026.1	Agentic-R: Learning to Retrieve for Agentic Search	Retriever‑Aware Optimization; Adaptive Search Decision	arXiv	Code
2026.1	Agentic Conversational Search with Contextualized Reasoning via Reinforcement Learning	Conversational Reformulation; Adaptive Search Decision	arXiv
2026.1	OffSeeker: Online Reinforcement Learning Is Not All You Need for Deep Research Agents	Search Efficiency; Offline Training	arXiv
2025.11	Thinker: Training LLMs in Hierarchical Thinking for Deep Search via Multi-Turn Interaction	Adaptive Search Decision; Search Intensity	AAAI 2026	Code
2025.11	MemSearcher: Training LLMs to Reason, Search and Manage Memory via End-to-End Reinforcement Learning	Adaptive Search Decision; Context & Memory Management	arXiv	Code
2025.10	Towards Agentic Self-Learning LLMs in Search Environment	Adaptive Search Decision; Self-Learning	arXiv
2025.10	Beneficial Reasoning Behaviors in Agentic Search and Effective Post-training to Obtain Them	Adaptive Search Decision; Context & Memory Management	arXiv	Code
2025.9	Fathom-DeepResearch: Unlocking Long Horizon Information Retrieval and Synthesis for SLMs	Adaptive Search Decision; Context & Memory Management; Multi-Tool	arXiv
2025.09	DeepDive: Advancing Deep Search Agents with Knowledge Graphs and Multi-Turn RL	Adaptive Search Decision; Search Efficiency; Structured Knowledge Navigation	arXiv	Code
2025.09	WebSailor‑V2: Bridging the Chasm to Proprietary Agents via Synthetic Data and Scalable Reinforcement Learning	Multi‑Tool; Context & Memory Management	arXiv	Code
2025.08	MedResearcher‑R1: Expert‑Level Medical Deep Researcher via a Knowledge‑Informed Trajectory Synthesis Framework	Adaptive Search Decision; Multi‑Tool	arXiv	Code
2025.08	ParallelSearch: Train Your LLMs to Decompose Query and Search Sub‑Queries in Parallel with Reinforcement Learning	Search Efficiency	arXiv
2025.08	SSRL: Self‑Search Reinforcement Learning	Adaptive Search Decision	arXiv	Code
2025.08	UR²: Unify RAG and Reasoning through Reinforcement Learning	Adaptive Search Decision	arXiv	Code
2025.07	RAG‑R1: Incentivizing the Search and Reasoning Capabilities of LLMs through Multi‑Query Parallelism	Search Efficiency; Convsational Reformulation	arXiv	Code
2025.7	Annotation-Free Reinforcement Learning Query Rewriting via Verifiable Search Reward	Query Reformulation; Retriever‑Aware Optimization	arXiv
2025.05	Pangu DeepDiver: Adaptive Search Intensity Scaling via Open‑Web Reinforcement Learning	Adaptive Search Decision; Search Intensity	NeurIPS 2025 (Spotlight)
2025.05	Process vs. Outcome Reward: Which Is Better for Agentic RAG Reinforcement Learning	Retriever‑Aware Optimization	NeurIPS 2025	Code
2025.05	Search Wisely: Mitigating Sub‑Optimal Agentic Searches by Reducing Uncertainty	Search Efficiency	EMNLP 2025 (Main)
2025.05	SimpleDeepSearcher: Deep Information Seeking via Web-Powered Reasoning Trajectory Synthesis	Adaptive Search Decision	EMNLP 2025 (Findings)	Code
2025.05	StepSearch: Igniting LLMs’ Search Ability via Step‑Wise Proximal Policy Optimization	Adaptive Search Decision; Search Intensity	EMNLP 2025 (Main)	Code
2025.05	R1‑Searcher++: Incentivizing the Dynamic Knowledge Acquisition of LLMs via Reinforcement Learning	Search Efficiency	arXiv	Code
2025.05	ZeroSearch: Incentivize the Search Capability of LLMs without Searching	Search Efficiency	arXiv	Code
2025.04	WebThinker: Empowering Large Reasoning Models with Deep Research Capability	Search Efficiency	NeurIPS 2025	Code
2025.04	ReZero: Enhancing LLM Search Ability by Trying One‑More‑Time	Search Intensity	arXiv	Code
2025.04	DeepResearcher: Scaling Deep Research via Reinforcement Learning in Real‑World Environments	Adaptive Search Decision; Search Efficiency; Context & Memory Management	arXiv	Code
2025.04	Synthetic Data Generation & Multi-Step RL for Reasoning & Tool Use	Adaptive Search Decision; Reasoning-Search Interaction	arXiv
2025.03	Learning to Reason with Search for LLMs via Reinforcement Learning	Adaptive Search Decision	NeurIPS 2025
2025.03	Search‑R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning	Adaptive Search Decision	COLM 2025	Code
2025.03	R1‑Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learning	Adaptive Search Decision	arXiv	Code
2025.02	DeepRAG: Thinking to Retrieve Step by Step for Large Language Models	Adaptive Search Decision; Search Efficiency	arXiv

Query Optimization

Time	Paper Title	Role	Venue	Code
2026.7	PCTD: Preference-Guided Counterfactual Task Decomposition for Agent Tool Retrieval	Retriever-Aware Optimization; Task Decomposition; Tool Retrieval	arXiv
2026.7	Bridge Evidence: Static Retrieval Utility Does Not Predict Causal Utility in Multi-Step Agentic Search	Retriever-Aware Evaluation; Counterfactual Trajectory Utility	arXiv
2026.7	EvoGraph-R1: Self-Evolving Multimodal Knowledge Hypergraphs for Agentic Retrieval	Structured Knowledge Navigation; Adaptive Graph Retrieval	CVPR 2026
2026.7	GRASP: GRanularity-Aware Search Policy for Agentic RAG	Retriever-Aware Optimization; Granularity Control	arXiv
2026.6	MCompassRAG: Topic Metadata as a Semantic Compass for Paragraph-Level Retrieval	Retriever-Aware Optimization; Metadata-Guided Retrieval	arXiv	Code
2026.7	When RAG Meets Query Planning: Logical Query Trees for Resolving Exploratory Reasoning Problems	Query Planning; Logical Query Trees	SIGMOD 2027
2026.7	Libra: Training the Environment for Agentic Information Retrieval	Environment Optimization; Navigable Catalogs	arXiv	Code Data
2026.7	Multi-Turn Agentic Scientific Literature Search via Workflow Induction	Workflow Induction; Scientific Literature Search	arXiv
2026.5	GrepSeek: Training Search Agents for Direct Corpus Interaction	Retriever‑Aware Optimization; Direct Corpus Interaction	arXiv
2026.1	RAGShaper: Eliciting Sophisticated Agentic RAG Skills via Automated Data Synthesis	Retriever‑Aware Optimization; Data Synthesis	arXiv
2026.5	LatentRAG: Latent Reasoning and Retrieval for Efficient Agentic RAG	Latent Retrieval; Search Efficiency	arXiv
2026.5	Rethinking Reasoning-Intensive Retrieval: Evaluating and Advancing Retrievers in Agentic Search Systems	Retriever‑Aware Optimization; Agentic Retrieval Evaluation	ACL 2026	Code
2026.5	Beyond Semantic Similarity: Rethinking Retrieval for Agentic Search via Direct Corpus Interaction	Retriever‑Aware Optimization; Direct Corpus Interaction	arXiv
2026.5	AgenticRAG: Agentic Retrieval for Enterprise Knowledge Bases	Agentic Retrieval; In-document Navigation	arXiv
2026.5	Rethinking Agentic RAG: Toward LLM-Driven Logical Retrieval Beyond Embeddings	Agentic Retrieval; Logical Querying	arXiv
2026.1	SmartSearch: Process Reward-Guided Query Refinement for Search Agents	Retriever‑Aware Optimization; Query Refinement	SIGIR 2026	Code
2026.4	CoSearch: Joint Training of Reasoning and Document Ranking via Reinforcement Learning for Agentic Search	Retriever‑Aware Optimization	arXiv	Code
2026.4	Learning to Retrieve from Agent Trajectories	Retriever‑Aware Optimization	arXiv
2026.2	A-RAG: Scaling Agentic Retrieval-Augmented Generation via Hierarchical Retrieval Interfaces	Agentic Retrieval Interface; Hierarchical Retrieval	arXiv	Code
2026.1	When should I search more: Adaptive Complex Query Optimization with Reinforcement Learning	Retriever‑Aware Optimization; Query Reformulation	arXiv
2025.11	CriticSearch: Fine-Grained Credit Assignment for Search Agents via a Retrospective Critic	Retriever‑Aware Optimization	arXiv
2025.09	WebExplorer: Explore and Evolve for Training Long‑Horizon Web Agents	Context & Memory Management; Convsational Reformulation	arXiv
2025.08	OPERA: A Reinforcement Learning–Enhanced Orchestrated Planner‑Executor Architecture for Reasoning‑Oriented Multi‑Hop Retrieval	Planner-Executor Orchestration	arXiv
2025.08	ParallelSearch: Train Your LLMs to Decompose Query and Search Sub‑Queries in Parallel with Reinforcement Learning	Search Efficiency	arXiv
2025.07	RAG‑R1: Incentivizing the Search and Reasoning Capabilities of LLMs through Multi‑Query Parallelism	Search Efficiency; Convsational Reformulation	arXiv	Code
2025.05	ConvSearch‑R1: Enhancing Query Reformulation for Conversational Search with Reasoning via Reinforcement Learning	Convsational Reformulation	arXiv
2025.05	MaskSearch: A Universal Pre‑Training Framework to Enhance Agentic Search Capability	Convsational Reformulation; Reasoning-Search Interaction	arXiv
2025.05	s3: You Don’t Need That Much Data to Train a Search Agent via RL	Retriever‑Aware Optimization	EMNLP 2025 (Main)	code
2025.05	ZeroSearch: Incentivize the Search Capability of LLMs without Searching	Search Efficiency	arXiv	Code
2025.04	WebThinker: Empowering Large Reasoning Models with Deep Research Capability	Search Efficiency	NeurIPS 2025	Code
2025.03	DeepRetrieval: Hacking Real Search Engines and Retrievers with Large Language Models via Reinforcement Learning	Retriever‑Aware Optimization	COLM 2025	Code
2025.01	Improving Retrieval‑Augmented Generation through Multi‑Agent Reinforcement Learning	Cooperative Multi-Agent Systems	NeurIPS 2025	Code
2024.10	DeepNote: Note-Centric Deep Retrieval-Augmented Generation	Conversational Reformulation; Context & Memory Management	EMNLP 2025 (Findings)

Reasoning–Retrieval Integration

Time	Paper Title	Role	Venue	Code
2026.7	Why Does Feedback-Augmented Self-Distillation Fail to Improve Retrieval-Interleaved Search Agents?	Reasoning-Search Interaction; Self-Distillation Stability	arXiv
2026.7	Bridge Evidence: Static Retrieval Utility Does Not Predict Causal Utility in Multi-Step Agentic Search	Reasoning-Search Interaction; Causal Retrieval Utility	arXiv
2026.7	Analogical Deep Research: Retrieving and Integrating Historical Analogies for Foresight Analysis	Reasoning-Search Interaction; Analogical Retrieval; Reflection	arXiv
2026.7	EvoGraph-R1: Self-Evolving Multimodal Knowledge Hypergraphs for Agentic Retrieval	Reasoning-Search Interaction; Self-Evolving Knowledge Hypergraph	CVPR 2026
2026.7	DeepSearch-World: Self-Distillation for Deep Search Agents in a Verifiable Environment	Reasoning-Search Interaction; Grounded Reflection; Failure Recovery	arXiv
2026.7	VideoSearcher: Empowering Video Deep Research with Multi-Tool Agentic Reasoning via Reinforcement Learning	Reasoning-Search Interaction; Multi-Modal; Multi-Tool	arXiv	Code
2026.4	Towards Knowledgeable Deep Research: Framework and Benchmark	Reasoning-Search Interaction; Structured and Unstructured Knowledge	arXiv
2026.7	LAPO: Leave-One-Turn Attribution for Self-Generated Process Rewards in Multi-Turn Search Reasoning	Reasoning-Search Interaction; Process Supervision	arXiv
2026.7	GRASP: GRanularity-Aware Search Policy for Agentic RAG	Reasoning-Search Interaction; Retrieval-Granularity Control	arXiv
2026.6	DEEPRUBRIC: Evidence-Tree Rubric Supervision for Efficient Reinforcement Learning of Deep Research Agents	Reasoning-Search Interaction; Rubric-Based Supervision	arXiv
2026.6	ScaffoldAgent: Utility-Guided Dynamic Outline Optimization for Open-Ended Deep Research	Context & Memory Management; Dynamic Outline Optimization	arXiv
2026.6	S1-DeepResearch: Beyond Search, Toward Real-World Long-Horizon Research Agents	Context & Memory Management; Long-Horizon Trajectory Synthesis	arXiv
2026.7	Evidence-State Rewards for Long-Context Reasoning	Evidence-State Memory; Action-level Rewards	arXiv
2026.7	When RAG Meets Query Planning: Logical Query Trees for Resolving Exploratory Reasoning Problems	Reasoning-Search Interaction; Query Planning	SIGMOD 2027
2026.7	Multi-Turn Agentic Scientific Literature Search via Workflow Induction	Reasoning-Search Interaction; Workflow Induction	arXiv
2026.5	SD-Search: On-Policy Hindsight Self-Distillation for Search-Augmented Reasoning	Reasoning-Search Interaction; Process Supervision	arXiv
2026.5	Search-E1: Self-Distillation Drives Self-Evolution in Search-Augmented Reasoning	Reasoning-Search Interaction; Self-Distillation	arXiv
2026.5	GrepSeek: Training Search Agents for Direct Corpus Interaction	Reasoning-Search Interaction; Direct Corpus Interaction	arXiv
2026.5	QUEST: Training Frontier Deep Research Agents with Fully Synthetic Tasks	Reasoning-Search Interaction; Context & Memory Management	arXiv
2026.5	SciResearcher: Scaling Deep Research Agents for Frontier Scientific Reasoning	Reasoning-Search Interaction; Domain-Specific	arXiv
2026.5	Argus: Evidence Assembly for Scalable Deep Research Agents	Reasoning-Search Interaction; Evidence Assembly	arXiv
2026.5	RubricEM: Meta-RL with Rubric-guided Policy Decomposition beyond Verifiable Rewards	Context & Memory Management; Rubric-Guided Process Supervision	arXiv
2026.5	LatentRAG: Latent Reasoning and Retrieval for Efficient Agentic RAG	Reasoning-Search Interaction; Search Efficiency	arXiv
2026.5	Rethinking Agentic RAG: Toward LLM-Driven Logical Retrieval Beyond Embeddings	Reasoning-Search Interaction; Logical Retrieval	arXiv
2025.10	DecEx-RAG: Boosting Agentic Retrieval-Augmented Generation with Decision and Execution Optimization via Process Supervision	Reasoning-Search Interaction; Process Supervision	arXiv	Code
2026.3	Evaluate-as-Action: Self-Evaluated Process Rewards for Retrieval-Augmented Agents	Reasoning-Search Interaction; Process Supervision	arXiv
2026.4	CoSearch: Joint Training of Reasoning and Document Ranking via Reinforcement Learning for Agentic Search	Reasoning-Search Interaction; Retriever‑Aware Optimization	arXiv	Code
2026.3	SE-Search: Self-Evolving Search Agent via Memory and Dense Reward	Reasoning-Search Interaction; Context & Memory Management	arXiv
2026.4	IG-Search: Step-Level Information Gain Rewards for Search-Augmented Reasoning	Reasoning-Search Interaction; Process Supervision	arXiv
2026.4	OThink-SRR1: Search, Refine and Reasoning with Reinforced Learning for Large Language Models	Reasoning-Search Interaction; Search Efficiency	arXiv
2026.4	VISOR: Agentic Visual Retrieval-Augmented Generation via Iterative Search and Over-horizon Reasoning	Reasoning-Search Interaction; Multi-modal Search Efficiency	arXiv
2026.4	Enhancing LLM-based Search Agents via Contribution Weighted Group Relative Policy Optimization	Reasoning-Search Interaction; Retriever‑Aware Optimization	ACL 2026 (Main)
2025.10	Information Gain-based Policy Optimization: A Simple and Effective Approach for Multi-Turn Search Agents	Reasoning-Search Interaction; Process Supervision	ICLR 2026 Poster	Code
2026.2	SIGHT: Reinforcement Learning with Self-Evidence and Information-Gain Diverse Branching for Search Agent	Reasoning-Search Interaction; Search Efficiency	arXiv
2026.2	Search-R2: Enhancing Search-Integrated Reasoning via Actor-Refiner Collaboration	Reasoning-Search Interaction; Process Supervision	arXiv
2026.2	A-RAG: Scaling Agentic Retrieval-Augmented Generation via Hierarchical Retrieval Interfaces	Reasoning-Search Interaction; Hierarchical Retrieval	arXiv	Code
2026.2	Adaptive Information Control for Search-Augmented LLM Reasoning	Reasoning-Search Interaction; Information Control	arXiv
2026.2	Search-P1: Path-Centric Reward Shaping for Stable and Efficient Agentic RAG Training	Reasoning-Search Interaction; Reward Shaping	arXiv
2026.1	ProRAG: Process-Supervised Reinforcement Learning for Retrieval-Augmented Generation	Reasoning-Search Interaction; Process Supervision	arXiv	Code
2026.1	Chaining the Evidence: Robust Reinforcement Learning for Deep Search Agents with Citation-Aware Rubric Rewards	Reasoning-Search Interaction; Faithfulness / Citation Reward	arXiv	Code
2026.1	D$^2$Plan: Dual-Agent Dynamic Global Planning for Complex Retrieval-Augmented Reasoning	Reasoning-Search Interaction; Context Condensation (Purifier)	arXiv
2026.1	TreePS-RAG: Tree-based Process Supervision for Reinforcement Learning in Agentic RAG	Reasoning-Search Interaction; Process Supervision	arXiv
2025.11	TeaRAG: A Token-Efficient Agentic Retrieval-Augmented Generation Framework	Search Efficiency; Reasoning-Search Interaction	arXiv	Code
2025.10	GlobalRAG: Enhancing Global Reasoning in Multi-hop Question Answering via Reinforcement Learning	Reasoning-Search Interaction; Retriever‑Aware Optimization	arXiv	Code
2025.10	RECON: Reasoning with Condensation for Efficient Retrieval‑Augmented Generation	Context & Memory Management	arXiv
2025.10	Search Self-play: Pushing the Frontier of Agent Capability without Supervision	Adaptive Search Decision; Self-Play Training	arXiv	Code
2025.9	Fathom-DeepResearch: Unlocking Long Horizon Information Retrieval and Synthesis for SLMs	Reasoning-Search Interaction; Context & Memory Management; Multi-Tool	arXiv
2025.09	AceSearcher: Bootstrapping Reasoning and Search for LLMs via Reinforced Self-Play	Reasoning-Search Interaction; Self-Play Training	arXiv	Code
2025.09	ReSum: Unlocking Long‑Horizon Search Intelligence via Context Summarization	Context & Memory Management	arXiv
2025.09	SFR‑DeepResearch: Towards Effective Reinforcement Learning for Autonomously Reasoning Single Agents	Context & Memory Management; Multi‑Tool	arXiv
2025.08	Atom‑Searcher: Enhancing Agentic Deep Research via Fine‑Grained Atomic Thought Reward	Reasoning-Search Interaction	arXiv	Code
2025.08	Beyond Ten Turns: Unlocking Long‑Horizon Agentic Search with Large‑Scale Asynchronous RL	Reasoning-Search Interaction; Retriever‑Aware Optimization	arXiv
2025.07	DynaSearcher: Dynamic Knowledge Graph Augmented Search Agent via Multi‑Reward Reinforcement Learning	Structured Knowledge Navigation	arXiv
2025.07	WebSailor: Navigating Super‑Human Reasoning for Web Agent	Adaptive Search Decision; Context & Memory Management	arXiv
2025.06	R‑Search: Empowering LLM Reasoning with Search via Multi‑Reward Reinforcement Learning	Reasoning-Search Interaction	arXiv
2025.05	Search and Refine During Think: Facilitating Knowledge Refinement for Improved Retrieval-Augmented Reasoning	Reasoning-Search Interaction	NeurIPS 2025	Code
2025.05	EvolveSearch: An Iterative Self‑Evolving Search Agent	Reasoning-Search Interaction	arXiv
2025.05	O²‑Searcher: A Searching‑Based Agent Model for Open‑Domain Open‑Ended Question Answering	Reasoning-Search Interaction	arXiv	Code
2025.05	Process vs. Outcome Reward: Which Is Better for Agentic RAG Reinforcement Learning	Reasoning‑Search Interaction; Retriever‑Aware Optimization	NeurIPS 2025	Code
2025.04	DeepResearcher: Scaling Deep Research via Reinforcement Learning in Real‑World Environments	Context & Memory Management	arXiv
2025.04	Synthetic Data Generation & Multi-Step RL for Reasoning & Tool Use	Reasoning‑Search Interaction	arXiv

Multi‑Agent Collaboration

Time	Paper Title	Role	Venue	Code
2026.7	SearchOS-V1: Towards Robust Open-Domain Information-Seeking Agent Collaboration	Cooperative Multi-Agent Systems; Search-State Management	arXiv	Code
2026.7	WebSwarm: Recursive Multi-Agent Orchestration for Deep-and-Wide Web Search	Cooperative Multi-Agent Systems; Recursive Delegation	arXiv
2026.6	Hybrid Open-Ended Tri-Evolution Makes Better Deep Researcher	Cooperative Multi-Agent Systems; Proposer-Solver-Judge Co-Evolution	arXiv
2026.6	SearchSwarm: Towards Delegation Intelligence in Agentic LLMs for Long-Horizon Deep Research	Cooperative Multi-Agent Systems; Delegation Intelligence	arXiv
2026.4	Towards Knowledgeable Deep Research: Framework and Benchmark	Cooperative Multi-Agent Systems; Hybrid Knowledge Analysis	arXiv
2026.5	Towards Verifiable Multimodal Deep Research: A Multi-Agent Harness for Interleaved Report Generation	Cooperative Multi-Agent Systems; Multi‑Modal	arXiv
2026.5	AgentDisCo: Towards Disentanglement and Collaboration in Open-ended Deep Research Agents	Cooperative Multi-Agent Systems; Planner-Executor Orchestration	arXiv
2026.1	O-Researcher: An Open Ended Deep Research Model via Multi-Agent Distillation and Agentic RL	Cooperative Multi-Agent Systems; Multi‑Tool	arXiv
2026.5	Argus: Evidence Assembly for Scalable Deep Research Agents	Planner-Executor Orchestration; Evidence Assembly	arXiv
2026.1	PRISMA: Reinforcement Learning Guided Two-Stage Policy Optimization in Multi-Agent Architecture for Open-Domain Multi-Hop Question Answering	Planner-Executor Orchestration	arXiv
2026.2	WideSeek-R1: Exploring Width Scaling for Broad Information Seeking via Multi-Agent Reinforcement Learning	Cooperative Multi-Agent Systems	arXiv
2026.1	Beyond Monolithic Architectures: A Multi-Agent Search and Knowledge Optimization Framework for Agentic Search	Planner-Executor Orchestration; Context & Memory Management	arXiv	Code
2025.10	In-the-Flow Agentic System Optimization for Effective Planning and Tool Use	Planner-Executor Orchestration; Multi‑Tool	arXiv	Code Project page
2025.09	AgentGym‑RL: Training LLM Agents for Long‑Horizon Decision Making through Multi‑Turn Reinforcement Learning	Cooperative Multi-Agent Systems; Multi‑Tool	arXiv
2025.09	SIRAG: Towards Stable and Interpretable RAG with a Process‑Supervised Multi‑Agent Framework	Cooperative Multi-Agent Systems	arXiv
2025.09	WebExplorer: Explore and Evolve for Training Long‑Horizon Web Agents	Context & Memory Management; Convsational Reformulation	arXiv
2025.08	AI‑SearchPlanner: Modular Agentic Search via Pareto‑Optimal Multi‑Objective Reinforcement Learning	Planner-Executor Orchestration	arXiv
2025.08	Chain‑of‑Agents: End‑to‑End Agent Foundation Models via Multi‑Agent Distillation and Agentic RL	Cooperative Multi-Agent Systems; Multi‑Tool	arXiv
2025.08	MAO‑ARAG: Multi‑Agent Orchestration for Adaptive Retrieval‑Augmented Generation	Planner-Executor Orchestration	arXiv
2025.08	OPERA: A Reinforcement Learning–Enhanced Orchestrated Planner‑Executor Architecture for Reasoning‑Oriented Multi‑Hop Retrieval	Planner-Executor Orchestration	arXiv
2025.05	Advancing Multi-Agent RAG Systems with Minimalist Reinforcement Learning	Cooperative Multi-Agent Systems	arXiv
2025.01	Improving Retrieval‑Augmented Generation through Multi‑Agent Reinforcement Learning	Cooperative Multi-Agent Systems	NeurIPS 2025	Code

Tool and Knowledge Integration

Time	Paper Title	Role	Venue	Code
2026.7	SearchOS-V1: Towards Robust Open-Domain Information-Seeking Agent Collaboration	Multi-Tool; Search-State Middleware; Context & Memory Management	arXiv	Code
2026.7	PCTD: Preference-Guided Counterfactual Task Decomposition for Agent Tool Retrieval	Multi-Tool; Task Decomposition; Tool Retrieval	arXiv
2026.7	EvoGraph-R1: Self-Evolving Multimodal Knowledge Hypergraphs for Agentic Retrieval	Multi-Modal; Structured Knowledge Navigation; Web Search	CVPR 2026
2026.7	DeepSearch-World: Self-Distillation for Deep Search Agents in a Verifiable Environment	Search Environment; Page Reading; Task / Data Synthesis	arXiv
2026.7	VideoSearcher: Empowering Video Deep Research with Multi-Tool Agentic Reasoning via Reinforcement Learning	Multi-Tool; Multi-Modal; Video Deep Research	arXiv	Code
2026.4	Towards Knowledgeable Deep Research: Framework and Benchmark	Multi-Tool; Structured and Unstructured Knowledge	arXiv
2026.7	GRASP: GRanularity-Aware Search Policy for Agentic RAG	Multi-Tool; Retriever-Aware Optimization	arXiv
2026.6	DEEPRUBRIC: Evidence-Tree Rubric Supervision for Efficient Reinforcement Learning of Deep Research Agents	Task / Data Synthesis; Rubric-Based Agentic RL	arXiv
2026.6	S1-DeepResearch: Beyond Search, Toward Real-World Long-Horizon Research Agents	Task / Trajectory Synthesis; Long-Horizon Research Skills	arXiv
2026.6	MCompassRAG: Topic Metadata as a Semantic Compass for Paragraph-Level Retrieval	Retriever-Aware Optimization; Metadata-Guided Retrieval	arXiv	Code
2026.7	Bringing Agentic Search to Earth Observation Data Discovery	Domain-Specific; Knowledge Graph Search	arXiv
2026.7	BaRA: Budget-constrained and Reliable Web Data Collection Agent	Multi‑Tool; Web Data Collection	arXiv	Code
2026.7	Multi-Turn Agentic Scientific Literature Search via Workflow Induction	Multi‑Tool; Scientific Literature Search	arXiv
2026.6	Harness-1: Reinforcement Learning for Search Agents with State-Externalizing Harnesses	Stateful Search Harness; Context & Memory Management	arXiv	Code Model
2026.5	Towards Verifiable Multimodal Deep Research: A Multi-Agent Harness for Interleaved Report Generation	Multi‑Modal; Multi‑Tool; Verification	arXiv
2026.5	SciResearcher: Scaling Deep Research Agents for Frontier Scientific Reasoning	Multi‑Tool; Domain-Specific; Task / Data Synthesis	arXiv
2026.5	QUEST: Training Frontier Deep Research Agents with Fully Synthetic Tasks	Task / Data Synthesis for Deep Research	arXiv
2026.5	GrepSeek: Training Search Agents for Direct Corpus Interaction	Structured Knowledge Navigation; Direct Corpus Interaction	arXiv
2026.5	Towards On-Policy Data Evolution for Visual-Native Multimodal Deep Search Agents	Multi‑Modal; Task / Data Synthesis	arXiv
2026.4	MTA-Agent: An Open Recipe for Multimodal Deep Search Agents	Multi‑Modal; Multi‑Tool; Task / Data Synthesis	arXiv
2026.1	RAGShaper: Eliciting Sophisticated Agentic RAG Skills via Automated Data Synthesis	Task / Data Synthesis for Agentic RAG	arXiv
2026.1	O-Researcher: An Open Ended Deep Research Model via Multi-Agent Distillation and Agentic RL	Multi‑Agent Distillation; Agentic RL	arXiv
2026.5	HyperEyes: Dual-Grained Efficiency-Aware Reinforcement Learning for Parallel Multimodal Search Agents	Multi‑Tool; Multi‑Modal; Search Efficiency	arXiv	Code
2026.4	VISOR: Agentic Visual Retrieval-Augmented Generation via Iterative Search and Over-horizon Reasoning	Multi‑Tool; Multi‑Modal; Search Efficiency	arXiv
2026.5	RubricEM: Meta-RL with Rubric-guided Policy Decomposition beyond Verifiable Rewards	Context & Memory Management; Rubric-Guided Agentic RL	arXiv
2026.3	Marco DeepResearch: Unlocking Efficient Deep Research Agents via Verification-Centric Design	Task / Trajectory Synthesis; Verification-Centric Design	arXiv
2026.5	OpenSearch-VL: An Open Recipe for Frontier Multimodal Search Agents	Multi‑Tool; Multi‑Modal; Adaptive Search Decision	arXiv	Code
2026.5	Maximizing Rollout Informativeness under a Fixed Budget: A Submodular View of Tree Search for Tool-Use Agentic Reinforcement Learning	Multi‑Tool; Training Efficiency	arXiv
2026.5	BioMedArena: An Open-source Toolkit for Building and Evaluating Biomedical Deep Research Agents	Domain-Specific; Evaluation Toolkit	arXiv	Code
2026.3	SynPlanResearch-R1: Encouraging Tool Exploration for Deep Research with Synthetic Plans	Multi‑Tool; Adaptive Search Decision	arXiv	Code
2026.4	LiteResearcher: A Scalable Agentic RL Training Framework for Deep Research Agent	Unified Agentic RL Framework; Training Data / Environment Simulation	arXiv	Code
2026.4	DR-Venus: Towards Frontier Edge-Scale Deep Research Agents with Only 10K Open Data	Training Data / Open-Data Recipe for Deep Research Agents; Edge-Scale Deep Research	arXiv	Code Models
2026.4	ProMMSearchAgent: A Generalizable Multimodal Search Agent Trained with Process-Oriented Rewards	Multi‑Tool; Multi‑Modal	arXiv
2026.4	ORBIT: Scalable and Verifiable Data Generation for Search Agents on a Tight Budget	Task / Data Synthesis for Search Agents	arXiv
2026.4	Beyond Stochastic Exploration: What Makes Training Data Valuable for Agentic Search	Training Data / Experience Curation for Search Agents	arXiv
2026.3	VSearcher: Long-Horizon Multimodal Search Agent via Reinforcement Learning	Multi‑Tool; Multi‑Modal	arXiv
2026.3	MM-DeepResearch: A Simple and Effective Multimodal Agentic Search Baseline	Multi‑Tool; Multi‑Modal	arXiv	Code
2026.3	Improving Search Agent with One Line of Code	Training Stability (TARL / PPO)	arXiv
2026.3	ProtRLSearch: A Multi-Round Multimodal Protein Search Agent with Large Language Models Trained via Reinforcement Learning	Multi‑Tool; Multi‑Modal; Domain-Specific	arXiv
2026.2	A-RAG: Scaling Agentic Retrieval-Augmented Generation via Hierarchical Retrieval Interfaces	Multi‑Tool; Hierarchical Retrieval	arXiv	Code
2026.2	ARLArena: A Unified Framework for Stable Agentic Reinforcement Learning	Unified Agentic RL Framework; Stability	arXiv
2026.3	RAPO: Expanding Exploration for LLM Agents via Retrieval-Augmented Policy Optimization	Agentic RL Exploration; Retrieval-Augmented Policy Optimization	arXiv
2026.2	Reasoning and Tool-use Compete in Agentic RL: From Quantifying Interference to Disentangled Tuning	Agentic RL Training Dynamics (Reasoning–Tool Interference)	arXiv
2026.1	GraphSearch: Agentic Search-Augmented Reasoning for Zero-Shot Graph Learning	Structured Knowledge Navigation; Graph Search	arXiv
2026.1	OpenTinker: Separating Concerns in Agentic Reinforcement Learning	Unified Agentic RL Framework	arXiv
2026.1	SCRIBE: Structured Mid-Level Supervision for Tool-Using Language Models	Multi‑Tool; Mid‑Level Reward Modeling (Skill Prototypes)	arXiv
2026.1	AT$^2$PO: Agentic Turn-based Policy Optimization via Tree Search	Multi‑Turn Agentic RL; Credit Assignment	arXiv	Code
2026.1	PEARL: Plan Exploration and Adaptive Reinforcement Learning for Multihop Tool Use	Multi‑Tool; Planner‑Executor Orchestration	PRICAI 2025
2025.12	CARL: Focusing Agentic Reinforcement Learning on Critical Actions	Credit Assignment for Multi‑Step Tool Use	arXiv
2025.12	On Group Relative Policy Optimization Collapse in Agent Search: The Lazy Likelihood-Displacement	Training Stability (GRPO)	arXiv
2025.12	RouteRAG: Efficient Retrieval-Augmented Generation from Text and Graph via Reinforcement Learning	Structured Knowledge Navigation; Multi‑Tool	arXiv	Code
2025.12	CuES: A Curiosity-driven and Environment-grounded Synthesis Framework for Agentic RL	Task / Curriculum Synthesis for Agentic RL (tool environments)	arXiv	Code
2025.12	SenseNova-MARS: Empowering Multimodal Agentic Reasoning and Search via Reinforcement Learning	Multi‑Tool; Multi‑Modal	arXiv
2025.11	Agent‑R1: Training Powerful LLM Agents with End‑to‑End Reinforcement Learning	Multi‑Tool	arXiv	Code
2025.10	MARAG-R1: Beyond Single Retriever via Reinforcement-Learned Multi-Tool Agentic Retrieval	Multi‑Tool; Retriever‑Aware Optimization	arXiv
2025.09	Empowering LLM Tool Invocation with Tool-call Reward Model	Multi‑Tool	ICLR 2026 Submission
2025.09	VerlTool: Towards Holistic Agentic Reinforcement Learning with Tool Use	Multi‑Tool	arXiv	Code
2025.9	Fathom-DeepResearch: Unlocking Long Horizon Information Retrieval and Synthesis for SLMs	Multi‑Tool; Context & Memory Management	arXiv
2025.09	WebResearcher: Unleashing unbounded reasoning capability in Long‑Horizon Agents	Multi‑Tool	arXiv	Code
2025.08	GRAIL: Learning to Interact with Large Knowledge Graphs for Retrieval‑Augmented Reasoning	Adaptive Search Decision; Structured Knowledge Navigation	arXiv	Code
2025.08	MedResearcher‑R1: Expert‑Level Medical Deep Researcher via a Knowledge‑Informed Trajectory Synthesis Framework	Adaptive Search Decision; Multi‑Tool	arXiv	Code
2025.08	WebWatcher: Breaking New Frontier of Vision‑Language Deep Research Agent	Multi‑Tool; Multi‑Modal	arXiv
2025.07	DynaSearcher: Dynamic Knowledge Graph Augmented Search Agent via Multi‑Reward Reinforcement Learning	Structured Knowledge Navigation	arXiv
2025.06	MMSearch‑R1: Incentivizing LMMs to Search	Multi‑Modal; Search Efficiency	arXiv	Code
2025.05	VRAG‑RL: Empower Vision‑Perception‑Based RAG for Visually Rich Information Understanding via Iterative Reasoning with RL	Multi‑Modal; Search Efficiency	arXiv
2025.05	Tool‑Star: Empowering LLM‑Brained Multi‑Tool Reasoner via Reinforcement Learning	Multi‑Tool	arXiv	Code
2025.05	Visual Agentic Reinforcement Fine‑Tuning	Multi‑Modal; Multi‑Tool; Adaptive Search Decision	arXiv	Code

Where RL is Applied: Optimization Scopes

Agent-level

📊 Click to expand long table (scrollable).

Time	Paper Title	Role	Venue	Code
2026.7	Why Does Feedback-Augmented Self-Distillation Fail to Improve Retrieval-Interleaved Search Agents?	Single-agent Optimization	arXiv
2026.7	SearchOS-V1: Towards Robust Open-Domain Information-Seeking Agent Collaboration	Multi-agent Coordination	arXiv	Code
2026.7	Analogical Deep Research: Retrieving and Integrating Historical Analogies for Foresight Analysis	Single-agent Optimization	arXiv
2026.7	EvoGraph-R1: Self-Evolving Multimodal Knowledge Hypergraphs for Agentic Retrieval	Single-agent Optimization	CVPR 2026
2026.7	WebSwarm: Recursive Multi-Agent Orchestration for Deep-and-Wide Web Search	Multi-agent Coordination	arXiv
2026.7	DeepSearch-World: Self-Distillation for Deep Search Agents in a Verifiable Environment	Single-agent Optimization	arXiv
2026.7	VideoSearcher: Empowering Video Deep Research with Multi-Tool Agentic Reasoning via Reinforcement Learning	Single-agent Optimization	arXiv	Code
2026.6	Hybrid Open-Ended Tri-Evolution Makes Better Deep Researcher	Multi-agent Coordination	arXiv
2026.6	SearchSwarm: Towards Delegation Intelligence in Agentic LLMs for Long-Horizon Deep Research	Multi-agent Coordination	arXiv
2026.4	Towards Knowledgeable Deep Research: Framework and Benchmark	Multi-agent Coordination	arXiv
2026.7	LAPO: Leave-One-Turn Attribution for Self-Generated Process Rewards in Multi-Turn Search Reasoning	Single-agent Optimization	arXiv
2026.7	To Answer or to Abstain: Mitigating Search-Agent Hallucinations via Abstention-Aware Reinforcement Learning	Single-agent Optimization	arXiv	Code
2026.7	GRASP: GRanularity-Aware Search Policy for Agentic RAG	Single-agent Optimization	arXiv
2026.6	DEEPRUBRIC: Evidence-Tree Rubric Supervision for Efficient Reinforcement Learning of Deep Research Agents	Single-agent Optimization	arXiv
2026.6	ScaffoldAgent: Utility-Guided Dynamic Outline Optimization for Open-Ended Deep Research	Single-agent Optimization	arXiv
2026.6	S1-DeepResearch: Beyond Search, Toward Real-World Long-Horizon Research Agents	Single-agent Optimization	arXiv
2026.7	Libra: Training the Environment for Agentic Information Retrieval	Single-agent Optimization	arXiv	Code Data
2026.7	Multi-Turn Agentic Scientific Literature Search via Workflow Induction	Single-agent Optimization	arXiv
2026.7	BaRA: Budget-constrained and Reliable Web Data Collection Agent	Single-agent Optimization	arXiv	Code
2026.7	Bringing Agentic Search to Earth Observation Data Discovery	Single-agent Optimization	arXiv
2026.6	Harness-1: Reinforcement Learning for Search Agents with State-Externalizing Harnesses	Single-agent Optimization	arXiv	Code Model
2026.6	Effective Reinforcement Learning for Agentic Search by Recycling Zero-Variance Queries During Training	Single-agent Optimization	arXiv
2026.5	SAAS: Self-Aware Reinforcement Learning for Over-Search Mitigation in Agentic Search	Single-agent Optimization	arXiv	Code
2026.5	SD-Search: On-Policy Hindsight Self-Distillation for Search-Augmented Reasoning	Single-agent Optimization	arXiv
2026.5	Search-E1: Self-Distillation Drives Self-Evolution in Search-Augmented Reasoning	Single-agent Optimization	arXiv
2026.5	GrepSeek: Training Search Agents for Direct Corpus Interaction	Single-agent Optimization	arXiv
2026.5	QUEST: Training Frontier Deep Research Agents with Fully Synthetic Tasks	Single-agent Optimization	arXiv
2026.5	SciResearcher: Scaling Deep Research Agents for Frontier Scientific Reasoning	Single-agent Optimization	arXiv
2026.5	Towards On-Policy Data Evolution for Visual-Native Multimodal Deep Search Agents	Single-agent Optimization	arXiv
2026.4	Negative Advantages Is a Double-Edged Sword: Calibrating advantages in GRPO for Search Agents	Single-agent Optimization	arXiv	Code
2026.4	OThink-SRR1: Search, Refine and Reasoning with Reinforced Learning for Large Language Models	Single-agent Optimization	arXiv
2026.4	VISOR: Agentic Visual Retrieval-Augmented Generation via Iterative Search and Over-horizon Reasoning	Single-agent Optimization	arXiv
2026.4	MTA-Agent: An Open Recipe for Multimodal Deep Search Agents	Single-agent Optimization	arXiv
2026.1	O-Researcher: An Open Ended Deep Research Model via Multi-Agent Distillation and Agentic RL	Single-agent Optimization	arXiv
2026.1	RAGShaper: Eliciting Sophisticated Agentic RAG Skills via Automated Data Synthesis	Single-agent Optimization	arXiv
2026.5	Argus: Evidence Assembly for Scalable Deep Research Agents	Multi-agent Coordination	arXiv
2026.5	CuSearch: Curriculum Rollout Sampling via Search Depth for Agentic RAG	Single-agent Optimization	arXiv	Code
2026.5	HyperEyes: Dual-Grained Efficiency-Aware Reinforcement Learning for Parallel Multimodal Search Agents	Single-agent Optimization	arXiv	Code
2026.5	RubricEM: Meta-RL with Rubric-guided Policy Decomposition beyond Verifiable Rewards	Single-agent Optimization	arXiv
2026.5	LatentRAG: Latent Reasoning and Retrieval for Efficient Agentic RAG	Single-agent Optimization	arXiv
2026.3	Marco DeepResearch: Unlocking Efficient Deep Research Agents via Verification-Centric Design	Single-agent Optimization	arXiv
2025.10	DecEx-RAG: Boosting Agentic Retrieval-Augmented Generation with Decision and Execution Optimization via Process Supervision	Single-agent Optimization	arXiv	Code
2026.5	OpenSearch-VL: An Open Recipe for Frontier Multimodal Search Agents	Single-agent Optimization	arXiv	Code
2026.5	OpenSeeker-v2: Pushing the Limits of Search Agents with Informative and High-Difficulty Trajectories	Single-agent Optimization	arXiv
2026.5	LongSeeker: Elastic Context Orchestration for Long-Horizon Search Agents	Single-agent Optimization	arXiv
2026.4	Search, Do not Guess: Teaching Small Language Models to Be Effective Search Agents	Single-agent Optimization	arXiv
2026.1	Deep Search with Hierarchical Meta-Cognitive Monitoring Inspired by Cognitive Neuroscience	Single-agent Optimization	arXiv
2026.3	APEX-Searcher: Augmenting LLMs' Search Capabilities through Agentic Planning and Execution	Single-agent Optimization	arXiv
2026.3	Meta-Reinforcement Learning with Self-Reflection for Agentic Search	Single-agent Optimization	arXiv	Code
2026.2	Search More, Think Less: Rethinking Long-Horizon Agentic Search for Efficiency and Generalization	Single-agent Optimization	arXiv	Code
2026.3	KARL: Knowledge Agents via Reinforcement Learning	Single-agent Optimization	arXiv
2026.4	LiteResearcher: A Scalable Agentic RL Training Framework for Deep Research Agent	Single-agent Optimization	arXiv	Code
2026.4	AutoSearch: Adaptive Search Depth for Efficient Agentic RAG via Reinforcement Learning	Single-agent Optimization	arXiv	Code
2026.4	DR-Venus: Towards Frontier Edge-Scale Deep Research Agents with Only 10K Open Data	Single-agent Optimization	arXiv	Code Models
2026.4	ProMMSearchAgent: A Generalizable Multimodal Search Agent Trained with Process-Oriented Rewards	Single-agent Optimization	arXiv
2026.3	SynPlanResearch-R1: Encouraging Tool Exploration for Deep Research with Synthetic Plans	Single-agent Optimization	arXiv	Code
2026.3	VSearcher: Long-Horizon Multimodal Search Agent via Reinforcement Learning	Single-agent Optimization	arXiv
2026.3	MM-DeepResearch: A Simple and Effective Multimodal Agentic Search Baseline	Single-agent Optimization	arXiv	Code
2026.3	ProtRLSearch: A Multi-Round Multimodal Protein Search Agent with Large Language Models Trained via Reinforcement Learning	Single-agent Optimization	arXiv
2026.3	RAPO: Expanding Exploration for LLM Agents via Retrieval-Augmented Policy Optimization	Single-agent Optimization	arXiv
2026.2	SIGHT: Reinforcement Learning with Self-Evidence and Information-Gain Diverse Branching for Search Agent	Single-agent Optimization	arXiv
2026.2	Search-R2: Enhancing Search-Integrated Reasoning via Actor-Refiner Collaboration	Single-agent Optimization	arXiv
2026.2	A-RAG: Scaling Agentic Retrieval-Augmented Generation via Hierarchical Retrieval Interfaces	Single-agent Optimization	arXiv	Code
2026.2	Adaptive Information Control for Search-Augmented LLM Reasoning	Single-agent Optimization	arXiv
2026.1	SearchGym: Bootstrapping Real-World Search Agents via Cost-Effective and High-Fidelity Environment Simulation	Single-agent Optimization	arXiv	Code
2026.1	GraphSearch: Agentic Search-Augmented Reasoning for Zero-Shot Graph Learning	Single-agent Optimization	arXiv
2025.11	Thinker: Training LLMs in Hierarchical Thinking for Deep Search via Multi-Turn Interaction	Single-agent Optimization	AAAI 2026	Code
2025.11	MemSearcher: Training LLMs to Reason, Search and Manage Memory via End-to-End Reinforcement Learning	Single-agent Optimization	arXiv	Code
2025.10	Beneficial Reasoning Behaviors in Agentic Search and Effective Post-training to Obtain Them	Single-agent Optimization	arXiv	Code
2025.9	Fathom-DeepResearch: Unlocking Long Horizon Information Retrieval and Synthesis for SLMs	Single-agent Optimization	arXiv
2025.09	DeepDive: Advancing Deep Search Agents with Knowledge Graphs and Multi-Turn RL	Single-agent Optimization	arXiv	Code
2025.09	SFR-DeepResearch: Towards Effective Reinforcement Learning for Autonomously Reasoning Single Agents	Single-agent Optimization	arXiv
2025.09	WebExplorer: Explore and evolve for training long-horizon web agents	Single-agent Optimization	arXiv
2025.09	WebResearcher: Unleashing unbounded reasoning capability in Long-Horizon Agents	Single-agent Optimization	arXiv
2025.09	AgentGym-RL: Training LLM Agents for Long-Horizon Decision Making through Multi-Turn Reinforcement Learning	Single-agent Optimization	arXiv
2025.09	WebSailor-V2: Bridging the Chasm to Proprietary Agents via Synthetic Data and Scalable Reinforcement Learning	Single-agent Optimization	arXiv	Code
2025.6	MMSearch-R1: Incentivizing LMMs to Search	Single-agent Optimization	arXiv	Code
2025.08	MedResearcher-R1: Expert-Level Medical Deep Researcher via A Knowledge-Informed Trajectory Synthesis Framework	Single-agent Optimization	arXiv	Code
2025.08	WebWatcher: Breaking New Frontier of Vision-Language Deep Research Agent	Single-agent Optimization	arXiv
2025.08	GRAIL: Learning to interact with large knowledge graphs for retrieval-augmented reasoning	Single-agent Optimization	arXiv	Code
2025.5	VRAG-RL: Empower Vision-Perception-Based RAG for Visually Rich Information Understanding via Iterative Reasoning with Reinforcement Learning	Single-agent Optimization	arXiv
2025.5	Visual Agentic Reinforcement Fine-Tuning	Single-agent Optimization	arXiv	Code
2025.7	WebSailor: Navigating Super-Human Reasoning for Web Agent	Single-agent Optimization	arXiv
2025.08	ParallelSearch: Train your LLMs to Decompose Query and Search Sub-queries in Parallel with Reinforcement Learning	Single-agent Optimization	arXiv
2025.08	UR2: Unify RAG and Reasoning through Reinforcement Learning	Single-agent Optimization	arXiv	Code
2025.08	SSRL: Self-Search Reinforcement Learning	Single-agent Optimization	arXiv	Code
2025.7	DynaSearcher: Dynamic Knowledge Graph Augmented Search Agent via Multi-Reward Reinforcement Learning	Single-agent Optimization	arXiv
2025.6	R-Search: Empowering LLM Reasoning with Search via Multi-Reward Reinforcement Learning	Single-agent Optimization	arXiv
2025.5	EvolveSearch: An Iterative Self-Evolving Search Agent	Single-agent Optimization	arXiv
2025.5	O2-Searcher: a searching-based agent model for open-domain open-ended question answering	Single-agent Optimization	arXiv	Code
2025.5	Pangu DeepDiver: Adaptive Search Intensity Scaling via Open-Web Reinforcement Learning	Single-agent Optimization	NeurIPS 2025 (Spotlight)
2025.05	Reinforced Internal-External Knowledge Synergistic Reasoning for Efficient Adaptive Search Agent	Single-agent Optimization	arXiv
2025.5	ZeroSearch: Incentivize the Search Capability of LLMs without Searching	Single-agent Optimization	arXiv	Code
2025.5	MaskSearch: A Universal Pre-Training Framework to Enhance Agentic Search Capability	Single-agent Optimization	arXiv
2025.4	ReZero: Enhancing LLM Search Ability by Trying One-More-Time	Single-agent Optimization	arXiv	Code
2025.5	Tool-Star: Empowering LLM-Brained Multi-Tool Reasoner via Reinforcement Learning	Single-agent Optimization	arXiv	Code
2025.3	DeepRetrieval: Hacking Real Search Engines and Retrievers with Large Language Models via Reinforcement Learning	Single-agent Optimization	COLM 2025	Code
2025.4	WebThinker: Empowering Large Reasoning Models with Deep Research Capability	Single-agent Optimization	NeurIPS 2025	Code
2025.05	Search Wisely: Mitigating Sub-optimal Agentic Searches By Reducing Uncertainty	Single-agent Optimization	EMNLP 2025 (Main)
2025.4	DeepResearcher: Scaling Deep Research via Reinforcement Learning in Real-world Environments	Single-agent Optimization	arXiv	Code
2025.3	Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning	Single-agent Optimization	COLM 2025	Code
2025.3	ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learning	Single-agent Optimization	NeurIPS 2025
2025.05	R1-Searcher++: Incentivizing the dynamic knowledge acquisition of llms via reinforcement learning	Single-agent Optimization	arXiv	Code
2025.3	Agent models: Internalizing Chain-of-Action Generation into Reasoning models	Single-agent Optimization	arXiv	Code
2025.08	Lucy: edgerunning agentic web search on mobile with machine generated task vectors	Single-agent Optimization	arXiv
2025.2	DeepRAG: Thinking to Retrieve Step by Step for Large Language Models	Single-agent Optimization	arXiv
2025.5	StepSearch: Igniting LLMs Search Ability via Step-Wise Proximal Policy Optimization	Single-agent Optimization	EMNLP 2025 (Main)	Code
2025.1	Improving Retrieval-Augmented Generation through Multi-Agent Reinforcement Learning	Multi-agent Coordination	NeurIPS 2025	Code
2025.06	Coordinating Search-Informed Reasoning and Reasoning-Guided Search in Claim Verification	Multi-agent Coordination	arXiv
2025.9	SIRAG: Towards Stable and Interpretable RAG with a Process-Supervised Multi-Agent Framework	Multi-agent Coordination	arXiv
2026.2	WideSeek-R1: Exploring Width Scaling for Broad Information Seeking via Multi-Agent Reinforcement Learning	Multi-agent Coordination	arXiv
2026.1	Beyond Monolithic Architectures: A Multi-Agent Search and Knowledge Optimization Framework for Agentic Search	Multi-agent Coordination	arXiv	Code
2025.8	MAO-ARAG: Multi-Agent Orchestration for Adaptive Retrieval-Augmented Generation	Multi-agent Coordination	arXiv
2025.1	MMOA-RAG: Improving Retrieval-Augmented Generation through Multi-Agent Reinforcement Learning	Multi-agent Coordination	NeurIPS 2025	Code
2025.8	OPERA: A Reinforcement Learning–Enhanced Orchestrated Planner-Executor Architecture for Reasoning-Oriented Multi-Hop Retrieval	Multi-agent Coordination	arXiv
2025.02	DeepRAG: Thinking to Retrieval Step by Step for Large Language Models	Single‑agent Optimization	arXiv
2025.01	Improving Retrieval‑Augmented Generation through Multi‑Agent Reinforcement Learning	Multi‑agent Coordination	NeurIPS 2025	Code
2024.10	DeepNote: Note-Centric Deep Retrieval-Augmented Generation	Single‑agent Optimization	EMNLP 2025 (Findings)

Step-level

📊 Click to expand long table (scrollable).

Time	Paper Title	Role	Venue	Code
2026.7	LAPO: Leave-One-Turn Attribution for Self-Generated Process Rewards in Multi-Turn Search Reasoning	Step-level Optimization	arXiv
2026.7	Evidence-State Rewards for Long-Context Reasoning	Step‑level Optimization	arXiv
2026.5	SD-Search: On-Policy Hindsight Self-Distillation for Search-Augmented Reasoning	Step‑level Optimization	arXiv
2026.5	Search-E1: Self-Distillation Drives Self-Evolution in Search-Augmented Reasoning	Step‑level Optimization	arXiv
2026.4	Negative Advantages Is a Double-Edged Sword: Calibrating advantages in GRPO for Search Agents	Step‑level Optimization	arXiv	Code
2025.10	DecEx-RAG: Boosting Agentic Retrieval-Augmented Generation with Decision and Execution Optimization via Process Supervision	Step‑level Optimization	arXiv	Code
2026.5	A$^2$TGPO: Agentic Turn-Group Policy Optimization with Adaptive Turn-level Clipping	Step‑level Optimization	arXiv
2026.5	Maximizing Rollout Informativeness under a Fixed Budget: A Submodular View of Tree Search for Tool-Use Agentic Reinforcement Learning	Step‑level Optimization	arXiv
2026.3	Evaluate-as-Action: Self-Evaluated Process Rewards for Retrieval-Augmented Agents	Step‑level Optimization	arXiv
2026.4	IG-Search: Step-Level Information Gain Rewards for Search-Augmented Reasoning	Step‑level Optimization	arXiv
2026.4	OThink-SRR1: Search, Refine and Reasoning with Reinforced Learning for Large Language Models	Step‑level Optimization	arXiv
2026.4	Enhancing LLM-based Search Agents via Contribution Weighted Group Relative Policy Optimization	Step‑level Optimization	ACL 2026 (Main)
2025.10	Information Gain-based Policy Optimization: A Simple and Effective Approach for Multi-Turn Search Agents	Step‑level Optimization	ICLR 2026 Poster	Code
2026.4	PRAISE: Prefix-Based Rollout Reuse in Agentic Search Training	Step‑level Optimization	arXiv
2026.2	Search-P1: Path-Centric Reward Shaping for Stable and Efficient Agentic RAG Training	Step‑level Optimization	arXiv
2026.2	Adaptive Information Control for Search-Augmented LLM Reasoning	Step‑level Optimization	arXiv
2025.10	Beyond Correctness: Rewarding Faithful Reasoning in Retrieval‑Augmented Generation	Step‑level Optimization	arXiv
2025.08	Atom‑Searcher: Enhancing Agentic Deep Research via Fine‑Grained Atomic Thought Reward	Step‑level Optimization	arXiv	Code
2025.05	StepSearch: Igniting LLMs Search Ability via Step-Wise Proximal Policy Optimization	Step‑level Optimization	EMNLP 2025	Code
2025.05	Process vs. Outcome Reward: Which Is Better for Agentic RAG Reinforcement Learning	Step‑level Optimization	NeurIPS 2025	Code
2025.05	Search and Refine During Think: Facilitating Knowledge Refinement for Improved Retrieval-Augmented Reasoning	Step‑level Optimization	NeurIPS 2025	Code
2025.05	ConvSearch‑R1: Enhancing Query Reformulation for Conversational Search with Reasoning via Reinforcement Learning	Step‑level Optimization	arXiv
2025.05	Reinforced Internal‑External Knowledge Synergistic Reasoning for Efficient Adaptive Search Agent	Step‑level Optimization	arXiv
2025.04	ReZero: Enhancing LLM Search Ability by Trying One‑More‑Time	Step‑level Optimization	arXiv	Code
2025.04	Synthetic Data Generation & Multi-Step RL for Reasoning & Tool Use	Step‑level Optimization	arXiv

Module-level

📊 Click to expand long table (scrollable).

Time	Paper Title	Role	Venue	Code
2026.7	PCTD: Preference-Guided Counterfactual Task Decomposition for Agent Tool Retrieval	Module-level Optimization	arXiv
2025.08	AI-SearchPlanner: Modular Agentic Search via Pareto-Optimal Multi-Objective Reinforcement Learning	Module-level Optimization	arXiv
2025.05	s3: You Don’t Need That Much Data to Train a Search Agent via RL	Module-level Optimization	arXiv
2025.04	Deepresearcher: Scaling deep research via reinforcement learning in real-world environments	Module-level Optimization	arXiv	Code

System-level

📊 Click to expand long table (scrollable).

Time	Paper Title	Role	Venue	Code
2026.7	SearchOS-V1: Towards Robust Open-Domain Information-Seeking Agent Collaboration	System-level Multi-Agent Search Framework	arXiv	Code
2026.7	WebSwarm: Recursive Multi-Agent Orchestration for Deep-and-Wide Web Search	Recursive Multi-Agent Search Framework	arXiv
2026.7	DeepSearch-World: Self-Distillation for Deep Search Agents in a Verifiable Environment	Verifiable Search Environment / Self-Distillation Framework	arXiv
2026.7	Libra: Training the Environment for Agentic Information Retrieval	Environment-level Agentic IR Framework	arXiv	Code Data
2026.6	Harness-1: Reinforcement Learning for Search Agents with State-Externalizing Harnesses	Stateful Search Harness / RL Search Agent	arXiv	Code Model
2026.6	Agentic Transformers Provably Learn to Search via Reinforcement Learning	Theoretical RL Search Behavior	arXiv
2025.09	AgentGym‑RL: Training LLM Agents for Long‑Horizon Decision Making through Multi‑Turn Reinforcement Learning	Unified RL‑based Agentic Framework	arXiv
2025.09	VerlTool: Towards Holistic Agentic Reinforcement Learning with Tool Use	Unified RL‑based Agentic Framework	arXiv
2025.08	Chain‑of‑Agents: End‑to‑End Agent Foundation Models via Multi‑Agent Distillation and Agentic RL	Unified RL‑based Agentic Framework	arXiv
2025.02	RAG‑Gym: Systematic Optimization of Language Agents for Retrieval‑Augmented Generation	Unified RL‑based Agentic Framework	arXiv
2024.09	HybridFlow: A Flexible and Efficient RLHF Framework	Unified RL‑based Agentic Framework (Verl)	arXiv

Evaluation

Metrics

Below is a compact checklist of commonly used evaluation signals for RL-based agentic search (see the survey for details):

• Answer quality (final output): Exact Match (EM), token-level F1, and LLM-as-a-judge scores for open-ended outputs (e.g., used by Search-R1, ReZero).
  • Similarity-based metrics (e.g., BERTScore) are also used for more free-form generations.
• Search effectiveness (retrieval / evidence): retrieval relevance/coverage metrics (e.g., Recall@k / Precision@k / nDCG) and evidence usefulness; some works explicitly optimize or evaluate retrieval behavior (e.g., DeepRetrieval).
• Search efficiency (cost / budget): number of search/tool calls, token/latency cost, and other explicit penalties/rewards (e.g., Pangu DeepDiver, MAO-ARAG, R1-Searcher++).
• Process / trajectory quality (intermediate steps): step-wise correctness, information gain, redundancy penalties, and other process-level signals (often combined with a PRM or heuristic shaping; e.g., StepSearch, VERITAS, ReasonRAG, ConvSearch-R1).
• System-level benchmarks: some benchmarks are designed specifically to evaluate deep-research style agents (e.g., RAG-Gym).

Recent Evaluation / Benchmarking Papers (Deep Research / Agentic Search)

Time	Paper Title	Focus	Venue
2026.7	WorldCupArena: Fine-Grained Evaluation of Language Models and Deep-Research Agents on Football Forecasting (Code)	Dynamic Football Forecasting Benchmark for LLMs and Deep-Research Agents	arXiv
2026.7	DRNOISE: Benchmarking Deep Research Agents in Misleading Evidence Environments	Misleading-Evidence Robustness Benchmark	arXiv
2026.7	PCTD: Preference-Guided Counterfactual Task Decomposition for Agent Tool Retrieval	Mobile Multi-Turn Task-Decomposition / Tool-Retrieval Benchmark (MTDTool)	arXiv
2026.7	Bridge Evidence: Static Retrieval Utility Does Not Predict Causal Utility in Multi-Step Agentic Search	Counterfactual Retrieval-Utility Analysis for Multi-Step Agentic Search	arXiv
2026.7	FinResearchBench II: A Deep Research Benchmark with Consensus-Derived Gold Rubrics for Distinguishing Financial Report Quality	Consensus-Derived Rubric Benchmark for Financial Deep Research	arXiv
2026.7	DeepSearch-World: Self-Distillation for Deep Search Agents in a Verifiable Environment	Deterministic Verifiable Search Environment with 420K Multi-Hop QA Tasks	arXiv
2026.7	FORGE: Research-Trajectory Hijacking Attacks on Deep Research Agents (Code)	Research-Trajectory Hijacking / Planning-Layer Poisoning	arXiv
2026.7	VideoSearcher: Empowering Video Deep Research with Multi-Tool Agentic Reasoning via Reinforcement Learning (Code)	Video Deep Research Benchmark (VideoSearch-QA)	arXiv
2026.7	KidnapRAG: A Black-Box Attack for Hijacking Reasoning in Agentic Retrieval-Augmented Generation Systems (Code)	Black-Box Sequential Poisoning of Agentic RAG	arXiv
2026.7	Analogical Deep Research: Retrieving and Integrating Historical Analogies for Foresight Analysis	Historical-Analogy Retrieval and Integration Benchmark (ADR-bench)	arXiv
2026.6	Can LLM-as-a-Judge Reliably Verify Rubrics in Agentic Scenarios? (Code)	Rubric-Verification Meta-Evaluation Benchmark (RuVerBench)	arXiv
2026.6	Multi-Turn Evaluation of Deep Research Agents Under Process-Level Feedback (Code)	Process-Level Feedback Evaluation for Deep Research Agents	SCALE @ ICML 2026 (Oral)
2026.6	Can AI Agents Synthesize Scientific Conclusions?	Scientific Conclusion-Synthesis Benchmark (SciConBench)	arXiv
2026.4	Towards Knowledgeable Deep Research: Framework and Benchmark	Structured + Unstructured Knowledge Benchmark (KDR-Bench)	arXiv
2025.7	ResearcherBench: Evaluating Deep AI Research Systems on the Frontiers of Scientific Inquiry (Code)	Frontier Scientific Inquiry Benchmark	arXiv
2026.7	DeepStress: Stress-Testing Deep Search Agents	Controlled Stress Testing under Unreliable Evidence	arXiv
2026.7	On-Device Deep Research at 4B: Exposure Bounds Faithfulness, Retrieval Bounds Coverage	Citation Faithfulness / Trustworthy Coverage on Edge-Scale Agents	arXiv
2026.7	HETERQA: Benchmarking Record Retrieval over Multiple Heterogeneous Sources (Code, Data)	Heterogeneous-Source Record Retrieval Benchmark	arXiv
2026.6	Breaking the Evaluation Paradox: Evaluating High-Entropy Search with Computationally Irreducible Constraints	VERITAS: Verifiable Exhaustive-Search Evaluation	ACL 2026
2026.6	Beyond Monolingual Deep Research: Evaluating Agents and Retrievers with Cross-Lingual BrowseComp-Plus	Cross-Lingual / Multilingual Deep-Research Benchmark (XBCP)	arXiv
2026.6	DRFLOW: A Deep Research Benchmark for Personalized Workflow Prediction	Personalized Workflow Prediction from Heterogeneous Sources	arXiv
2026.6	ICBCBench: An Industry Consortium Benchmark for Financial Deep Research (Code)	Objective + Long-Form Financial Deep-Research Evaluation	arXiv
2026.4	AutoResearchBench: Benchmarking AI Agents on Complex Scientific Literature Discovery (Code)	Deep / Wide Scientific Literature Discovery Benchmark	arXiv
2026.6	Deep Research in Physical Sciences: A Multi-Agent Framework and Comprehensive Benchmark	Physical-Science Deep Research Benchmark (PhySciBench) + DelveAgent	arXiv
2026.6	Search-Time Contamination in Deep Research Agents: Measuring Performance Inflation in Public Benchmark Evaluation	Search-Time Contamination / Benchmark Leakage Evaluation	arXiv
2026.5	DeepWeb-Bench: A Deep Research Benchmark Demanding Massive Cross-Source Evidence and Long-Horizon Derivation	Cross-Source Evidence + Long-Horizon Derivation Benchmark	arXiv
2026.4	Deep FinResearch Bench: Evaluating AI's Ability to Conduct Professional Financial Investment Research	Professional Financial Investment Research Benchmark	arXiv
2025.6	Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judge	Long-Horizon Agentic Search Benchmark + Agent-as-a-Judge	arXiv
2026.7	Bringing Agentic Search to Earth Observation Data Discovery	Earth Observation Search Benchmark (NASA-EO-Bench)	arXiv
2026.7	When RAG Meets Query Planning: Logical Query Trees for Resolving Exploratory Reasoning Problems	Exploratory Reasoning Query-Planning Dataset (WikiWeb-ERP)	SIGMOD 2027
2026.5	Towards Verifiable Multimodal Deep Research: A Multi-Agent Harness for Interleaved Report Generation	Multimodal Deep Research Harness + PtahEval	arXiv
2026.5	AgentDisCo: Towards Disentanglement and Collaboration in Open-ended Deep Research Agents	Open-ended Deep Research Architecture + GALA Benchmark	arXiv
2026.5	Evaluating Deep Research Agents on Expert Consulting Work: A Benchmark with Verifiers, Rubrics, and Cognitive Traps	Expert Consulting Work Benchmark	arXiv
2026.5	Deep-Research Agents Can Be Poisoned via User-Generated Content	Safety / Poisoning Evaluation for Deep Research Agents	arXiv
2026.1	FinDeepForecast: A Live Multi-Agent System for Benchmarking Deep Research Agents in Financial Forecasting	Live Financial Forecasting Benchmark	arXiv
2025.10	A Rigorous Benchmark with Multidimensional Evaluation for Deep Research Agents: From Answers to Reports	Multidimensional Report Evaluation Benchmark	arXiv
2025.6	Deep Research Bench: Evaluating AI Web Research Agents	Frozen-Web RetroSearch Evaluation	arXiv
2026.5	InterLV-Search: Benchmarking Interleaved Multimodal Agentic Search	Interleaved Language-Vision Agentic Search Benchmark	arXiv
2026.5	HyperEyes: Dual-Grained Efficiency-Aware Reinforcement Learning for Parallel Multimodal Search Agents	Parallel Multimodal Search + IMEB Efficiency Benchmark	arXiv
2025.10	Agentic Reinforcement Learning for Search is Unsafe	Safety Evaluation for RL-Trained Search Agents	arXiv
2026.5	BioMedArena: An Open-source Toolkit for Building and Evaluating Biomedical Deep Research Agents	Biomedical Deep Research Evaluation Toolkit	arXiv
2026.5	Rethinking Reasoning-Intensive Retrieval: Evaluating and Advancing Retrievers in Agentic Search Systems	Reasoning-Intensive Retrieval Benchmark (BRIGHT-Pro)	ACL 2026
2026.5	Cited but Not Verified: Parsing and Evaluating Source Attribution in LLM Deep Research Agents	Source Attribution / Citation Faithfulness Evaluation	arXiv
2026.5	Partial Evidence Bench: Benchmarking Authorization-Limited Evidence in Agentic Systems	Authorization-Limited Evidence Benchmark	arXiv
2026.4	DR$^3$-Eval: Towards Realistic and Reproducible Deep Research Evaluation	Realistic / Reproducible Multimodal Evaluation Framework	arXiv
2026.4	PaperScope: A Multi-Modal Multi-Document Benchmark for Agentic Deep Research Across Massive Scientific Papers	Multi-Modal Multi-Document Scientific Deep Research Benchmark	arXiv
2026.3	VisBrowse-Bench: Benchmarking Visual-Native Search for Multimodal Browsing Agents	Visual-Native Multimodal Browsing Benchmark	CVPR 2026
2026.2	Vision-DeepResearch Benchmark: Rethinking Visual and Textual Search for Multimodal Large Language Models	Visual + Textual Search Benchmark for Multimodal Deep Research	arXiv
2026.1	MMDeepResearch-Bench: A Benchmark for Multimodal Deep Research Agents	Multimodal Deep Research Benchmark	arXiv
2026.3	MiroEval: Benchmarking Multimodal Deep Research Agents in Process and Outcome	Multimodal Deep Research Benchmark + Process/Outcome Evaluation	arXiv
2026.3	Evaluating the Search Agent in a Parallel World	Parallel-World Interactive Benchmark (MPW-Bench)	arXiv
2026.3	Total Recall QA: A Verifiable Evaluation Suite for Deep Research Agents	Verifiable Total-Recall QA Benchmark	arXiv
2026.3	LongVidSearch: An Agentic Benchmark for Multi-hop Evidence Retrieval Planning in Long Videos	Agentic Multi-hop Video Search Benchmark	arXiv
2026.3	DeepFact: Co-Evolving Benchmarks and Agents for Deep Research Factuality	Claim-Level Factuality Benchmark + Verifier	arXiv
2026.3	KARL: Knowledge Agents via Reinforcement Learning	Multi-Capability Grounded Reasoning Benchmark Suite (KARLBench)	arXiv
2026.3	DeepResearch-9K: A Challenging Benchmark Dataset of Deep-Research Agent	Benchmark Dataset + Open Training Framework	arXiv
2026.2	DRACO: a Cross-Domain Benchmark for Deep Research Accuracy, Completeness, and Objectivity	Cross-Domain Deep Research Benchmark	arXiv
2026.2	DREAM: Deep Research Evaluation with Agentic Metrics	Agentic Evaluation Protocol / Capability-Parity Metrics	arXiv
2026.2	Wiki Live Challenge: Challenging Deep Research Agents with Expert-Level Wikipedia Articles	Live Wikipedia-Writing Benchmark	arXiv
2026.2	Revisiting Text Ranking in Deep Research	Retriever / Ranker Analysis for Deep Research	arXiv
2026.2	SAGE: Benchmarking and Improving Retrieval for Deep Research Agents	Retrieval Benchmark + Strong Baselines	arXiv
2026.2	W&D: Scaling Parallel Tool Calling for Efficient Deep Research Agents	Parallel Tool Calling (Width) vs Turns (Depth)	arXiv
2026.1	ScholarGym: Benchmarking Large Language Model Capabilities in the Information-Gathering Stage of Deep Research	Academic Literature Information-Gathering Benchmark	arXiv
2026.1	DeepSearchQA: Bridging the Comprehensiveness Gap for Deep Research Agents	Multi-Step Deep Research QA Benchmark	arXiv
2026.1	IDRBench: Interactive Deep Research Benchmark	Interactive Deep Research Benchmark	arXiv
2026.1	DR-Arena: an Automated Evaluation Framework for Deep Research Agents	Dynamic Automated Evaluation Framework	arXiv
2026.1	DeepResearch Bench II: Diagnosing Deep Research Agents via Rubrics from Expert Report	Rubric-Based Expert-Report Benchmark	arXiv
2026.1	DeepResearchEval: An Automated Framework for Deep Research Task Construction and Agentic Evaluation	Automated Task Construction + Agentic Evaluation	arXiv
2026.4	AgentWebBench: Benchmarking Multi-Agent Coordination in Agentic Web	Multi-Agent Coordination Benchmark for Agentic Web	arXiv
2026.4	ClawBench: A Benchmark for Evaluating AI Agents on Real-World Online Tasks (Code, Project)	Live-website, long-horizon browser-agent benchmark with reproducible task execution and layered traces	arXiv
2026.4	MERRIN: A Benchmark for Multimodal Evidence Retrieval and Reasoning in Noisy Web Environments	Multimodal Noisy-Web Retrieval + Reasoning Benchmark	arXiv
2026.2	LiveNewsBench: Evaluating LLM Web Search Capabilities with Freshly Curated News	Fresh News Benchmark for Agentic Web Search	arXiv
2025.12	LocalSearchBench: Benchmarking Agentic Search in Real-World Local Life Services	Vertical-Domain Benchmark for Local-Life Agentic Search	arXiv
2026.1	Agentic Search in the Wild: Intents and Trajectory Dynamics from 14M+ Real Search Requests	Real‑World Agentic Search Logs / Trajectory Analysis	arXiv
2025.8	DeepScholar-Bench: A Live Benchmark and Automated Evaluation for Generative Research Synthesis	Live Research-Synthesis Benchmark + Automated Evaluation	arXiv
2025.8	ReportBench: Evaluating Deep Research Agents via Academic Survey Tasks	Academic Survey-Task Benchmark	arXiv
2025.5	DeepResearchGym: A Free, Transparent, and Reproducible Evaluation Sandbox for Deep Research	Reproducible Evaluation Sandbox / Search API	arXiv

Datasets

Category	Datasets
Knowledge Source	wiki-dump, Common Crawl, KILT, PubMed, arXiv
Knowledge-Intensive QA	NQ, TriviaQA, HotpotQA, 2WikiMultiHopQA, MuSiQue, PopQA, CAG, C-SimpleQA, SuperGPQA BRIGHT, BRIGHT-Pro, SealQA, BLUR, NaturalReasoning FEVER, EX-FEVER, FEVEROUS, FactBench, RealFactBench, LongFact, FRAMES, RAG-Bench BEIR, AmbigQA, MetaQA, WebQuestions, CWQ, CheckWhy, BeerQA HETERQA
Web-based Search	WebQA, Bamboogle, Mind2Web, Mind2Web 2, WebArena, WebWalkerQA, AgentBench BrowseComp-en, BrowseComp-zh, BrowseComp-Plus, GAIA, GAIA-2, XbenchDeepSearch WebPuzzle, InfoDeepSeek, ORION, WebShaperQA, LocalSearchBench, LiveNewsBench, AgentWebBench, WikiWeb-ERP, DeepResearch-9K, DeepWeb-Bench DRACO, Total Recall QA, DeepSearchQA, ScholarGym, DR3-Eval VERITAS (Traversal Assessment), XBCP DeepSearch-World
Multi-modal	InfoSeek, MMSearch, MMSearch-Plus, SimpleVQA, LiveVQA, MM-BrowseComp MAT-Search, Mocheg, MFC-Bench, ViDoSeek, SlideVQA, MMLongBench MMDeepResearch-Bench, Vision-DeepResearch Benchmark, MM-SearchExam, VisBrowse-Bench, LongVidSearch, VideoDR, MTA-Vision-DeepSearch, PtahEval, SearchVL-SFT/RL, PaperScope, MERRIN VideoSearch-QA
Conversational	CoQA, QuAC, MSMarco, TopiOCQA, QReCC, OR-QuAC, NarrativeQA, Doc2Dial
Domain-specific	MATH, MATH500, AIME24, AIME25, GSM8K, MinervaMath MMLU, MMLU-Pro, NuminaMath MedQA, MedMCQA, MedBrowseComp, ProtMCQs, BioMedArena OlympiadBench, USACO, HLE FinSearchBench-24, Deep FinResearch Bench, FinAgentBench, KARLBench, PhySciBench, NASA-EO-Bench, Partial Evidence Bench, TaxoBench, xbench MIRAGE, SolutionBench, DQA, AirQA, HERB SciQ, SciFact, ARC, ScIRGen-Geo, DeepShop, NFCorpus, OpenThoughts, SciResearcher ICBCBench, DRFLOW, AutoResearchBench MTDTool, WorldCupArena, FinResearchBench II, ADR-bench, KDR-Bench, SciConBench, ResearcherBench
Safety / Robustness & Meta-Evaluation	DRNOISE, FORGE, KidnapRAG, RuVerBench