CriticSearch

Official implementation of CriticSearch: Fine-Grained Credit Assignment for Search Agents via a Retrospective Critic.

CriticSearch uses a frozen retrospective critic to evaluate each <search> action in a trajectory, converting sparse outcome rewards into dense turn-level feedback for RL training of search agents.

Installation

conda create -n criticsearch python=3.9
conda activate criticsearch

pip install torch==2.4.0 --index-url https://download.pytorch.org/whl/cu121
pip3 install vllm==0.6.3
pip install -e .
pip3 install flash-attn --no-build-isolation
pip install wandb

# Retriever (recommended: separate environment)
conda create -n retriever python=3.10
conda activate retriever
conda install pytorch==2.4.0 torchvision==0.19.0 torchaudio==2.4.0 pytorch-cuda=12.1 -c pytorch -c nvidia
pip install transformers datasets pyserini uvicorn fastapi
conda install -c pytorch -c nvidia faiss-gpu=1.8.0

Data & Model Preparation

# Retrieval corpus and index
save_path=corpus
python scripts/download.py --save_path $save_path
cat $save_path/part_* > $save_path/e5_Flat.index
gzip -d $save_path/wiki-18.jsonl.gz

# Training data
bash scripts/hotpotqa_search/data_process.sh

# Models (download and place as needed)
# - model/e5-base-v2                       Retriever
# - model/qwen-2.5-3b                      Policy LLM (BASE_MODEL in train.sh)
# - model/Qwen3-30B-A3B-Instruct-2507      Critic LLM (model_path in vllm_reward_server.py)

mkdir -p data/rl_logs data/verl_checkpoints

Quick Start

Launch the following in three separate terminals:

Step 1: Start the Retriever

conda activate retriever
bash retrieval_launch.sh

Service endpoint: http://127.0.0.1:8000/retrieve

Step 2: Start the Retrospective Critic

conda activate criticsearch
python search_r1/contribute_reward/vllm_reward_server.py

Update model_path in vllm_reward_server.py to your local critic model before launching.

Service endpoint: http://127.0.0.1:8111/batch_process

Step 3: Start Training

conda activate criticsearch
bash train.sh

Training logs: data/rl_logs/ · Checkpoints: data/verl_checkpoints/

Acknowledgement

This work is implemented based on Search-R1. We sincerely thank the authors of these projects for their valuable contributions to the open-source community.

Citation

@article{zhang2025criticsearch,
  title={CriticSearch: Fine-Grained Credit Assignment for Search Agents via a Retrospective Critic},
  author={Zhang, Yaocheng and Huang, Haohuan and Song, Zijun and Zhu, Yuanheng and Zhang, Qichao and Zhao, Zijie and Zhao, Dongbin},
  journal={arXiv preprint arXiv:2511.12159},
  year={2025}
}