RSIPO: Recursive Self-Improving Policy Optimization for Agentic Reinforcement Learning via Skill-Policy Co-Evolution

RSIPO is a self-evolving agentic RL framework in which agentic skills and the policy co-evolve during training. It builds on the hybrid of RLVR and on-policy self-distillation (OPSD) in SEED, where the policy summarizes its own trajectories into skills and a skill-enhanced copy of the policy acts as the teacher.

• [2026.10] Our paper is available on arXiv [📄 arXiv].
• [2026.10] We released the code and the RSIPO checkpoints of Qwen2.5-7B-Instruct, Qwen3-1.7B and Qwen2.5-3B-Instruct on ALFWorld and search-based QA [🤗 Models].

Figure 1. Overview of the RSIPO framework. Replay gains measured by the execution check reward on-policy skill generation and calibrate the student's trust in the skill-enhanced teacher.

RSIPO adds an execution check: for every training task, the policy rewrites a skill from one of its rollouts, the skill-enhanced teacher replays the task from the same initial state, and the replays are compared with control rollouts of the student. The measured replay gains are used twice:

1. On-policy skill evolution. The gain, centered by the mean gain of the task type, rewards the skill the policy wrote, so skill generation is optimized on-policy together with task execution.
2. Trust-calibrated self-distillation. The gain, shrunk towards its task-type mean with empirical Bayes, sets the weight of the OPSD loss of the task, so the student follows the teacher only as far as the teacher helps.

No skill is used at test time.

RSIPO is compared with 13 baselines on ALFWorld and seven search-based QA datasets with three base models. It achieves the best average success rate on ALFWorld with all three base models and the best average exact match on search-based QA with Qwen2.5-7B-Instruct and Qwen3-1.7B; with Qwen2.5-3B-Instruct it ranks second on search-based QA, 0.4 points below SEED.

ALFWorld: success rate (%). Search: exact match (%) averaged over the seven datasets. Methods marked with * use skills at test time. Within each column, the best result is in bold and the second best is underlined.

Figure 2. (a) Training dynamics of RSIPO with Qwen2.5-3B-Instruct, where thick lines are 10-step moving averages. (b) Ablation on ALFWorld with Qwen2.5-3B-Instruct, where dots mark the components of each variant.

The dynamic trust tracks the reliability of the teacher: on ALFWorld it exceeds 1.1 within 20 steps as the skill-enhanced teacher increasingly outperforms the control rollouts, and on search-based tasks, where the teacher offers little extra benefit, it stays close to one. In the ablation, dynamic trust and skill evolution add 3.6 and 5.6 points to the hybrid pipeline with SFT initialization (86.4), and the full RSIPO reaches 95.7; removing either component costs 3.7 to 5.7 points.

Figure 3. On-policy skill evolution of Qwen2.5-7B-Instruct on ALFWorld. Left: skill content and replay gain Δ_x. Right: condensed skills for tasks that place two credit cards, where step 0 is the SFT policy.

Early skills inherited from SFT often concern action format or recovery from failed actions and have a negative average replay gain. Procedure skills exceed 94% in the last 45 steps, and the replay gain turns positive.

The policies reported in Table 1 (Qwen2.5-7B-Instruct and Qwen3-1.7B) and Table 3 (Qwen2.5-3B-Instruct) of the paper, after 150 RL steps, in bf16:

ALFWorld: success rate (%). Search-based QA: average exact match (%) over NQ, TriviaQA, PopQA, HotpotQA, 2WikiMultihopQA, MuSiQue and Bamboogle. The Qwen3 models run in non-thinking mode. See Evaluation for evaluating a downloaded checkpoint.

We trained with Python 3.10, CUDA 11.8, PyTorch 2.6.0, vLLM 0.8.5, Transformers 4.51.3, Ray 2.46.0 and FlashAttention 2.7.3.

conda create -n rsipo python=3.10 -y
conda activate rsipo
pip install torch==2.6.0 --index-url https://download.pytorch.org/whl/cu118
pip install vllm==0.8.5 transformers==4.51.3 ray[default]==2.46.0
pip install flash-attn==2.7.3 --no-build-isolation
pip install -e .

# ALFWorld
pip install alfworld==0.4.2 textworld==1.6.2 gym==0.24.0 gymnasium==0.29.1 stable-baselines3==2.6.0
alfworld-download -f                      # games and PDDL files, stored in ~/.cache/alfworld

# Search-based QA
pip install -e agent_system/environments/env_package/search/third_party
pip install faiss-cpu==1.8.0.post1 fastapi uvicorn

Copy .env.example to .env and fill in the paths, the conda environment names, and the OpenAI-compatible endpoint of the external LLM that writes the skills for the SFT data (GLM-5.2 in the paper). All scripts assume the repository root as the working directory.

ALFWorld. alfworld-download -f provides everything; set ALFWORLD_DATA to its directory.

Search-based QA. The Search-R1 data (test set of the seven datasets), the e5 index and the 2018 Wikipedia corpus:

source scripts/rsipo/env.sh
python examples/data_preprocess/preprocess_search_r1_dataset.py --local_dir $SEARCH_DATA/searchR1_processed_direct
python examples/search/searchr1_download.py --local_dir $SEARCH_DATA/searchR1
cat $SEARCH_DATA/searchR1/part_* > $SEARCH_DATA/searchR1/e5_Flat.index
gzip -d $SEARCH_DATA/searchR1/wiki-18.jsonl.gz

The training set and the SFT tasks of each base model are built by a script under scripts/data/ from the training splits of Search-R1 and FlashRAG:

hf download RUC-NLPIR/FlashRAG_datasets --repo-type dataset --local-dir $SEARCH_DATA/flashrag \
  --include "triviaqa/train.jsonl" "2wikimultihopqa/train.jsonl" "musique/train.jsonl" "trex/train.jsonl"
python scripts/data/trex_popqa.py --trex $SEARCH_DATA/flashrag/trex/train.jsonl --out $SEARCH_DATA/trex

python scripts/data/search_train_qwen2.5-7b.py --data-dir $SEARCH_DATA --flashrag-dir $SEARCH_DATA/flashrag \
  --out $SEARCH_DATA/train/qwen2.5-7b
python scripts/data/search_train_qwen2.5-3b.py --data-dir $SEARCH_DATA --flashrag-dir $SEARCH_DATA/flashrag \
  --trex-popqa $SEARCH_DATA/trex/trex_popqa_relations.jsonl --out $SEARCH_DATA/train/qwen2.5-3b
python scripts/data/search_train_qwen3-1.7b.py --data-dir $SEARCH_DATA --flashrag-dir $SEARCH_DATA/flashrag \
  --trex-popqa $SEARCH_DATA/trex/trex_popqa_relations.jsonl --out $SEARCH_DATA/train/qwen3-1.7b

Each output directory holds train.parquet (RL) and sft_tasks.parquet (SFT). The last two builders use intfloat/e5-base-v2 and cache the embeddings of the test questions in $SEARCH_DATA/test_e5.npy.

Retriever. Training and evaluation on search-based QA need the retriever running on the same node (CPU, port 8000):

bash scripts/rsipo/start_retriever.sh > retriever.log 2>&1 &

Download the base models (Qwen/Qwen2.5-7B-Instruct, Qwen/Qwen2.5-3B-Instruct, Qwen/Qwen3-1.7B) into $MODELS_ROOT. Every run in the paper used one node with 8 GPUs (96 GB each).

The base model collects 8 rollouts on each of the 500 SFT tasks, the external LLM writes a skill for every trajectory, and the base model is fine-tuned to write these skills (3 epochs). For ALFWorld with Qwen2.5-7B-Instruct:

out=$RSIPO_WORK/data/sft/alfworld-qwen2.5-7b
python scripts/sft/alfworld/sample_tasks.py --output-dir $out
OUTPUT_DIR=$out RESUME=true MODEL_PATH=$MODELS_ROOT/Qwen2.5-7B-Instruct bash scripts/sft/alfworld/prepare_data.sh
DATA_DIR=$out MODEL_PATH=$MODELS_ROOT/Qwen2.5-7B-Instruct \
  EXPORT_MODEL_DIR=$MODELS_ROOT/Qwen2.5-7B-Instruct-alfworld-sft bash scripts/sft/alfworld/train_sft.sh

For search-based QA (with the retriever running):

out=$RSIPO_WORK/data/sft/search-qwen2.5-7b
OUTPUT_DIR=$out TRAIN_DATA=$SEARCH_DATA/train/qwen2.5-7b/sft_tasks.parquet NUM_TASKS=500 \
  MODEL_PATH=$MODELS_ROOT/Qwen2.5-7B-Instruct bash scripts/sft/search/prepare_data.sh
DATA_DIR=$out MODEL_PATH=$MODELS_ROOT/Qwen2.5-7B-Instruct \
  EXPORT_MODEL_DIR=$MODELS_ROOT/Qwen2.5-7B-Instruct-search-sft bash scripts/sft/search/train_sft.sh

See scripts/sft/README.md for Qwen3 and other overrides.

bash scripts/rsipo/run_rsipo.sh alfworld Qwen2.5-7B-Instruct-alfworld-sft
RSIPO_SEARCH_TRAIN_DATA=$SEARCH_DATA/train/qwen2.5-7b/train.parquet \
  bash scripts/rsipo/run_rsipo.sh search Qwen2.5-7B-Instruct-search-sft

run_rsipo.sh sets the hyperparameters of the paper for the benchmark and base model (Appendix B) and trains for 150 steps; checkpoints, logs and per-step metrics go to $RSIPO_WORK/runs/rsipo-<model>-<env>/rsipo/, and the records of every execution check to its rsipo-audit/. RSIPO_RESUME_MODE=auto resumes an interrupted run.

bash scripts/rsipo/evaluate.sh alfworld Qwen2.5-7B-Instruct-alfworld-sft 150
bash scripts/rsipo/evaluate.sh search Qwen2.5-7B-Instruct-search-sft 150

ALFWorld is evaluated on the seen (140 games) and unseen (134 games) splits in three rounds with seeds 1, 2 and 3; the results are in eval-step150-max50/summary.json of the run. Search-based QA is evaluated on all 51,713 test questions; the results are in eval-step150/. Both use temperature 0.4 and no skill in the prompt.

To evaluate a downloaded checkpoint, place it under $MODELS_ROOT and set RSIPO_EVAL_HF=1:

hf download Karl28/RSIPO-Qwen2.5-7B-Instruct-ALFWorld --local-dir $MODELS_ROOT/RSIPO-Qwen2.5-7B-Instruct-ALFWorld
RSIPO_EVAL_HF=1 bash scripts/rsipo/evaluate.sh alfworld RSIPO-Qwen2.5-7B-Instruct-ALFWorld

scripts/rsipo/export_hf.sh RUN_DIR [STEP] [HF_REPO] merges a training checkpoint into a Hugging Face model and can upload it.

If you find RSIPO useful, please cite our paper:

@misc{zhang2026rsipo,
      title={RSIPO: Recursive Self-Improving Policy Optimization for Agentic Reinforcement Learning via Skill-Policy Co-Evolution},
      author={Fan Zhang and Hao Wu and Jinyang Wu and Hongyu Qu and Zheng Lian and Xiao Luo and Yefeng Zheng and Bo Zhang and Haoxuan Li and Pheng-Ann Heng},
      year={2026},
      eprint={2610.15772},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2610.15772},
}

RSIPO builds on SEED, verl-agent and veRL, and uses ALFWorld, Search-R1 and FlashRAG. We thank their authors.

RSIPO is released under the MIT License (LICENSE, inherited from SEED). Code from veRL and verl-agent remains under the Apache License 2.0 (Notice.txt).