Doing More with Less Tokens: Hierarchical Reinforcement Learning for Efficient Coding Agents
HERO (HiErarchical ReinfOrcement learning) trains coding agents to solve software-engineering tasks efficiently while preserving resolution performance. Following our paper, it couples:
- Hierarchical policy optimization: prioritize task resolution and activate efficiency optimization once sufficient problem-solving capability is acquired.
- Multi-granularity efficiency credit assignment: encourage concise reasoning trajectories and penalize potentially unproductive turns using turn-level entropy.
This repository provides HERO training and evaluation on SWE-bench Verified and SWE-bench Multilingual, with released HERO-4B, HERO-9B, and HERO-35B-A3B model.
Follow Prepare.md to prepare the environment, task images, and model weights. Then launch HERO-4B training from the repository root:
source .venv/bin/activate
WANDB_MODE=disabled bash run_rl/train_4b.shFor evaluation only, follow the one-task evaluation setup.
Use Linux x86-64, Python 3.12, CUDA 13.0, and local Docker storage. Follow Prepare.md in order; it covers every resource below.
| Resource | How to prepare it |
|---|---|
| Python/CUDA environment | Host and environment setup; the installer prepares the Python stack, but NVIDIA drivers and CUDA must already be installed |
| Training and evaluation data | Extract the bundled Git LFS archive, data.tar.gz, into ../data/ |
| Task Docker images | Image download commands read the registry names from the data and save archives under ../docker/ |
| Agent scaffold | Claude Code 2.1.111 + Node.js 22.11.0, downloaded automatically into ../cli/claude_code/ |
| Grading harnesses | Harness table explains SWE-Gym, SWE-bench, Multi-SWE, and SWE-rebench routing and installation |
| Model weights | Model download table and conversion; HF weights for evaluation, HF + Megatron weights for training |
| Training frameworks | Slime and Megatron-LM are bundled in framework/ and installed by the environment script |
The dataset archive is stored with Git LFS; install Git LFS before cloning.
Extract the bundled dataset archive from the repository root into the sibling
../data/ directory:
git lfs pull --include="data.tar.gz"
tar -xzf data.tar.gz -C ..| File | Tasks | Use |
|---|---|---|
swe_train_640.jsonl |
640 | Training: 352 SWE-Gym, 238 Multi-SWE, 50 SWE-ReBench |
swe_bench_verified.jsonl |
500 | SWE-bench Verified evaluation |
swe_bench_multilingual.jsonl |
300 | SWE-bench Multilingual evaluation |
The archive includes the three JSONL files and their source/license README.
The image helper downloads their referenced
Docker images into ../docker/.
After completing environment setup and weight conversion, run from the repository root:
source .venv/bin/activate
WANDB_MODE=disabled bash run_rl/train_4b.shDefault GPU allocations by model size:
| Model | Default GPUs |
|---|---|
| Qwen3.5-4B | 1 node × 8 NVIDIA H200 |
| Qwen3.5-9B | 1 node × 8 NVIDIA H200 |
| Qwen3.5-35B-A3B | 8 nodes × 8 NVIDIA H200 |
The launch scripts contain the default hyperparameters. For online logging,
run wandb login and use WANDB_MODE=online WANDB_PROJECT=hero.
The method implementation is in run_rl/HERO/; environment setup, model
recipes, and Docker execution live in run_rl/_shared/. Training and evaluation
do not depend on any optional baseline directory. See the
method implementation for the advantage and loss hooks.
| Model | Download |
|---|---|
| HERO-4B | XLearning-SCU/HERO-4B |
| HERO-9B | XLearning-SCU/HERO-9B |
| HERO-35B-A3B | XLearning-SCU/HERO-35B-A3B |
Evaluation trajectories for SWE-bench Verified and SWE-bench Multilingual are available at HERO-Trajectories. For training, start from the original Qwen3.5 weights.
Download one of the HERO checkpoints above into ../models/HERO-4B/, ../models/HERO-9B/, or ../models/HERO-35B-A3B/.
Then evaluate the selected model:
MODEL=../models/HERO-4B bash eval/run_eval_swebench_verified.sh
MODEL=../models/HERO-4B bash eval/run_eval_swebench_multilingual.shSet MODEL to the corresponding directory for HERO-9B or HERO-35B-A3B.
Evaluation uses a 131,072-token context and up to 200 agent turns. Start with the
one-task command to check your environment.
../results/
├── train/HERO/<model>/<timestamp>_<pid>_<experiment>/
└── eval/<dataset>/<model>/<timestamp>_<pid>_<experiment>/
Training saves checkpoints, settings, and logs. Evaluation saves results.jsonl,
eval_result.json, and per-task artifacts. Set RESULTS_ROOT to change the output
root, or RUN_ROOT / OUT to select one training / evaluation run.
HERO builds on ProRL-Agent-Server / Polar, Slime, Megatron-LM, SGLang, and the Qwen models. It uses tasks and grading resources from SWE-Gym, Multi-SWE, SWE-ReBench, and SWE-bench.
If you find HERO useful, please cite our paper:
@article{li2026hero,
title={Doing More with Less Tokens: Hierarchical Reinforcement Learning for Efficient Coding Agents},
author={Haobin Li and Liang Jiang and Zhenyu Huang and Mouxing Yang and Xi Peng},
journal={arXiv preprint arXiv:2609.38885},
year={2026}
}HERO is distributed under the Apache-2.0 License. Bundled frameworks retain their license files; external datasets, models, and tools retain their respective licenses and terms.