Beyond Majority Voting: Towards Fine-grained and More Reliable Reward Signal for Test-Time Reinforcement Learning

🎉 News • 📌 Overview • 📊 Main Results

✨ Getting Started • 🤝 Acknowledgements

📨 Contact • 🎈 Citation

🎉News

• [2026/4] Our paper is accepted by ACL 2026 as Main Conference Paper.

• [2025/12] We release both the paper and code for SCOPE.

---

📌Overview

We propose SCOPE (Subgroup-specific step-wise COnfidence-weighted Pseudo-label Estimation), a test-time reinforcement learning framework that mitigates confirmation bias and reward sparsity during the unsupervised RL. By leveraging step-wise confidence and dynamic subgroup partitioning, SCOPE provides more reliable supervision and enables diverse, high-quality reasoning exploration. It achieves superior performance across representative reasoning benchmarks, consistently surpassing strong baselines.

---

📊Main Results

Experimental results demonstrate that SCOPE consistently achieves superior performance across diverse model scales. On lightweight models (e.g., Qwen2.5-Math-1.5B), SCOPE effectively filters incorrect reasoning paths despite limited capacity, yielding a remarkable 36.5% relative improvement on the challenging AIME 2024 benchmark compared to TTRL. This advantage scales effectively to medium-sized models; for instance, SCOPE boosts LLaMA-3.1-8B's AIME 2024 performance by 50.3% and enables Qwen3-8B to outperform strong baselines like EVOL-RL by 11.5% on competition-level tasks. Overall, SCOPE demonstrates a strong capability to leverage dense reward signals for rectifying subtle errors, establishing a dominant lead in complex problem-solving scenarios.

---

✨Getting Started

Clone our repository and install the required environment:

# Clone the repository
git clone https://github.com/szu-tera/SCOPE.git
cd SCOPE
# We use uv to manage the environment
uv sync
# Install SCOPE
uv pip install -e . --no-deps

Reproduce the results on AMC 2024 with Qwen3-8B:

# Preprocess the data
uv run --active python data/preprocess.py
# Run the script
bash examples/SCOPE/Qwen3-8B/amc.sh

---

🤝Acknowledgements

This project builds upon the following open-source projects:

• verl: Volcano Engine Reinforcement Learning for LLMs
• TTRL: Test-Time Reinforcement Learning

We sincerely thank the authors and contributors for their valuable work.

---

📨Contact

Weiqin Wang: here1swqw@gmail.com

---

🎈Citation

If you find this work useful for your research, please consider citing our paper:

@article{wang2025beyond,
  title={Beyond Majority Voting: Towards Fine-grained and More Reliable Reward Signal for Test-Time Reinforcement Learning},
  author={Wang, Weiqin and Wang, Yile and Chen, Kehao and Huang, Hui},
  journal={arXiv preprint arXiv:2512.15146},
  year={2025}
}