SCOPE-RL

Step-level Cognitive quality Optimization for Process-aware rEasoning via Reinforcement Learning.

A two-stage RLVR (Reinforcement Learning with Verifiable Rewards) framework that improves mathematical reasoning quality — not just answer correctness — by providing fine-grained, step-level reward signals.

Method

Stage	Reward	Description
ASR (Answer-aware Step Reward)	Rule-based	Decomposes problems into sub-problems; rewards per-step correctness
QPR (Quality Process Reward)	LLM-judged	Classifies each reasoning step and penalizes errors, reversions, redundancy

QPR formula:

Q_process = S_u × Φ × κ

• S_u: useful-step ratio
• Φ: multiplicative penalty for error / reversion / redundant / mechanical steps
• κ = 1/(1 + α·ln(N+1)): soft length penalty

Repository Structure

├── benchmark/      # Evaluation pipeline (step-level reasoning quality metrics)
├── data/           # Data preparation (problem decomposition for ASR training)
└── verl/           # RL training (verl framework + custom reward recipe)
    └── recipe/step_reward_rl/   # Custom reward functions (ASR & QPR)

Quick Start

1. Data Preparation (ASR)

Decompose math problems into sub-problems with verifiable answers:

cd data
python decompose.py --input <problems.jsonl> --output <decomposed.jsonl> --workers 32

2. RL Training

Train with verl using the custom step-level rewards:

cd verl/recipe/step_reward_rl
bash run_qwen.sh

3. Evaluation

Evaluate model reasoning quality:

cd benchmark
python eval.py

Requirements

• Python 3.10+
• verl and its dependencies
• A vLLM-compatible API endpoint (for model rollout and LLM judge)