Process Reward Model for Large Language Model

Installation

git clone https://github.com/BiNLP/PRM4LLM.git
cd process-reward-ppo

conda create -n prm4llm python=3.12
conda activate prm4llm

pip install -r requirements.txt

Usage

Train a Process Reward Model

python scripts/train_reward_model.py --config configs/reward_model.json

你可以在configs/reward_model.json中配置：

• 使用的模型
• 数据集名称
• 训练参数
• 输出目录等

Use PPO to finetune LLMs

python scripts/run_ppo_training.py --config configs/ppo_config.json --reward_model_path models/reward_model/final_model

PPO训练配置可在configs/ppo_config.json中修改。

Evaluation

python scripts/evaluate_model.py --model_path models/ppo_finetuned/final-model --config configs/eval_config.json

Citation

@misc{PRM4LLM,
  author = {Zhenbin Chen},
  title = {Process Reward Model for Large Language Model},
  year = {2025},
  publisher = {GitHub},
  url = {https://github.com/BiNLP/PRM4LLM.git}
}