-
Notifications
You must be signed in to change notification settings - Fork 193
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Online DPO 支持 #385
Comments
这个是在训练之前从reference model中推理数据,reward model打分 |
这个实现的就是每轮用最新的checkpoint 推理数据打分,只不过batch size 默认设置的比较大。如果要实现PPO那种完全online的会复杂一点,可以参考train_ppo_ray.py的设计 |
好的,谢谢,我再仔细看看这一段 |
我想实现Online DPO,即类似PPO,每轮batch的数据都由policy model 推理,reward model打分得到
但是我不会在边训练模型参数、边使用vllm进行推理加速,导致我目前的模型训练过程很慢
我发现这个仓库的基于vllm的ppo训练加速方式应该可以解决我的问题,但是阅读仓库源码来理解如何在训练过程使用vllm给我带来一些困难,请问可以有一些文档、手册来帮助我吗
The text was updated successfully, but these errors were encountered: