Add PPO training.

[guoshengCS]

PR types

New features

PR changes

Others

Description

Add PPO training.

[paddle-bot]

Thanks for your contribution!

[codecov]

Codecov Report

Attention: 35 lines in your changes are missing coverage. Please review.

Comparison is base (16d3c49) 56.68% compared to head (ec150b6) 56.70%.
Report is 1 commits behind head on develop.

Files	Patch %	Lines
paddlenlp/generation/utils.py	46.66%	32 Missing ⚠️
paddlenlp/transformers/llama/modeling.py	57.14%	3 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #7305      +/-   ##
===========================================
+ Coverage    56.68%   56.70%   +0.02%     
===========================================
  Files          588      588              
  Lines        89243    89305      +62     
===========================================
+ Hits         50584    50639      +55     
- Misses       38659    38666       +7     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[guoshengCS]

相较于Beaver(DeepSpeed)这里PPOTrainer的代码实现会更复杂一些，主要是要从粗粒度的Trainer.train中拷贝抽离出来囊括完整单步训练（forward+backward+opt.step）的处理逻辑代码(full_train_step)，另外一些是为了和Trainer尽可能功能和表现对齐以及复用有一些代码拷贝和适配。

如果Trainer有更细粒度的包括forward、backward、opt.step方法提供出来，对于PPOTrainer以及其他训练逻辑复杂一些的算法实现可能会更容易些，类似于下面Beaver使用DeepSpeed engine的情况(actor_model和reward_critic_model都是DeepSpeed engine) https://github.com/PKU-Alignment/safe-rlhf/blob/main/safe_rlhf/algorithms/ppo/trainer.py#L171

[ZHUI]

这个可以初始化的时候 搞掉吗? 我记得embeding那边有判断。

[guoshengCS]

@wj-Mcat 看Test CI挂了，需要多卡运行(python -m paddle.distributed.launch)运行的测试要如何加入了

[wawltor]

LGTM