PPO is on-policy, but its training used a replay buffer?

[houghtonweihu]

PPO is on-policy, but the training of it in https://github.com/hpcaitech/ColossalAI/blob/main/applications/Chat/coati/trainer/ppo.py
used a replay buffer, which is only possible for off-policy.

[13918985654]

what is the brawlstar

[Camille7777]

Hi, @houghtonweihu, in the PPO we still collect experience (PPO uses importance sampling) , we don't store it in a “replay buffer” as off-policy but just use the replay buffer class to hold the batch, and then clear the buffer after immediate use.

[houghtonweihu]

This makes sense, thanks!