max_grad_norm and use_clipped_value_loss

[seungjaeryanlee]

Hello! I was documenting your PPO code algo/ppo.py to improve my understanding of the algorithm, and I got confused on max_grad_norm and _use_clipped_value_loss.

If I am understanding this correctly, max_grad_norm is given to nn.utils.clip_grad_norm_() to set maximum gradient size, and _use_clipped_value_loss. However, I could not find relevent detail in the paper Proximal Policy Optimization Algorithm. If it was explicitly mentioned here, would you please point it out for me?

For L^VF, The paper seems to use the simple squared loss, equivalent to use_clipped_value_loss=False, but I could not find anything about the case when use_clipped_value_loss=True. Is this a trick not mentioned in the paper?

Thank you in advance for your help. Happy holidays!

[ikostrikov]

They introduced the new loss in the implementation of PPO2:
https://github.com/openai/baselines/blob/master/baselines/ppo2/model.py#L63

Also see grad normalization here:
https://github.com/openai/baselines/blob/master/baselines/ppo2/model.py#L102

[seungjaeryanlee]

Thank you for the links! I see how they correspond to those parts of PPO2 in OpenAI Baselines.

It's unfortunate that these changes are not written in any paper. Guess I will have to read openai/baselines code as well.