There is no training log dumping with the first rollout. Instead, the training log starts to dump to the screen from the second rollout. After checking the code, I found the training log dumping together with the second rollout is actually the training info from the first rollout. I think it may be better to dump the training info together with the first rollout?
PPO log output
First rollout
| rollout/ | |
| ep_len_mean | 3 |
| ep_rew_mean | -0.574 |
| time/ | |
| fps | 62 |
| iterations | 1 |
| time_elapsed | 2 |
| total_timesteps | 144 |
Second rollout
| rollout/ | |
| ep_len_mean | 3 |
| ep_rew_mean | -0.543 |
| time/ | |
| fps | 77 |
| iterations | 2 |
| time_elapsed | 3 |
| total_timesteps | 288 |
| train/ | |
| approx_kl | 0.0031222207 |
| clip_fraction | 0.00417 |
| clip_range | 0.2 |
| entropy_loss | -1.42 |
| explained_variance | 0.00404 |
| learning_rate | 0.0003 |
| loss | 0.00362 |
| n_updates | 10 |
| policy_gradient_loss | -0.0147 |
| std | 0.997 |
| value_loss | 0.223 |
