Learning Curve and Rendering Mismatch

[ecada]

I've trained PPO1 with the given hyperparameters but at the end of 1M timesteps I see a decline in the episode rewards. However when rendering the environment the humanoid is able to walk better after being trained 20M timesteps(where the episode reward is at minimum) compared to being trained 1M (where the episode reward is maximum). Has anyone noticed this dilemma? Am I missing something?

[pzhokhov]

interesting... in ppo1 we use rewards from environment directly (without any adaptive normalization) so decline in rewards should correspond to the actual decline in the environment rewards. Could you post here an example of learning curve that you are getting?

I get the following learning curve in ppo1/run_humanoid.py with 16 processes:

As another note, we are deprecating PPO1 (you can do everything that PPO1 can do with PPO2, including MPI parallelization), so if I would recommend trying out PPO2 as well

[ecada]

Thank you very much for the quick reply. My laptop gets overheated when I run 16 processes so I run it with a single python command. I believe that is caused by training too much in a single environment. I was using the tensorboard for plotting. However I'm still curious why after 600 iterations the rewards decrease. I think the problem was that checkpoints for simulation was created with (16 or 8 I don't remember well but I forgot to use tensorboard at that run) processes and the tensorboard logging was with 1 process. But I'm still not quite sure because the rewards shouldn't decrease after some point. I will be really grateful if you elaborate on that. Also PPO2 takes too long even with mpi for mujoco continuous control tasks on a Macbook Pro. Is there any additional benefit of using it instead of PPO1 that I'm missing?

[NishanthVAnand]

can I please know what versions of gym and mujoco are you using?

[pzhokhov]

@ecada if the rewards bounce up and down but on average increase steadily, that is a reasonable behaviour; however, if on average the rewards decrease steadily, that usually indicates some problem with hyperparameters. I have not actually tried training a humanoid with a single environment using PPO1 (or PPO2), but I will and let you know; but it is possible that having 1 env instead of 16 being run in parallel creates insufficient variance in the minibatches, so that gradient estimates are not accurate.
Not sure I understand your question about tensorboard... logger module that saves tensorboard data should be smart enough to save data from different MPI processes into different files so those would not collide.
PPO2 vs PPO1 - they should be roughly the same speed, altough I think in the case of MPI PPO2 does not quite count the steps correctly (so if you pass --num_timesteps=1e7, each of the MPI processes will take 1e7 timesteps, whereas in PPO1 each MPI process will take 1e7 / number_of_processes steps) - this could be the reason for the apparent slowness of PPO2.
Additional benefits of using PPO2 - you can make it simulate 16 environments in parallel without actually running 16 processes in parallel (in your case, that should fight the overheating and still allow for 16 envs being run in parallel - to do that, modify make_vec_env method in baselines/baselines/common/cmd_util.py to use DummyVecEnv instead of SubprocVecEnv, and run without mpi and with --num_env=16 option). On top of that, PPO2 uses dynamic observation and reward normalization, is compatible with recurrent neural nets and more easily comparable to other algorithms... but I am slipping into an advertisement speech here :)

@NishanthVAnand the plot above is obtained with
mujoco-py==1.50.1.56 and
gym==0.10.5

[ecada]

Thank you for the detailed answer. I trained PPO2 as you suggested and got the same results in less time with less overheating :) I was expecting the rewards to have some variance but the overall average rewards decreased after some iterations when I train PPO1 on just 1 environment. Since the problem is resolved I'm closing this issue.