[RLlib] DD-PPO training iteration fn #23906
Conversation
sven1977
requested review from
gjoliver,
avnishn,
ArturNiederfahrenhorst and
smorad
as code owners
rllib/agents/ppo/ddppo.py
Outdated
| sample_and_update_results = asynchronous_parallel_requests( | ||
| remote_requests_in_flight=self.remote_requests_in_flight, | ||
| actors=self.workers.remote_workers(), | ||
| ray_wait_timeout_s=1000.0, # 0.0 |
There was a problem hiding this comment.
Ah, sorry, was just trying some stuff. Basically, this makes it synchronous :) Will revert. ...
There was a problem hiding this comment.
Huh? Ok, now it's fixed ... Forgot to push.
| env: Pendulum-v1 | ||
| run: DDPPO | ||
| stop: | ||
| episode_reward_mean: -300 |
There was a problem hiding this comment.
Which part?
- reward: It's able to get to -300.
- timesteps: Yeah, it does need up to 1M sometimes. DD-PPO is not a very stable algo, it seems. Especially on cont. action tasks. But even on Atari I have yet to find a good choice of hyperparams.
There was a problem hiding this comment.
Oh for some reason I read this as CartPole and not Pendulum, my b
There was a problem hiding this comment.
Haha, yeah -300 would be pretty bad for CartPole :)
| @@ -249,6 +249,16 @@ py_test( | |||
| args = ["--yaml-dir=tuned_examples/ppo"] | |||
| ) | |||
|
|
|||
| py_test( | |||
There was a problem hiding this comment.
This task is now working properly. Due to proper hyperparam tuning.
There was a problem hiding this comment.
awesome! How difficult did you find it to tune hparams?
There was a problem hiding this comment.
It was pretty hard, actually.
It's good to start with just one worker and using the exact same hparams as the respective PPO version. Then increase num_workers and at the same time carefully adjust:
rollout_fragment_length
num_envs_per_worker
sgd_minibatch_size
num_sgd_iter
in short, anything that affects (per-worker) batch-size, and the time each worker spends on a decentralized update.
| @@ -249,6 +249,16 @@ py_test( | |||
| args = ["--yaml-dir=tuned_examples/ppo"] | |||
| ) | |||
|
|
|||
| py_test( | |||
There was a problem hiding this comment.
awesome! How difficult did you find it to tune hparams?
rllib/agents/ppo/ddppo.py
Outdated
| sample_and_update_results = asynchronous_parallel_requests( | ||
| remote_requests_in_flight=self.remote_requests_in_flight, | ||
| actors=self.workers.remote_workers(), | ||
| ray_wait_timeout_s=1000.0, # 0.0 |
| env: Pendulum-v1 | ||
| run: DDPPO | ||
| stop: | ||
| episode_reward_mean: -300 |
There was a problem hiding this comment.
Oh for some reason I read this as CartPole and not Pendulum, my b
…o_training_itr
The DDPPO LR scheduler test is broken because the learner_info_dictionary that is returned by the training iteration function does not consistently return a learner info for every training iteration, but the test expects that it does. We'll need to fix the test then re-merge Reverts #23906
DD-PPO training iteration fn implementation:
Why are these changes needed?
Related issue number
Checks
scripts/format.shto lint the changes in this PR.