-
Notifications
You must be signed in to change notification settings - Fork 5.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[RLlib] Fix frame stacking connector for experiments with more than 1 Learner. #45588
[RLlib] Fix frame stacking connector for experiments with more than 1 Learner. #45588
Conversation
…dqn_multi_agent_debugging
Signed-off-by: sven1977 <svenmika1977@gmail.com>
…dqn_multi_agent_debugging
…dqn_multi_agent_debugging
…ti_agent_debugging
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM. Some changes from another PR are included here.
self, | ||
stats_dicts, | ||
stats_dicts: List[Dict[str, Any]], |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is already committed in the other PR today?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah, yes, very sorry. I had to make sure the other PR (MetricsLogger log_n_dicts()
fix) is also in here otherwise, the Atari benchmark would have failed.
Thanks for pointing this out.
… Learner. (ray-project#45588) Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>
… Learner. (ray-project#45588) Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>
Fix frame stacking connector for experiments with more than 1 Learner.
The bug was that in
SingleAgentEpisode.slice()
, which the LearnerGroup has to do in order to slice the received episodes in n shards to be sent to the n Learners, we would NOT properly set the observation/action spaces in the returned slices. This then lead to an incorrect fill-value (shape=1D instead of Atari 4D) for the left edges of the episodes (framestacking at the first m episodes timesteps needs to be left-0-padded), wherem = num_frames - 1
.Why are these changes needed?
Related issue number
Checks
git commit -s
) in this PR.scripts/format.sh
to lint the changes in this PR.method in Tune, I've added it in
doc/source/tune/api/
under thecorresponding
.rst
file.