Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[RLlib] Implementation of MultiAgentEpisodeReplayBuffer for the new API stack. #44450

Merged
merged 45 commits into from
Apr 25, 2024
Merged

[RLlib] Implementation of MultiAgentEpisodeReplayBuffer for the new API stack. #44450

merged 45 commits into from
Apr 25, 2024

Commits on Mar 22, 2024

  1. wip 

    Signed-off-by: sven1977 <svenmika1977@gmail.com>
    @sven1977
    sven1977 committed Mar 22, 2024
    Configuration menu
    Copy the full SHA
    b16ab62 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' of https://github.com/ray-project/ray into mult… 

    …i_agent_episode_add_module_for_api
    @sven1977
    sven1977 committed Mar 22, 2024
    Configuration menu
    Copy the full SHA
    72a4bb5 View commit details
    Browse the repository at this point in the history

  3. wip 

    Signed-off-by: sven1977 <svenmika1977@gmail.com>
    @sven1977
    sven1977 committed Mar 22, 2024
    Configuration menu
    Copy the full SHA
    9b60b8c View commit details
    Browse the repository at this point in the history

  4. wip 

    Signed-off-by: sven1977 <svenmika1977@gmail.com>
    @sven1977
    sven1977 committed Mar 22, 2024
    Configuration menu
    Copy the full SHA
    297b711 View commit details
    Browse the repository at this point in the history

Commits on Apr 2, 2024

  1. wip 

    Signed-off-by: sven1977 <svenmika1977@gmail.com>
    @sven1977
    sven1977 committed Apr 2, 2024
    Configuration menu
    Copy the full SHA
    eb6d04c View commit details
    Browse the repository at this point in the history

  2. wip 

    Signed-off-by: sven1977 <svenmika1977@gmail.com>
    @sven1977
    sven1977 committed Apr 2, 2024
    Configuration menu
    Copy the full SHA
    3819712 View commit details
    Browse the repository at this point in the history

  3. merge 

    Signed-off-by: sven1977 <svenmika1977@gmail.com>
    @sven1977
    sven1977 committed Apr 2, 2024
    Configuration menu
    Copy the full SHA
    2cebf0b View commit details
    Browse the repository at this point in the history

  4. merge 

    Signed-off-by: sven1977 <svenmika1977@gmail.com>
    @sven1977
    sven1977 committed Apr 2, 2024
    Configuration menu
    Copy the full SHA
    3a009c8 View commit details
    Browse the repository at this point in the history

Commits on Apr 3, 2024

  1. Merge branch 'master' of https://github.com/ray-project/ray into mult… 

    …i_agent_episode_add_slice_api
    @sven1977
    sven1977 committed Apr 3, 2024
    Configuration menu
    Copy the full SHA
    05d8610 View commit details
    Browse the repository at this point in the history

  2. wip 

    Signed-off-by: sven1977 <svenmika1977@gmail.com>
    @sven1977
    sven1977 committed Apr 3, 2024
    Configuration menu
    Copy the full SHA
    11ee869 View commit details
    Browse the repository at this point in the history

  3. wip 

    Signed-off-by: sven1977 <svenmika1977@gmail.com>
    @sven1977
    sven1977 committed Apr 3, 2024
    Configuration menu
    Copy the full SHA
    559f90d View commit details
    Browse the repository at this point in the history

  4. wip 

    Signed-off-by: sven1977 <svenmika1977@gmail.com>
    @sven1977
    sven1977 committed Apr 3, 2024
    Configuration menu
    Copy the full SHA
    8db92d7 View commit details
    Browse the repository at this point in the history

  5. Defined 'MultiAgentEpisodeReplayBuffer' class and added '__init__' an… 

    …d 'add' method. Fixed a minor bug in the 'MultiAgentEpisode' b/c the 'agent_module_ids' mapping was not assigned in the 'MultiAgentEpisode.__init__'. Furthermore, added some tests for the new buffer.

Signed-off-by: Simon Zehnder <simon.zehnder@gmail.com>
    @simonsays1980
    simonsays1980 committed Apr 3, 2024
    Configuration menu
    Copy the full SHA
    fc4dc53 View commit details
    Browse the repository at this point in the history

  6. Defined 'sample' method for 'MultiAgentEpisodeReplayBuffer' to sample… 

    … either 'independent' or

'synchronized'. There appears to be an inconsistency between the action getters of the SingleAgentEpisode' and the 'MultiAgentEpisode' - the latter retrieves the timestep one earlier than observations.

Signed-off-by: Simon Zehnder <simon.zehnder@gmail.com>
    @simonsays1980
    simonsays1980 committed Apr 3, 2024
    Configuration menu
    Copy the full SHA
    050b445 View commit details
    Browse the repository at this point in the history

  7. Merge branch 'master' into multi-agent-episode-replay-buffer

    @sven1977
    sven1977 committed Apr 3, 2024
    Configuration menu
    Copy the full SHA
    38e68cf View commit details
    Browse the repository at this point in the history

  8. wip 

    Signed-off-by: sven1977 <svenmika1977@gmail.com>
    @sven1977
    sven1977 committed Apr 3, 2024
    Configuration menu
    Copy the full SHA
    92ed1d7 View commit details
    Browse the repository at this point in the history

Commits on Apr 4, 2024

  1. wip 

    Signed-off-by: sven1977 <svenmika1977@gmail.com>
    @sven1977
    sven1977 committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    db3259b View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' of https://github.com/ray-project/ray into mult… 

    …i_agent_episode_add_slice_api
    @sven1977
    sven1977 committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    4c0195e View commit details
    Browse the repository at this point in the history

  3. unit tests passing 

    Signed-off-by: sven1977 <svenmika1977@gmail.com>
    @sven1977
    sven1977 committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    e3b8468 View commit details
    Browse the repository at this point in the history

  4. LINT 

    Signed-off-by: sven1977 <svenmika1977@gmail.com>
    @sven1977
    sven1977 committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    2b8c1de View commit details
    Browse the repository at this point in the history

  5. unit tests passing, incl. MAEpisode slice fix for scenario with one a… 

    …gent shorter sa-episode and we slice until somewhere past the end of that sa-episode.

Signed-off-by: sven1977 <svenmika1977@gmail.com>
    @sven1977
    sven1977 committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    b08bb21 View commit details
    Browse the repository at this point in the history

  6. fix finalized problem 

    Signed-off-by: sven1977 <svenmika1977@gmail.com>
    @sven1977
    sven1977 committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    007a140 View commit details
    Browse the repository at this point in the history

  7. add proper terminateds/truncateds support to slicing. 

    Signed-off-by: sven1977 <svenmika1977@gmail.com>
    @sven1977
    sven1977 committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    e5e2bf3 View commit details
    Browse the repository at this point in the history

  8. LINT 

    Signed-off-by: sven1977 <svenmika1977@gmail.com>
    @sven1977
    sven1977 committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    403c541 View commit details
    Browse the repository at this point in the history

  9. doctest fix 

    Signed-off-by: sven1977 <svenmika1977@gmail.com>
    @sven1977
    sven1977 committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    d01f6b0 View commit details
    Browse the repository at this point in the history

  10. Merge branch 'master' of https://github.com/ray-project/ray into mult… 

    …i_agent_episode_add_slice_api
    @sven1977
    sven1977 committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    d9eec24 View commit details
    Browse the repository at this point in the history

  11. Merge remote-tracking branch 'sven1977/multi_agent_episode_add_slice_… 

    …api' into multi-agent-episode-replay-buffer

Signed-off-by: Simon Zehnder <simon.zehnder@gmail.com>
    @simonsays1980
    simonsays1980 committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    c6f0323 View commit details
    Browse the repository at this point in the history

  12. Added review from @sven1977 and merged changes in 'MultiAgentEpisode'… 

    … and 'SingleAgentEpisode'.

Signed-off-by: Simon Zehnder <simon.zehnder@gmail.com>
    @simonsays1980
    simonsays1980 committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    d3e2263 View commit details
    Browse the repository at this point in the history

  13. LINTER. 

    Signed-off-by: Simon Zehnder <simon.zehnder@gmail.com>
    @simonsays1980
    simonsays1980 committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    fdaa40b View commit details
    Browse the repository at this point in the history

  14. Merge branch 'multi-agent-episode-replay-buffer' of github.com:simons… 

    …ays1980/ray into multi-agent-episode-replay-buffer

Signed-off-by: Simon Zehnder <simon.zehnder@gmail.com>
    @simonsays1980
    simonsays1980 committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    4f9e991 View commit details
    Browse the repository at this point in the history

Commits on Apr 5, 2024

  1. Merge branch 'master' into multi-agent-episode-replay-buffer 

    Signed-off-by: Simon Zehnder <simon.zehnder@gmail.com>
    @simonsays1980
    simonsays1980 committed Apr 5, 2024
    Configuration menu
    Copy the full SHA
    624ad91 View commit details
    Browse the repository at this point in the history

Commits on Apr 8, 2024

  1. Added getter for metrics and 'get/set_state' for pickling or storing … 

    …a buffer.

Signed-off-by: Simon Zehnder <simon.zehnder@gmail.com>
    @simonsays1980
    simonsays1980 committed Apr 8, 2024
    Configuration menu
    Copy the full SHA
    1188c36 View commit details
    Browse the repository at this point in the history

  2. Merged master. 

    Signed-off-by: Simon Zehnder <simon.zehnder@gmail.com>
    @simonsays1980
    simonsays1980 committed Apr 8, 2024
    Configuration menu
    Copy the full SHA
    68a287c View commit details
    Browse the repository at this point in the history

  3. Refactored getters to override super's getters. In addition added tes… 

    …ts for getters and counters.

Signed-off-by: Simon Zehnder <simon.zehnder@gmail.com>
    @simonsays1980
    simonsays1980 committed Apr 8, 2024
    Configuration menu
    Copy the full SHA
    1d4d535 View commit details
    Browse the repository at this point in the history

  4. Commented out sampling tests until getters in MultiAgentEpisode are f… 

    …ixed.

Signed-off-by: Simon Zehnder <simon.zehnder@gmail.com>
    @simonsays1980
    simonsays1980 committed Apr 8, 2024
    Configuration menu
    Copy the full SHA
    45c5164 View commit details
    Browse the repository at this point in the history

  5. Apply suggestions from code review 

    Signed-off-by: Sven Mika <sven@anyscale.io>
    @sven1977
    sven1977 committed Apr 8, 2024
    Configuration menu
    Copy the full SHA
    df045e3 View commit details
    Browse the repository at this point in the history

  6. Added 'MultiAgentEpisodeReplayBuffer' to the list of registered buffers. 

    Signed-off-by: Simon Zehnder <simon.zehnder@gmail.com>
    @simonsays1980
    simonsays1980 committed Apr 8, 2024
    Configuration menu
    Copy the full SHA
    a2e9339 View commit details
    Browse the repository at this point in the history

  7. Merge branch 'multi-agent-episode-replay-buffer' of github.com:simons… 

    …ays1980/ray into multi-agent-episode-replay-buffer

Signed-off-by: Simon Zehnder <simon.zehnder@gmail.com>
    @simonsays1980
    simonsays1980 committed Apr 8, 2024
    Configuration menu
    Copy the full SHA
    deea2f7 View commit details
    Browse the repository at this point in the history

  8. Added a 'concat_episode' method to MAE and fixed some testing logic. 

    Signed-off-by: Simon Zehnder <simon.zehnder@gmail.com>
    @simonsays1980
    simonsays1980 committed Apr 8, 2024
    Configuration menu
    Copy the full SHA
    372d804 View commit details
    Browse the repository at this point in the history

Commits on Apr 9, 2024

  1. Fixed some minor bugs in the tests. 

    Signed-off-by: Simon Zehnder <simon.zehnder@gmail.com>
    @simonsays1980
    simonsays1980 committed Apr 9, 2024
    Configuration menu
    Copy the full SHA
    5108914 View commit details
    Browse the repository at this point in the history

Commits on Apr 15, 2024

  1. Merged master 

    Signed-off-by: Simon Zehnder <simon.zehnder@gmail.com>
    @simonsays1980
    simonsays1980 committed Apr 15, 2024
    Configuration menu
    Copy the full SHA
    4313a78 View commit details
    Browse the repository at this point in the history

  2. Merged master with new 'MultiAGentEpisode.concat' and uncommented tes… 

    …ts that depend on it.

Signed-off-by: Simon Zehnder <simon.zehnder@gmail.com>
    @simonsays1980
    simonsays1980 committed Apr 15, 2024
    Configuration menu
    Copy the full SHA
    ce8628a View commit details
    Browse the repository at this point in the history

Commits on Apr 23, 2024

  1. Added @sven1977's fix in the 'InfiniteLookbackBuffer' and uncommented… 

    … the test for synchronized sampling'.

Signed-off-by: Simon Zehnder <simon.zehnder@gmail.com>
    @simonsays1980
    simonsays1980 committed Apr 23, 2024
    Configuration menu
    Copy the full SHA
    94a931e View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' into multi-agent-episode-replay-buffer 

    Signed-off-by: Simon Zehnder <simon.zehnder@gmail.com>
    @simonsays1980
    simonsays1980 committed Apr 23, 2024
    Configuration menu
    Copy the full SHA
    a3664cc View commit details
    Browse the repository at this point in the history

  3. Fixed remaining bugs in tests. 

    Signed-off-by: Simon Zehnder <simon.zehnder@gmail.com>
    @simonsays1980
    simonsays1980 committed Apr 23, 2024
    Configuration menu
    Copy the full SHA
    daf9507 View commit details
    Browse the repository at this point in the history