[RLlib] MultiAgentEpisode for Multi-Agent Reinforcement Learning with the new EnvRunner API.

[simonsays1980]

The MultiAgentEpisode should be used as a container to store all episode information of MultiAgentEnvs.

Specifically it has to track which agent stepped at which environmen step.

Why are these changes needed?

With the change from RolloutWorker to EnvRunner we switch from EpisodeV2 to SingleAgentEpisode and MultiAgentEpisode.

Related issue number

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[sven1977]

Let's explain this here a little (what we discussed on slack):

For MultiAgentEpisode: self.is_terminated should be True, if all agents are terminated
and self.is_truncated should be True if all agents are truncated.
If only one or more agents (but not all!) are terminated or truncated, the MultiAgentEpisode.is_terminated/is_truncated should both be False.
The information about single agents' terminated/truncated states can always be
retrieved from the SingleAgentEpisodes inside the MultiAgent one.
If all agents are either terminated or truncated (but in a mixed fashion: some agents terminated, some truncated): This is currently undefined and could potentially be
a problem (if a user really implemented such a multi-agent env that behaves this way).
My guess is that we should probably then set both is_terminated AND
is_truncated in the MultiAgentEpisode to True. Question here:
Does this have practical relevance?

[simonsays1980]

I think it has for postprocessing: If truncated, the GAE uses for example bootstrapping, while if terminated the value of $o_T$ is set to 0.0.

[sven1977]

I think what you said on slack is correct:

Simon:

But was with the case The environment is not terminated and some agents are terminated earlier on, e.g. timestep 10 agent 1 temrinated (because it fell down a cliff) the other agents are still alive at timestep 100. Do we need here probably in the _init_  of the MultiAgentEpisode`  is_terminated and is_terminated as a list of `MultiAgentDict`s like [{...}, ... , {"agent_1": True, "agent_2": False, ...}, ....] such that we know which agent is still alive and where agent 1 died?

Sven:

Great point! Maybe we should NOT allow users to construct a MA Episode via observations and actions and ... But rather by providing a bunch of agentIDs mapping to SingleAgentEpisodes?

[sven1977]

Just to keep this in mind: Thus far, initializing a new episode (single- or multi-agent) with already existing data is an edge case, that we have - to the best of my knowledge - never used in RLlib. So it's not a super urgent decision we need to make, just to keep this in the back of our heads.

[simonsays1980]

See my comment to this on slack. There might be a hurdle with MA episodes.

[sven1977]

LGTM now. Thanks for this PR, @simonsays1980 ! Let's wait for tests to pass, then we can merge this.