[RLlib] Issue 39453: PettingZoo wrappers should use correct multi-agent dict action- and observation spaces.

[sven1977]

See issue: #39453

Solution:
Change both PettingZoo adapters of RLlib to abide to RLlib's multi-agent action- and observation space convention (gym.spaces.Dict mapping agent Ids to individual spaces).

Closes: #39453 and #32889

Why are these changes needed?

Related issue number

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[chrisyeh96]

Thanks @sven1977 for responding so quickly to #39453!

I have a suggestion: can we make RLLib's PettingZooEnv and ParallelPettingZooEnv wrappers support PettingZoo environments where agents have different action_spaces and observation_spaces? This would actually simplify the RLLib wrapper code quite significantly.

For example, here's what the __init__() method for ParallelPettingZooEnv would look like:

class ParallelPettingZooEnv(MultiAgentEnv):
    def __init__(self, env):
        super().__init__()
        self.par_env = env
        self.par_env.reset()
        self._agent_ids = set(self.par_env.agents)

        self.observation_space = gym.spaces.Dict(self.par_env.observation_spaces)
        self.action_space = gym.spaces.Dict(self.par_env.action_spaces)

[sven1977]

Hey @chrisyeh96 , thanks for the hint. Changed this to be even more flexible now for both parallel and non-parallel env wrappers.

[chrisyeh96]

self._agent_ids should be a set, so this line should be changed to:

self._agent_ids = set(self.env.agents)

[sven1977]

done

[chrisyeh96]

self._agent_ids should be a set, so this line should be changed to:

self._agent_ids = set(self.par_env.agents)

[sven1977]

done

[chrisyeh96]

Thanks @sven1977 for the changes! I've suggested 2 minor changes in my code review.

There's also one other place that should be updated. The docstring for PettingZooEnv currently still says that the wrapper only supports environments with homogeneous action / observation spaces. With this pull request, this comment should be removed. See below:

ray/rllib/env/wrappers/pettingzoo_env.py

Lines 23 to 27 in 5a4389b

	1. All agents have the same action_spaces and observation_spaces.
	Note: If, within your aec game, agents do not have homogeneous action /
	observation spaces, apply SuperSuit wrappers
	to apply padding functionality: https://github.com/Farama-Foundation/
	SuperSuit#built-in-multi-agent-only-functions

Thanks for your work on this!

[chrisyeh96]

Thanks @sven1977! Looking forward to seeing this change merged. Do you know if it will be included in the next release of RLLib?

[sven1977]

Hey @chrisyeh96 , yes, it'll be part of Ray 2.8, though, and won't make it into the upcoming 2.7 anymore.

[chrisyeh96]

Understood. Any idea on what the timeline for Ray 2.8 will be? For me, this issue is quite important.

I have a project called SustainGym where I've made 5 RL environments related to energy / sustainability applications. 3 of the 5 environments are multi-agent RL environments following the PettingZoo API.

Currently, I include RLLib as a dependency of SustainGym, because I have special versions of my environments to fit the RLLib MultiAgentEnv API. In theory, SustainGym should not depend on RLLib, because a RL environment package should be agnostic to the RL algorithm. Once this pull request gets merged, I can remove the RLLib dependency and instead direct SustainGym users to simply use the RLLib wrapper around the SustainGym PettingZoo env if they want to use RLLib algorithms.

[sven1977]

2.8 will be released a few weeks after the Ray Summit (which is next week). So roughly beginning to mid October.