[RLlib] Issue 21489: Unity3D env lacks group rewards.

[grypesc]

## Why are these changes needed?

## Related issue number

Checks

• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :( - It is tested for v1.9.1

[grypesc]

Works for 1.9.1. I was not able to test it for the 1.12.0, as I am getting a bug that was IMHO introduced in some other commit:
(Steps to reproduce: python rllib/examples/unity3d_env_local.py --env 3DBall)
(PPOTrainer pid=17856)ray::PPOTrainer.init() File "C:\Projects\uragun-rl\venv\lib\site-packages\ray\rllib\evaluation\worker_set.py", line 162, in init (pid=17856, ip=127.0.0.1, repr=PPOTrainer)
File "python\ray_raylet.pyx", line 656, in ray._raylet.execute_task
File "python\ray_raylet.pyx", line 697, in ray._raylet.execute_task
File "python\ray_raylet.pyx", line 663, in ray._raylet.execute_task
File "python\ray_raylet.pyx", line 667, in ray._raylet.execute_task
File "python\ray_raylet.pyx", line 614, in ray._raylet.execute_task.function_executor
File "C:\Projects\uragun-rl\venv\lib\site-packages\ray_private\function_manager.py", line 701, in actor_method_executor
return method(__ray_actor, *args, **kwargs)
File "C:\Projects\uragun-rl\venv\lib\site-packages\ray\util\tracing\tracing_helper.py", line 462, in _resume_span
return method(self, *_args, **_kwargs)
File "C:\Projects\uragun-rl\venv\lib\site-packages\ray\rllib\agents\trainer.py", line 830, in init
super().init(
File "C:\Projects\uragun-rl\venv\lib\site-packages\ray\tune\trainable.py", line 149, in init
self.setup(copy.deepcopy(self.config))
File "C:\Projects\uragun-rl\venv\lib\site-packages\ray\util\tracing\tracing_helper.py", line 462, in _resume_span
return method(self, *_args, **_kwargs)
File "C:\Projects\uragun-rl\venv\lib\site-packages\ray\rllib\agents\trainer.py", line 911, in setup
self.workers = WorkerSet(
File "C:\Projects\uragun-rl\venv\lib\site-packages\ray\rllib\evaluation\worker_set.py", line 162, in init
self._local_worker = self._make_worker(
File "C:\Projects\uragun-rl\venv\lib\site-packages\ray\rllib\evaluation\worker_set.py", line 567, in _make_worker
worker = cls(
File "C:\Projects\uragun-rl\venv\lib\site-packages\ray\rllib\evaluation\rollout_worker.py", line 507, in init
check_env(self.env)
File "C:\Projects\uragun-rl\venv\lib\site-packages\ray\rllib\utils\pre_checks\env.py", line 63, in check_env
check_multiagent_environments(env)
File "C:\Projects\uragun-rl\venv\lib\site-packages\ray\rllib\utils\pre_checks\env.py", line 197, in check_multiagent_environments
_check_if_element_multi_agent_dict(env, reset_obs, "reset()")
File "C:\Projects\uragun-rl\venv\lib\site-packages\ray\rllib\utils\pre_checks\env.py", line 475, in _check_if_element_multi_agent_dict
raise ValueError(error)
ValueError: The element returned by reset() has agent_ids that are not the names of the agents in the env.
Agent_ids in this MultiAgentDict: ['3DBall?team=0_0', '3DBall?team=0_1', '3DBall?team=0_2', '3DBall?team=0_3', '3DBall?team=0_4', '3DBall?team=0_5', '3DBall?team=0_6', '3DBall?team=0_7', '3DBall?team=0_8', '3DBall?team=0_9', '3DBall?te
am=0_10', '3DBall?team=0_11']
Agent_ids in this env:['all']. You likley need to add the private attribute _agent_ids to your env, which is a set containing the ids of agents supported by your env.

[gjoliver]

this issue is likely related #22943

[grypesc]

Well, the number of agents in UnityEnv changes during episodes. I don't see the reason for this check. The only thing that matters are policies.

[sven1977]

Hey @grypesc thanks for this PR! A very nice enhancement and fix for Unity3D :)

[sven1977]

Merged with upstream master and fixed the LINTer errors. Just waiting for tests to pass, then will merge
@grypesc ^