[RLlib] Allow policies to be added/deleted on the fly. #16359
Conversation
…cy_support_add_and_delete
…cy_support_add_and_delete
sven1977
changed the title
| for pid in self.policies}, lw, self.num_sgd_iter, | ||
| self.sgd_minibatch_size, []) | ||
| batch, { | ||
| pid: lw.get_policy(pid) |
There was a problem hiding this comment.
Did you try this for PPO? Don't think this line of code returns the right policies as intended.
| @@ -457,9 +457,9 @@ def test_multi_agent_complex_spaces(self): | |||
| PGTFPolicy, DICT_SPACE, act_space, | |||
| {"model": {"custom_model": "dict_spy"}}), | |||
| }, | |||
| "policy_mapping_fn": lambda a: { | |||
| "policy_mapping_fn": lambda agent_id, **k: { | |||
There was a problem hiding this comment.
Explicitly write out kwargs
There was a problem hiding this comment.
Removed the * and renamed it **kwargs in all tests/examples.
rllib/agents/trainer.py
Outdated
| @PublicAPI | ||
| def add_policy( | ||
| self, | ||
| *, |
There was a problem hiding this comment.
Good idea to remove * so users dont have to manually specific argument name.
rllib/agents/trainer.py
Outdated
| @PublicAPI | ||
| def remove_policy( | ||
| self, | ||
| *, |
rllib/env/wrappers/unity3d_env.py
Outdated
| @@ -313,7 +313,7 @@ def get_policy_configs_for_game( | |||
| action_spaces["Striker"], {}), | |||
| } | |||
|
|
|||
| def policy_mapping_fn(agent_id): | |||
| def policy_mapping_fn(*, agent_id, episode, **kwargs): | |||
There was a problem hiding this comment.
Yeah, good point, I think I'll remove it here.
I do like these *'s to enforce named args to be used. Makes things much more explicit. But I agree, here it would be too much.
rllib/evaluation/sampler.py
Outdated
| if policies is not None: | ||
| deprecation_warning(old="policies") | ||
| if policy_mapping_fn is not None: | ||
| deprecation_warning(old="policy_mapping_fn") |
There was a problem hiding this comment.
Will need documentation changes since Sampler is core part of RLlib
There was a problem hiding this comment.
Added comments that explain that we can use the worker arg, which is passed in as well, for all these 4 deprecated args. E.g. policies = worker.policy_map or policy_mapping_fn = worker.policy_mapping_fn.
…cy_support_add_and_delete
…cy_support_add_and_delete
…cy_support_add_and_delete
…cy_support_add_and_delete
|
@sven1977 I also have a quick follow-up question to this: It seems that it's still required that the agents in |
We currently do not support turn-based games in RLlib (e.g. user1 action -> env obs for user2 -> user2 action -> env obs for user1 -> user1 action, etc..). This PR adds that functionality to RLlib in addition to:
Why are these changes needed?
Related issue number
Checks
scripts/format.shto lint the changes in this PR.