Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[RLlib] Re-instantiate on_episode_created callback for multi-agent (for now). #43779

Merged
merged 1 commit into from
Mar 7, 2024
Merged

[RLlib] Re-instantiate on_episode_created callback for multi-agent (for now). #43779

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 15 additions & 9 deletions rllib/algorithms/algorithm_config.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3925,8 +3925,14 @@ def _validate_new_api_stack_settings(self):
)

# Check and error if `on_episode_created` callback has been overridden on the
# new API stack.
if self.uses_new_env_runners and self.callbacks_class is not DefaultCallbacks:
# new API stack AND this is a single-agent setup (multi-agent does not use
# gym.vector.Env yet and therefore the reset call is still made manually,
# allowing for the callback to be fired).
if (
self.uses_new_env_runners
and not self.is_multi_agent()
and self.callbacks_class is not DefaultCallbacks
):
default_src = inspect.getsource(DefaultCallbacks.on_episode_created)
try:
user_src = inspect.getsource(self.callbacks_class.on_episode_created)
Expand All @@ -3935,13 +3941,13 @@ def _validate_new_api_stack_settings(self):
user_src = default_src
if default_src != user_src:
raise ValueError(
"When using the new API stack with EnvRunners, you cannot override "
"the `DefaultCallbacks.on_episode_created()` method anymore! "
"This particular callback is no longer supported as we are now "
"using gym.vector.Env, which automatically resets individual "
"sub-environments when they are terminated. Override the "
"`on_episode_start` method instead, which gets fired right after "
"the `env.reset()` call."
"When using the new API stack in single-agent and with EnvRunners, "
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah that's a problem I encountered when evaluating against benchmarks, i.e. running a benhcmark against the policy which means the environment needs to be reconfigured for a benchmark run. The solution is to run episodes and after x episodes switch the configuration and call the sampler again.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense. Some users need this opportunity to send config-information to the env before(!) it is reset, but after(!) an episode ended.

We'll have to figure out a way around this for single-agent now, using vector Env. :|

"you cannot override the `DefaultCallbacks.on_episode_created()` "
"method anymore! This particular callback is no longer supported "
"b/c we are using `gym.vector.Env`, which automatically resets "
"individual sub-environments when they are terminated. Instead, "
"override the `on_episode_start` method, which gets fired right "
"after the `env.reset()` call."
)

# This is not compatible with RLModules, which all have a method
Expand Down
2 changes: 2 additions & 0 deletions rllib/env/multi_agent_env_runner.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -350,6 +350,8 @@ def _sample_timesteps(

# Create a new episode instance.
self._episode = self._new_episode()
self._make_on_episode_callback("on_episode_created", self._episode)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could even write self.make_on_episode_callback("on_episode_created") as self._episode is used anyways.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed (in next PR :) )


# Reset the environment.
obs, infos = self.env.reset()
# Add initial observations and infos.
Expand Down
Loading