Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[RLlib] External env enhancements + more examples. #16583

Merged
merged 15 commits into from
Jun 23, 2021
Merged

[RLlib] External env enhancements + more examples. #16583

merged 15 commits into from
Jun 23, 2021

Conversation

sven1977
Copy link
Contributor

@sven1977 sven1977 commented Jun 21, 2021 •
edited
Loading

This PR:

  • adds support for running an RLlib policy server with n listen ports (n workers) accepting one or more client connections.
  • makes sure that in case not all ports have active client connections (or incoming data in general), learning on the server can still continue (not blocked) on all other workers using whatever data comes in from different clients.
  • removes the need to create a RandomEnv dummy env on the server, only to specify the action/observation space through that env. The spaces can be given now in the config. In a future PR, action and observation spaces will be sent from connected clients and may not be specified at all anymore on the server side.
  • updates the cartpole server/client example script to show how a multi-port setup can be achieved.
  • also enhances the policy server/client test cases to 3 connected clients (from just 1) connecting to different ports.

Why are these changes needed?

Related issue number

Checks

  • I've run scripts/format.sh to lint the changes in this PR.
  • I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
  • Testing Strategy
    • Unit tests
    • Release tests
    • This PR is not tested :(

sven1977 added 14 commits May 3, 2021 08:54
@sven1977
wip
a18629d
@sven1977
wip
5132d78
@sven1977
Merge branch 'master' of https://github.com/ray-project/ray into exte…
72534ba 
…rnal_env_more_examples
@sven1977
Merge branch 'master' of https://github.com/ray-project/ray into exte…
f95a80f 
…rnal_env_more_examples
@sven1977
wip
0ec6149
@sven1977
wip
bf376b9
@sven1977
Merge branch 'master' of https://github.com/ray-project/ray into exte…
b1f5b94 
…rnal_env_more_examples
@sven1977
Merge branch 'master' of https://github.com/ray-project/ray into exte…
122fd3e 
…rnal_env_more_examples
@sven1977
Merge branch 'master' of https://github.com/ray-project/ray into cql_…
f4382a8 
…fix_bc_loss_term

# Conflicts:
#	rllib/evaluation/sampler.py
@sven1977
Merge branch 'master' of https://github.com/ray-project/ray into cql_…
2ae08de 
…fix_bc_loss_term

# Conflicts:
#	rllib/evaluation/sampler.py
@sven1977
Merge branch 'master' of https://github.com/ray-project/ray into exte…
66bb50d 
…rnal_env_more_examples

# Conflicts:
#	rllib/evaluation/rollout_worker.py
@sven1977
fix and lint
aa8b532
@sven1977
fix and lint
8c1ad58
@sven1977
fix and lint
f0efb3e
@sven1977 sven1977 added the tests-ok The tagger certifies test failures are unrelated and assumes personal liability. label Jun 22, 2021
michaelzhiluo
michaelzhiluo approved these changes Jun 23, 2021
View reviewed changes
rllib/env/tests/test_local_inference.sh
sleep 1
done

# Start client 1 (port 9900).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What does this do?

rllib/env/vector_env.py
"""Renders a single environment.

Args:
index (Optional[int]): An optional sub-env index to render.

Returns:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(w,h,3)

rllib/examples/env/ant_rand_goal.py
from ray.rllib.env.apis.task_settable_env import TaskSettableEnv


class AntRandGoalEnv(gym.utils.EzPickle, MujocoEnv, TaskSettableEnv):
class AntRandGoalEnv(EzPickle, MujocoEnv, TaskSettableEnv):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:) sg

rllib/examples/serving/cartpole_server.py

# TODO: (sven) make these settings unnecessary and get the information
# about the env spaces from the client.
"observation_space": gym.spaces.Box(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't this implicit in the environment defined?

@sven1977 sven1977 merged commit c95dea5 into ray-project:master Jun 23, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@michaelzhiluo michaelzhiluo michaelzhiluo approved these changes

Assignees

@michaelzhiluo michaelzhiluo

Labels
tests-ok The tagger certifies test failures are unrelated and assumes personal liability.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@sven1977 @michaelzhiluo