[rllib][minor] Fix multiple Unity3DEnvs trying to connect to the same custom port

[yurirocha15]

Why are these changes needed?

When running the Unity3D env with several workers while using a custom port, every environment will try to connect to the same port.

The current logic increments the base_port while keeping the worker_id fixed. This works without issues if the default base port is used. However, if the user decides to use a custom port, this will override the base_port, trying to create several environments using the same custom port. The first environment will succeed, while the remaining environments will be trapped in an infinite loop.

This PR fixes that by keeping the base_port fixed and incrementing the worker_id instead.

This PR also changed the base port value when calling a compiled app (to 5005, the same default as ml-agents uses internally. The editor port stays the same). There is a use case where the user might be running a distributed training in the background while debugging/prototyping/testing new features in the editor. If the first compiled environment is using the editor default port, the editor will fail to connect.

Related issue number

Checks

• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(
    Added unit tests to check the default ports, and to check if two consecutive calls would have different base_port + worker_id (for either default and user-defined ports).
    Also tested using the rllib/examples/unity3d_env_local.py.

[sven1977]

Awesome! Thanks for the fix @yurirocha15 !
Could you do one more thing and add your new tests to the rllib/BUILD file?
Somewhere here?

py_test(
    name = "env/wrappers/tests/test_unity3d_env",
    tags = ["env"],
    size = "small",
    srcs = ["env/wrappers/tests/test_unity3d_env.py"]
)

# .... before this one here vv

py_test(
    name = "env/wrappers/tests/test_recsim_wrapper",
    tags = ["env"],
    size = "small",
    srcs = ["env/wrappers/tests/test_recsim_wrapper.py"]
)

[yurirocha15]

Travis is failing to find mlagents_envs (which is imported in the tests). I guess it might be necessary to add the mlagents requirements to python/requirements_rllib.txt.

[yurirocha15]

@sven1977 Would it be ok to add mlagents to rllib requirements?

[sven1977]

@YuriRocha thanks a lot for adding the test.

Yes, you can add

mlagents
mlagents_env

to ray/python/requirements_rllib.txt. Then it should work.

[sven1977]

I added the two modules to the requirements_rllib.txt. Just waiting for tests now to pass. ...

[sven1977]

Thanks for the fix @yurirocha15 ! Looks good now.

[yurirocha15]

@sven1977 Thank you!