Develop side channels: migrate reset parameters by vincentpierre · Pull Request #2990 · Unity-Technologies/ml-agents

vincentpierre · 2019-11-27T19:52:40Z

TODO before merging

Documentation (Will do in another PR)

Feature changes :

One on the main breaking changes is that curriculum no longer checks if the reset parameters are in the environment (since parameters can now be added during the simulation)

* Added the side channel for the Engine Configuration. Note that this change does not require modifying a lot of files : - Adding a sender in Python - Adding a receiver in C# - subscribe the receiver to the communicator (here is a one liner in the Academy) - Add the side channel to the Python UnityEnvironment (not represented here) Adding the side channel to the environment would look like such : ```python from mlagents.envs.environment import UnityEnvironment from mlagents.envs.side_channel.raw_bytes_channel import RawBytesChannel from mlagents.envs.side_channel.engine_configuration_channel import EngineConfigurationChannel channel0 = RawBytesChannel() channel1 = EngineConfigurationChannel() env = UnityEnvironment(base_port = 5004, side_channels = [channel0, channel1]) ``` * renamings * addressing comments

…ed by Unity

…deChannel

Co-Authored-By: Chris Elion <chris.elion@unity3d.com>

…ologies/ml-agents into develop-side-channel

…the UnityEnvironment

chriselion · 2019-11-28T00:15:05Z

        SubprocessEnvManager.create_worker = MagicMock()
-        env = SubprocessEnvManager(mock_env_factory, 2)
+        env = SubprocessEnvManager(
+            mock_env_factory, EngineConfig(80, 80, 1, 20.0, -1), 2


How about adding a .default_config() staticmethod to EngineConfig? (yes, you can do that to NamedTuples)

chriselion

Looks good! Can you make sure you update the docs:

--slow removed and new command line args added
Add note about CustomResetParameters removed in the Migration Guide
anywhere else we mention the engine configuration? (maybe just https://github.com/Unity-Technologies/ml-agents/blob/develop/docs/Learning-Environment-Design-Academy.md#academy-properties ?)
But those can be in a subsequent PR.

Co-Authored-By: Chris Elion <chris.elion@unity3d.com>

properties

vincentpierre · 2019-12-02T18:54:04Z

@chriselion

Looks good! Can you make sure you update the docs

There are quite a few changes to the docs. I will make a PR to this branch with the docs edits and merge this afterwards.

* Docs changes for the Side Channels feature * replace deprecated with removed on the CustomResetPratmeters` * Update docs/Python-API.md Co-Authored-By: Chris Elion <chris.elion@unity3d.com> * Update docs/Training-Generalized-Reinforcement-Learning-Agents.md Co-Authored-By: Chris Elion <chris.elion@unity3d.com> * Removing the console outputs in the docs * Update docs/Training-ML-Agents.md Co-Authored-By: Chris Elion <chris.elion@unity3d.com> * replace does not work with ignored * adding a note on side channels * adding some steps to migrate * addressing comments * adding more docs to the LL-API * added a blob on how to access the properties in C# * adding space between ResetParameters * fix typo

chriselion · 2019-12-02T23:15:20Z

    public void SetLaserLengths()
    {
-        m_LaserLength = m_MyAcademy.resetParameters.TryGetValue("laser_length", out m_LaserLength) ? m_LaserLength : 1.0f;
+        m_LaserLength = m_MyAcademy.FloatProperties.GetPropertyWithDefault("laser_length", 1.0f) * m_LaserLength;


I don't think you want * m_LaserLength here.

chriselion · 2019-12-02T23:21:56Z


            if (Communicator != null)
            {
+                Communicator.RegisterSideChannel(new EngineConfigurationChannel());


Doesn't have to be in this PR, but we should think some more about how we expect users to register their own side channels. Might need a virtual call here that they can override.

We could also go the route of having a side channel component (a little bit like what you did with the sensors)

I like that idea, though if the academy won't be in the scene in the future, maybe that doesn't make sense. I'm not sure though.

chriselion · 2019-12-02T23:23:22Z

-            if (m_IsInference)
-            {
-                ConfigureEnvironmentHelper(m_InferenceConfiguration);
-                Monitor.SetActive(true);


I've never used the Monitor so I don't know much about it, but these were the only places we're calling SetActive on it. Do these calls need to be moved somewhere?

The monitor is on by default now, regardless of wether we are training or not.

chriselion · 2019-12-02T23:33:34Z


        parameters = self.data["parameters"]
        for key in parameters:
-            if key not in default_reset_parameters:


Do we have a corresponding check (or at least warning) for this now?

(at reset time, not init time)

Unfortunately, the call on curriculum to get a configuration get_config(self, lesson=None) does not have the environment, or the reset_parameters as input. The curriculum does not have a way to check. I would put the check in FloatPropertiesSideChannel, but then I do not have a way to know if reset has been called on the environment yet.

OK. Doesn't have to be in this PR, but I think we should consider the Academy declaring the properties it knows about, and complain (either on python or C#) if it gets a property it doesn't recognize.

surfnerd · 2019-12-03T19:16:22Z

+* `CustomResetParameters` are now removed.
+* `reset()` on the Low-Level Python API no longer takes a `train_mode` argument. To modify the performance/speed of the engine, you must use an `EngineConfigurationChannel`
+* `reset()` on the Low-Level Python API no longer takes a `config` argument. `UnityEnvironment` no longer has a `reset_parameters` field. To modify float properties in the environment, you must use a `FloatPropertiesChannel`. For more information, refer to the [Low Level Python API documentation](Python-API.md)
+* The Academy no longer has a `Training Configuration` nor `Inference Configuration` field in the inspector. To modify the configuration form the Low-Level Python API, use an `EngineConfigurationChannel`. To modify it during training, use the new command line arguments `--width`, `--height`, `--quality-level`, `--time-scale` and `--target-frame-rate` in `mlagents-learn`.


nit: "To modify the configuration ~~form~~ from the Low-Level Python API"

surfnerd · 2019-12-03T19:20:27Z

+
+#### EngineConfigurationChannel
+An `EngineConfigurationChannel` will allow you to modify how
+fast and how accurate the Unity engine is. For example, modifying the


nit: Not sure accurate is the correct word here. Maybe

An EngineConfiguration will allow you to modify the time scale and graphics quality of the Unity engine.

Perhaps the second sentence is redundant.

vincentpierre and others added 30 commits November 21, 2019 17:34

[WIP] Side Channel initial layout

6749d38

Working prototype for raw bytes

15a8b92

fixing format mistake

02317df

Added some errors and some unit tests in C#

93d7dd0

Logging a message when an unknown side channel number has been receiv…

b65195e

…ed by Unity

Addressing comments

d513e2f

renamings

83beef9

renamings

5daadb0

Adding FloatProperties to the side channels (#2968)

8980e7a

renaming m_SideChannelsDict to m_SideChannel

3aec6b9

renaming and some comments

2dff366

renaming and adding a GetAndClearReceivedMessages() in the RawBytesSi…

c81b324

…deChannel

micro-optimization

37786fd

more errors and some nit

cf867fe

addressing comments

d6cfd52

Using little-endian format in Python

12857e2

adding some comments

5b35693

Code comments

22b790a

some changes and added the unit tests on both Python and C#

efa6de3

removing default default in get default

0d2c337

Update UnitySDK/Assets/ML-Agents/Scripts/SideChannel/SideChannel.cs

55b596e

Co-Authored-By: Chris Elion <chris.elion@unity3d.com>

Update ml-agents-envs/mlagents/envs/side_channel/raw_bytes_channel.py

14689b0

Co-Authored-By: Chris Elion <chris.elion@unity3d.com>

addressing comments

bfaa732

Merge branch 'develop-side-channel' of https://github.com/Unity-Techn…

5f1c174

…ologies/ml-agents into develop-side-channel

fixing tests

c3c2bdf

removing the arguments to reset and the property reset_parameters on …

e70b89c

…the UnityEnvironment

curriculum works but removed the check for reset parameters in the scene

ef1c7ce

processing side channels before the reset command

2b0398d

Removing engine configuration from C#

e76938d

chriselion reviewed Nov 28, 2019

View reviewed changes

Comment thread ml-agents-envs/mlagents/envs/subprocess_env_manager.py Outdated

chriselion reviewed Nov 28, 2019

View reviewed changes

chriselion approved these changes Nov 28, 2019

View reviewed changes

vincentpierre and others added 6 commits December 2, 2019 10:40

Update ml-agents-envs/mlagents/envs/subprocess_env_manager.py

a510304

Co-Authored-By: Chris Elion <chris.elion@unity3d.com>

Removing register callbacks with empty calls in FloarProperties

1b3e8ee

Clamp instead of min max

06aa969

removing the brain names from the environment.py print

b5ffe12

renaming reset_parameters to get

1b70d10

properties

made a default engine config

c971386

vincentpierre added 4 commits December 2, 2019 12:17

bug fix

53d39a0

Resolving merge conflicts

f180c29

Empty commit

41a3ad9

vincentpierre requested a review from surfnerd December 2, 2019 23:11

chriselion reviewed Dec 2, 2019

View reviewed changes

Comment thread UnitySDK/Assets/ML-Agents/Scripts/SideChannel/EngineConfigurationChannel.cs Outdated

bug fixes

076703f

chriselion reviewed Dec 2, 2019

View reviewed changes

Comment thread ml-agents/mlagents/trainers/tests/test_simple_rl.py

surfnerd reviewed Dec 3, 2019

View reviewed changes

addressing comments

5a83b73

surfnerd approved these changes Dec 4, 2019

View reviewed changes

vincentpierre merged commit fa7d381 into develop Dec 4, 2019

delete-merged-branch Bot deleted the develop-side-channel-2 branch December 4, 2019 00:40

BlueFisher mentioned this pull request Jan 3, 2020

Cannot run in inference mode if connected to a python process #3053

Closed

github-actions Bot locked as resolved and limited conversation to collaborators May 17, 2021

Conversation

vincentpierre commented Nov 27, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TODO before merging

Feature changes :

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chriselion left a comment

Choose a reason for hiding this comment

Uh oh!

vincentpierre commented Dec 2, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

vincentpierre commented Nov 27, 2019 •

edited

Loading