Save / restore environment (wishlist)

[hagrid67]

Would it be possible to provide functionality to save and restore a running environment? Ie read the complete state of the environment, all the bone/object locations and orientations, velocities and angular velocities, and any state associated with the muscles, and then recreate (or reset) another environment to the same state.

This means more than just starting two environments with the same seed - it means copying an environment which has already evolved over lots of steps and actions.

It should be possible to copy the env to another machine (ie OS instance).

This would be to support the "vine" method in Section 5.2 in the 2017 Schulman TRPO paper which uses a "short rollout", in order to test the effect of several different actions from a given state.

I suggest It's probably not necessary to "read" the contents of the serialised state, in case it's easier or desirable to make it opaque. The main requirement is essentially to clone a running environment. However it would be better to do this by providing the serialised state, so "we" users/programmers/competitors can manage the copy. (ie it wouldn't be so helpful if there was simply a clone() call returning a handle to a new env running on the same machine, as we wouldn't then be able to manage the load-balancing)

[hagrid67]

See this similar issue for OpenAI - openai/gym#402

Here is the relevant diagram from Schulman 2017 TRPO (CRN means common random numbers):

[kidzik]

These are all very good points, thanks for sharing. Rollback is possible in the current environment. I made an experimental example here https://github.com/stanfordnmbl/osim-rl/blob/rollback/tests/test.rollback.py . Saving to a file and restoring is, unfortunately, tricky and we won't be able to support it.

Since the environment is deterministic, for a quick hack, one can record trajectories of actions, which led to a certain point. For generating the same obstacles and muscles parameters, it's enough to use the same seed and difficulty.

[hagrid67]

Thanks Łukasz, I think your example should be enough to allow us to implement "consecutive" branching, if not parallel. It means we don't need to replay actions to get back to the same point, so we can get the environment to try out some different actions / rollouts, without significant computational overhead - just a bit of inflexibility around the timing. If I'm understanding correctly :)

[kidzik]

It seems this is actually to large extent doable now, please refer to #125 and this test 7ecae69
While it's supported, the documentation is beyond the scope of osim-rl for now -- you can find the details on the SimBody side https://github.com/simbody/simbody