Asynchronous plotting for TensorFlow

[ryanjulian]

Task

Presently asynchronous plotting/3D rendering is not supported in the part of rllab based on TensorFlow (sandbox.rocky.tf), but it is supported in the rllab code which uses Theano.

This means that when you turn plotting on for a Theano training session, the plot does not block the training process. The TensorFlow implementation runs the rendering loop directly in the training algorithm (rather than a worker), so it blocks. This makes training using TensorFlow much slower than Theano when plotting is turned on (they are about the same without plotting).

TensorFlow's notion of a session makes this tricky. I'm not 100% sure that there is a solution. If you figure out that it is impossible, or requires rewriting large parts of the repository, email me with what you tried and some explanations why.

Current behavior
3D plotting with MuJoCo in TensorFlow is synchronous, blocks the training process.

Desired behavior
3D plotting with MuJoCo in TensorFlow is asynchronous and does not block the training process (just as with Theano)

Pointers to relevant source

• To run a basic training algorithm which has 3D plots using Theano (you will need to pass plot=True to TRPO):
  https://github.com/ryanjulian/rllab/blob/master/examples/trpo_swimmer.py

• To run a basic training algorithm with TensorFlow:
  https://github.com/ryanjulian/rllab/blob/master/sandbox/rocky/tf/launchers/trpo_cartpole.py
  Note: this does not use a MuJoCo environment/3D plotting, but you just need to change the environment to SwimmerEnv() and it will.

• Asynchronous plotter API used by the Theano code
  https://github.com/ryanjulian/rllab/tree/master/rllab/plotter

• Theano and TF implementations of BatchPolopt (calls the plotter)
  Theano: https://github.com/ryanjulian/rllab/blob/master/rllab/algos/batch_polopt.py
  TF: https://github.com/ryanjulian/rllab/blob/master/sandbox/rocky/tf/algos/batch_polopt.py

• rllab implementation of Serializable (used to transfer objects across multiprocessing queues): https://github.com/ryanjulian/rllab/blob/master/rllab/core/serializable.py

Submission instructions

Fork this repository into you own Github account, and implement this feature in its own branch, based on the master branch. When you are done and would like a code review, send me an email with a link to your feature branch. DO NOT SUBMIT A PULL REQUEST TO THIS REPOSITORY.

Consider this a professional software engineering task, and provide a high-quality solution which does not break existing users, minimizes change, and is stable. Tests are welcome where appropriate. Please always use PEP8 style in your code, and format it using YAPF (with the PEP8 setting).

Notes

• Testing the software requires a freely-available student license for MuJoCo available here. It takes a couple days to get approved, so do it early.
• rllab has setup instructions at http://rllab.readthedocs.io
• You can find examples of how to launch rllab in examples and sandbox/rocky/tf/launchers. Note that everything must run using the run_experiment_lite wrapper with the parameter n_parallel greater than 1 (this triggers multiprocess operation).
• rllab currently has two parallel implementation of the neural network portions of the library. The original is written in Theano and is found in rllab/. The tree sandbox/rocky/tf re-implements classes from the original tree using TensorFlow, and is backwards-compatible with the Theano tree. We are working towards using only one NN library soon, but for now your implementation needs to work in both trees.
• rllab is an upstream dependency to many projects, so it is important we do not break the existing APIs. Adding to APIs is fine as long as there is a good reason.

[jonashen]

Pull request here

[ryanjulian]

See rlworkgroup/garage#22