Bullet support

[ryanjulian]

No description provided.

[ryanjulian]

We would like to add support for the Bullet physics engine to rllab. Thankfully, the Bullet team have recently provided Python bindings in the form of pybullet, and even provides examples of how to implement the gym.Env interface (from OpenAI Gym) using pyBullet.

This task is to add pybullet to the rllab conda environment, and implement a class (similar to GymEnv, e.g. BulletEnv) which allows any rllab algorithm to learn against pybullet environments. You will also need to implement the plot interface, if pybullet does not already, which shows the user a 3D animation of the environment. Essentially, you should duplicate the experience of running one of the MuJoCo-based examples (e.g. trpo_swimmer.py), but using a Bullet environment instead. You should include examples (in examples/ and sandbox/rocky/tf/launchers/) of launcher scripts which use an algorithms (suggestion: TRPO) to train the KukaGymEnv environment.

This is conceptually the same as GymEnv, which allows rllab users to import any OpenAI Gym environment and learn against them. In fact, pybullet environments implement the Gym interface, so in theory we should be done as soon as we can import pybullet. In practice, our constructor for Gym environments only takes the string name (e.g. "Humanoid-v1") of a Gym environment, not the class of a Gym environment. The pybullet environments do not have string shortcuts because they are not part of the official Gym repository. Furthermore, we'd like to use other unofficial Gym environments in rllab, but it is currently difficult for the same reason.

So you might structure this task as two pull requests (1) adding pybullet to the conda environment and (2) Modifying GymEnv to support arbitrary environments which implement the gym.Env interface (attempted in #12).

Consider this a professional software engineering task, and provide a high-quality solution which does not break existing users, minimizes change, and is stable. Please always use PEP8 style in your code, and format it using YAPF (with the PEP8 setting). Submit your pull request against the integration branch of this repository.

Some notes:

• You can find examples of how to launch rllab in examples and sandbox/rocky/tf/launchers. Note that everything must run using the run_experiment_lite wrapper.
• rllab currently has two parallel implementation of the neural network portions of the library. The original is written in Theano and is found in rllab/. The tree sandbox/rocky/tf re-implements classes from the original tree using TensorFlow, and is backwards-compatible with the Theano tree. We are working towards using only one NN library soon, but for now your implementation needs to work in both trees.
• rllab is an upstream dependency to many projects, so it is important we do not break the existing APIs. Adding to APIs is fine as long as there is a good reason.

[Anirudhkashi]

So, if I understand this correctly,
PR 1) Add pybullet to environment.yml, create a BulletEnv class similar to GymEnv, add examples of existing environments in examples folder.
PR 2) Modifying GymEnv to accept env class object, instead of the string.

[ryanjulian]

You decide. The goal is to enable Bullet support. Try it out and see what works and makes sense.

As I noted above, since Bullet implements the Gym interface, a lot of the GymEnv code is probably reusable to achieve this, but it might require some modification. If you modify GymEnv, you still need to keep your changes backwards-compatible with the rest of rllab (and test the backwards compatibility)

If you make changes to existing code to support a new feature, a good practice is to split the change into two parts: one for the underlying changes (which enable the new feature) and one for the new feature.

[Anirudhkashi]

Here are the changes I have made as of now: Diff.

1. I have added the pybullet to conda env, increased OpenAI gym version.
2. Have provided a way to register any class into the pybullet env: BulletEnv
3. Have added an example trpo_bullet_kuka.py in examples folder.

Have not added the tf version yet. I am stuck with an issue in which the plot window is duplicating.
Can you please check if this is the correct way?

[ryanjulian]

Please create a pull request so I can leave comments!

On Fri, Mar 23, 2018 at 8:09 PM Anirudh Kashi ***@***.***> wrote: Here are the changes I have made as of now: Diff <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_ryanjulian_rllab_compare_integration...Anirudhkashi-3Apybulletsupport&d=DwMFaQ&c=clK7kQUTWtAVEOVIgvi0NU5BOUHhpN0H8p7CSfnc_gI&r=VEuK-fS6-fUHjpi4P-BnNQ&m=1ihPKFGBDSP5AJd6htW7CXyi3an40qP0-Ni-zpyE3KI&s=S2kbE6y-dYHg1vHyK8UhaIYlxE2Hu28loxx9EEt36_0&e=> . 1. I have added the pybullet to conda env, increased OpenAI gym version. 2. Have provided a way to register any class into the pybullet env: BulletEnv 3. Have added an example trpo_bullet_kuka.py in examples folder. Have not added the tf version yet. I am stuck with an issue in which the plot window is duplicating. Can you please check if this is the correct way? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_ryanjulian_rllab_issues_5-23issuecomment-2D375842344&d=DwMFaQ&c=clK7kQUTWtAVEOVIgvi0NU5BOUHhpN0H8p7CSfnc_gI&r=VEuK-fS6-fUHjpi4P-BnNQ&m=1ihPKFGBDSP5AJd6htW7CXyi3an40qP0-Ni-zpyE3KI&s=_MBTrdjlxAPtVdUGQH6as698c5-Omu7tyiG8tLlVb80&e=>, or mute the thread <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_ABT-5F2wpb-5F2si737K55NmtfX9OvhZNKJeks5thbjwgaJpZM4SXXZG&d=DwMFaQ&c=clK7kQUTWtAVEOVIgvi0NU5BOUHhpN0H8p7CSfnc_gI&r=VEuK-fS6-fUHjpi4P-BnNQ&m=1ihPKFGBDSP5AJd6htW7CXyi3an40qP0-Ni-zpyE3KI&s=XdcWrB7kAkUyC3341qqwvYHpaUwlVLKdjXHlS565aj0&e=> .

-- PhD Student USC CLMC/RESL (510) 859-3224 rjulian@usc.edu

[ryanjulian]

See my comments here. #22 (review)

On Sun, Mar 25, 2018 at 12:21 PM Ryan Julian ***@***.***> wrote: Please create a pull request so I can leave comments! On Fri, Mar 23, 2018 at 8:09 PM Anirudh Kashi ***@***.***> wrote: > Here are the changes I have made as of now: Diff > <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_ryanjulian_rllab_compare_integration...Anirudhkashi-3Apybulletsupport&d=DwMFaQ&c=clK7kQUTWtAVEOVIgvi0NU5BOUHhpN0H8p7CSfnc_gI&r=VEuK-fS6-fUHjpi4P-BnNQ&m=1ihPKFGBDSP5AJd6htW7CXyi3an40qP0-Ni-zpyE3KI&s=S2kbE6y-dYHg1vHyK8UhaIYlxE2Hu28loxx9EEt36_0&e=> > . > > 1. I have added the pybullet to conda env, increased OpenAI gym > version. > 2. Have provided a way to register any class into the pybullet env: > BulletEnv > 3. Have added an example trpo_bullet_kuka.py in examples folder. > > Have not added the tf version yet. I am stuck with an issue in which the > plot window is duplicating. > Can you please check if this is the correct way? > > — > You are receiving this because you authored the thread. > Reply to this email directly, view it on GitHub > <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_ryanjulian_rllab_issues_5-23issuecomment-2D375842344&d=DwMFaQ&c=clK7kQUTWtAVEOVIgvi0NU5BOUHhpN0H8p7CSfnc_gI&r=VEuK-fS6-fUHjpi4P-BnNQ&m=1ihPKFGBDSP5AJd6htW7CXyi3an40qP0-Ni-zpyE3KI&s=_MBTrdjlxAPtVdUGQH6as698c5-Omu7tyiG8tLlVb80&e=>, > or mute the thread > <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_ABT-5F2wpb-5F2si737K55NmtfX9OvhZNKJeks5thbjwgaJpZM4SXXZG&d=DwMFaQ&c=clK7kQUTWtAVEOVIgvi0NU5BOUHhpN0H8p7CSfnc_gI&r=VEuK-fS6-fUHjpi4P-BnNQ&m=1ihPKFGBDSP5AJd6htW7CXyi3an40qP0-Ni-zpyE3KI&s=XdcWrB7kAkUyC3341qqwvYHpaUwlVLKdjXHlS565aj0&e=> > . > -- PhD Student USC CLMC/RESL (510) 859-3224 ***@***.***

-- PhD Student USC CLMC/RESL (510) 859-3224 rjulian@usc.edu

[Anirudhkashi]

Okay, I will send the PR after I make these changes. I have added few questions for required clarifications on the comments.
Review

[ryanjulian]

See rlworkgroup/garage#46