This repo replaces Mujoco by PyBullet

(WHY? Because poor grads need open source tools!)

Modification:

1. Create envs folder, copy Mujoco xml files to envs/xmls, and change xml files for compatiblility.

   (Refer to This Note)

2. Change ModularEnv.py, using PyBullet instead of Mujoco.

   (ALERT: observation state, alive condition have been changed!)

   2.1 Observation state: PyBullet provides less information for the client, there's no rotational velocity. And one-hot mask information for each body part has been changed to the index number of that part.

   2.2 Alive condition: Gym has a very specific alive condition including how high is the center of mass, the angle of the body. In this context, we could have arbitrary body, so I changed the alive condition to: if the "torso"(root) touch the ground, then done. (self.robot.feet_contact[0])

3. Add --render to available argument lists, in case you want to see the robot instead of just numbers.

   (Use Ctrl+Mouse Left to adjust camera.)

   (too fast? Uncomment ModularEnv.py #104, so it'll sleep a while at every step)

4. To start, try:

# clone this repo
git clone https://github.com/liusida/modular-rl.git modular-rl-bullet
cd modular-rl-bullet
# create a clean environment
conda create -n modular-rl-bullet python=3.6
source activate modular-rl-bullet
# install python dependencies, this will take a while
pip install -r requirement.txt
cd src
# run all 3 hoppers
python main.py --morphologies=hopper --render

If you have trouble installing mp4py, use conda install:

conda install -c anaconda mpi4py

Original Readme:

One Policy to Control Them All:
Shared Modular Policies for Agent-Agnostic Control

ICML 2020

[Project Page] [Paper] [Demo Video] [Long Oral Talk]

Wenlong Huang¹, Igor Mordatch², Deepak Pathak^{3 4}

¹University of California, Berkeley, ²Google Brain, ³Facebook AI Research, ⁴Carnegie Mellon University

This is a PyTorch-based implementation of our Shared Modular Policies. We take a step beyond the laborious training process of the conventional single-agent RL policy by tackling the possibility of learning general-purpose controllers for diverse robotic systems. Our approach trains a single policy for a wide variety of agents which can then generalize to unseen agent shapes at test-time without any further training.

If you find this work useful in your research, please cite using the following BibTeX:

@inproceedings{huang2020smp,
  Author = {Huang, Wenlong and
  Mordatch, Igor and Pathak, Deepak},
  Title = {One Policy to Control Them All:
  Shared Modular Policies for Agent-Agnostic Control},
  Booktitle = {ICML},
  Year = {2020}
  }

Setup

Requirements

• Python-3.6
• PyTorch-1.1.0
• CUDA-9.0
• CUDNN-7.6
• MuJoCo-200: download binaries, put license file inside, and add path to .bashrc

Setting up repository

git clone https://github.com/huangwl18/modular-rl.git
cd modular-rl/
python3.6 -m venv mrEnv
source $PWD/mrEnv/bin/activate

Installing Dependencies

pip install --upgrade pip
pip install -r requirements.txt

Running Code

Flags and Parameters	Description
--morphologies <List of STRING>	Find existing environments matching each keyword for training (e.g. walker, hopper, humanoid, and cheetah; see examples below)
--custom_xml <PATH>	Path to custom xml file for training the modular policy. When <PATH> is a file, train with that xml morphology only. When <PATH> is a directory, train on all xml morphologies found in the directory.
--td	Enable top-down message passing (pass --td --bu for both-way message passing)
--bu	Enable bottom-up message passing (pass --td --bu for both-way message passing)
--expID <INT>	Experiment ID for creating saving directory
--seed <INT>	(Optional) Seed for Gym, PyTorch and Numpy

Train with existing environment

• Train both-way SMP on Walker++ (12 variants of walker):

python main.py --expID 001 --td --bu --morphologies walker

• Train both-way SMP on Humanoid++ (8 variants of 2d humanoid):

python main.py --expID 002 --td --bu --morphologies humanoid

• Train both-way SMP on Cheetah++ (15 variants of cheetah):

python main.py --expID 003 --td --bu --morphologies cheetah

• Train both-way SMP on Hopper++ (3 variants of hopper):

python main.py --expID 004 --td --bu --morphologies hopper

• To train both-way SMP for only one environment (e.g. walker_7_main), specify the full name of the environment without the .xml suffix:

python main.py --expID 005 --td --bu --morphologies walker_7_main

To run with one-way message passing, disable --td for bottom-up-only message passing or disable --bu for top-down-only message passing. To run without any message passing, disable both --td and --bu.

Train with custom environment

• Train both-way SMP for only one environment:

python main.py --expID 006 --td --bu --custom_xml <PATH_TO_XML_FILE>

• Train both-way SMP for multiple environments (xml files must be in the same directory):

python main.py --expID 007 --td --bu --custom_xml <PATH_TO_XML_DIR>

Note that the current implementation assumes all custom MuJoCo agents are 2D planar and contain only one body tag with name torso attached to worldbody.

Visualization

• To visualize all walker environments with the both-way SMP model from experiment expID 001:

python visualize.py --expID 001 --td --bu --morphologies walker

• To visualize only walker_7_main environment with the both-way SMP model from experiment expID 001:

python visualize.py --expID 001 --td --bu --morphologies walker_7_main

Provided Environments

Walker
walker_2_main	walker_3_main	walker_4_main	walker_5_main	walker_6_main	walker_7_main
walker_2_flipped	walker_3_flipped	walker_4_flipped	walker_5_flipped	walker_6_flipped	walker_7_flipped

2D Humanoid
humanoid_2d_7_left_arm	humanoid_2d_7_left_leg	humanoid_2d_7_lower_arms	humanoid_2d_7_right_arm
humanoid_2d_7_right_leg	humanoid_2d_8_left_knee	humanoid_2d_8_right_knee	humanoid_2d_9_full

Cheetah
cheetah_2_back	cheetah_2_front	cheetah_3_back	cheetah_3_balanced	cheetah_3_front
cheetah_4_allback	cheetah_4_allfront	cheetah_4_back	cheetah_4_front	cheetah_5_back
cheetah_5_balanced	cheetah_5_front	cheetah_6_back	cheetah_6_front	cheetah_7_full

Hopper
hopper_3	hopper_4	hopper_5

Note that each walker agent has an identical instance of itself called flipped, for which SMP always flips the torso message passed to both legs (e.g. the message that is passed to the left leg in the main instance is now passed the right leg).

Acknowledgement

The TD3 code is based on this open-source implementation. The code for Dynamic Graph Neural Networks is adapted from Modular Assemblies (Pathak*, Lu* et al., NeurIPS 2019).