Skip to content
Switch branches/tags
Go to file

Latest commit


Git stats


Failed to load latest commit information.
Latest commit message
Commit time


This python package is an extension to OpenAI Gym for auxiliary tasks (multitask learning, transfer learning, inverse reinforcement learning, etc.)



Check out the latest code:

git clone

Install without MuJoCo Support

pip3 install -e .

Install with MuJoCo Support

Install MuJoCo according to mujoco-py.

  1. Obtain license for MuJoCo
  2. Download MuJoCo 1.50 binaries
  3. Unzip into mjpro150 directory ~/.mujoco/mjproj150 and place license at ~/.mujoco/mjkey.txt
  4. Finally, install gym-extensions with mujoco-py enabled:
pip3 install -e .[mujoco]

Test Installation

nosetests -v gym_extensions

Possible Issues

Due to the dependency on OpenAI gym you may have some trouble when installing gym on macOS, to remedy:

# as per:
export MACOSX_DEPLOYMENT_TARGET=10.12; pip install -e .

Also, if you get the following error:

>>> import matplotlib.pyplot
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "~/anaconda2/lib/python2.7/site-packages/matplotlib/", line 115, in <module>
    _backend_mod, new_figure_manager, draw_if_interactive, _show = pylab_setup()
  File "~/anaconda2/lib/python2.7/site-packages/matplotlib/backends/", line 32, in pylab_setup
  File "~/anaconda2/lib/python2.7/site-packages/matplotlib/backends/", line 19, in <module>
    raise ImportError("Gtk* backend requires pygtk to be installed.")
ImportError: Gtk* backend requires pygtk to be installed.

the easiest fix is to switch backends for matplotlib. You can do this by setting backend: TkAgg in ~/.config/matplotlib/matplotlibrc or ~/.matplotlib/matplotlibrc


For specific environments (you don't necessarily want to import the whole project)

from gym_extensions.continuous import gym_navigation_2d
env = gym.make("State-Based-Navigation-2d-Map1-Goal1-v0")
from gym_extensions.continuous import mujoco
env = gym.make("HopperWall-v0")

More info

More information will be provided on our doc website:


To contributing environments please use the general directory structure we have in place and provide pull requests. We're still working on making this extension to OpenAI gym the best possible so things may change. Any changes to existing environments should involve an incremental update to the name of the environment (i.e. Hopper-v0 vs. Hopper-v1). If you are not associated with McGill and contribute significantly, please add your association to:



Some of this work borrowed ideas and code from OpenAI rllab and OpenAI Gym. We thank those creators for their work and cite links to reference code inline where possible.


Here's a list of contributors!

Who's using this?

Works that have used this framework include:

Klissarov, Martin, Pierre-Luc Bacon, Jean Harb, and Doina Precup. "Learnings Options End-to-End for Continuous Action Tasks." arXiv preprint arXiv:1712.00004 (2017).

Henderson, Peter, Wei-Di Chang, Pierre-Luc Bacon, David Meger, Joelle Pineau, and Doina Precup. "OptionGAN: Learning Joint Reward-Policy Options using Generative Adversarial Inverse Reinforcement Learning." arXiv preprint arXiv:1709.06683 (2017).

Citation for this work!

If you use this work please use the following citation. If using the Space X environment, please also reference @vBarbaros for credit.

   author = {{Henderson}, P. and {Chang}, W.-D. and {Shkurti}, F. and {Hansen}, J. and 
	{Meger}, D. and {Dudek}, G.},
    title = {Benchmark Environments for Multitask Learning in Continuous Domains},
  journal = {ICML Lifelong Learning: A Reinforcement Learning Approach Workshop},


This repo is intended as an extension for OpenAI Gym for auxiliary tasks (multitask learning, transfer learning, inverse reinforcement learning, etc.)




No releases published


No packages published