Code for the paper "Quantifying Transfer in Reinforcement Learning"
Switch branches/tags
Nothing to show
Clone or download
act65 and christopherhesse fix bug in the random agent example script (#9)
* fix bug in the random agent example script

When I ran the original script I got 

Logging to /tmp/openai-2018-12-14-17-36-08-425220
make: Entering directory '/home/act65/repos/coinrun/coinrun'
make: Nothing to be done for 'all'.
make: Leaving directory '/home/act65/repos/coinrun/coinrun'
Traceback (most recent call last):
  File "", line 6, in <module>
    env = make('standard', num_envs=16)
  File "/home/act65/repos/coinrun/coinrun/", line 216, in make
    return CoinRunVecEnv(env_id, num_envs, **kwargs)
  File "/home/act65/repos/coinrun/coinrun/", line 156, in __init__
    obs_space = gym.spaces.Box(0, 255, shape=[self.RES_H, self.RES_W, num_channels], dtype=np.uint8)
AttributeError: module 'gym' has no attribute 'spaces'

Adding `import gym.spaces` fixes the bug ([see issue here](openai/gym#376)). Although maybe it should be imported elsewhere!?

* moved import gym.spaces to

* moved import gym.spaces to
Latest commit 601de52 Dec 14, 2018
Type Name Latest commit message Commit time
Failed to load latest commit information.
coinrun fix bug in the random agent example script (#9) Dec 14, 2018
.gitignore Initial Commit Dec 6, 2018 add MIT license Dec 13, 2018
LICENSE add MIT license Dec 13, 2018 update linux dependencies Dec 13, 2018
coinrun.png Initial Commit Dec 6, 2018
requirements.txt Initial Commit Dec 6, 2018 Initial Commit Dec 6, 2018

Status: Archive (code is provided as-is, no updates expected)

Quantifying Generalization in Reinforcement Learning

[Blog Post] [Paper]

This is code for the environments used in the paper Quantifying Generalization in Reinforcement Learning along with an example training script.

Authors: Karl Cobbe, Oleg Klimov, Chris Hesse, Taehoon Kim, John Schulman



You should install the package in development mode so you can easily change the files. You may also want to create a virtualenv before installing since it depends on a specific version of OpenAI baselines.

This environment has been used on Mac OS X and Ubuntu 16.04 with Python 3.6.

# Linux
apt-get install mpich build-essential qt5-default pkg-config
# Mac
brew install qt open-mpi pkg-config

git clone
cd coinrun
pip install tensorflow  # or tensorflow-gpu
pip install -r requirements.txt
pip install -e .

Note that this does not compile the environment, the environment will be compiled when the coinrun package is imported.

Try it out

Try the environment out with the keyboard:

python -m coinrun.interactive

If this fails, you may be missing a dependency or may need to fix coinrun/Makefile for your machine.

Train an agent using PPO:

python -m coinrun.train_agent --run-id myrun --save-interval 1

After each parameter update, this will save a copy of the agent to ./saved_models/. Results are logged to /tmp/tensorflow by default.

Run parallel training using MPI:

mpiexec -np 8 python -m coinrun.train_agent --run-id myrun

Train an agent on a fixed set of N levels:

python -m coinrun.train_agent --run-id myrun --num-levels N

Train an agent on the same 500 levels used in the paper:

python -m coinrun.train_agent --run-id myrun --num-levels 500

Train an agent on a different set of 500 levels:

python -m coinrun.train_agent --run-id myrun --num-levels 500 --set-seed 13

Continue training an agent from a checkpoint:

python -m coinrun.train_agent --run-id newrun --restore-id myrun

Evaluate an agent's final training performance across N parallel environments. Evaluate K levels on each environment (NxK total levels). Default N=20 is reasonable. Evaluation levels will be drawn from the same set as those seen during training.

python -m coinrun.enjoy --train-eval --restore-id myrun -num-eval N -rep K

Evaluate an agent's final test performance on PxNxK distinct levels. All evaluation levels are uniformly sampled from the set of all high difficulty levels. Although we don't explictly enforce that the test set avoid training levels, the probability of collisions is negligible.

mpiexec -np P python -m coinrun.enjoy --test-eval --restore-id myrun -num-eval N -rep K

Run simulateous training and testing using MPI. Half the workers will train and half will test.

mpiexec -np 8 python -m coinrun.train_agent --run-id myrun --test

View training options:

python -m coinrun.train_agent --help

Watch a trained agent play a level:

python -m coinrun.enjoy --restore-id myrun --hres

Train an agent to play RandomMazes:

python --run-id random_mazes --game-type maze --use-lstm 1

Train an agent to play CoinRun-Platforms, using a larger number of environments to stabilize learning:

python --run-id coinrun_plat --game-type platform --num-envs 96 --use-lstm 1

Example random agent script:

import numpy as np
from coinrun import setup_utils, make

env = make('standard', num_envs=16)
for _ in range(100):
    acts = np.array([env.action_space.sample() for _ in range(env.num_envs)])
    _obs, _rews, _dones, _infos = env.step(acts)

See LICENSES for asset license information.