Replicating "Asynchronous Methods for Deep Reinforcement Learning" (http://arxiv.org/abs/1602.01783)
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
trained_model Replot A3C LSTM scores May 17, 2016
LICENSE Create LICENSE May 11, 2016
README.md Update README.md Feb 25, 2017
a3c.py Support ViZDoom environment May 21, 2016
a3c_ale.py Support A3C LSTM May 17, 2016
ale.py
async.py Add necessary files May 4, 2016
copy_param.py
demo_a3c_ale.py Support A3C LSTM May 17, 2016
demo_a3c_doom.py Support ViZDoom environment May 21, 2016
doom_env.py Support ViZDoom environment May 21, 2016
dqn_head.py Add necessary files May 4, 2016
dqn_phi.py Support A3C LSTM May 17, 2016
environment.py Add necessary files May 4, 2016
init_like_torch.py
nonbias_weight_decay.py Add necessary files May 4, 2016
plot_scores.py
policy.py
policy_output.py
prepare_output_dir.py Fix linebreak problems May 8, 2016
random_seed.py
rmsprop_async.py
run_a3c.py Support ViZDoom environment May 21, 2016
train_a3c_doom.py
v_function.py Add necessary files May 4, 2016

README.md

Async-RL

(2017/02/25) Now the A3C implementation in this repository has been ported into ChainerRL, a Chainer-based deep reinforcement learning library, with some enhancement such as support for continuous actions by Gaussian policies and n-step Q-learning, so I recommend using it instead of this repository.

A3C FF playing Breakout A3C LSTM playing Space Invaders

This is a repository where I attempt to reproduce the results of Asynchronous Methods for Deep Reinforcement Learning. Currently I have only replicated A3C FF/LSTM for Atari.

Any feedback is welcome :)

Supported Features

  • A3C FF/LSTM (only for discrete action space)
  • Atari environment
  • ViZDoom environment (experimental)

Current Status

A3C FF

I trained A3C FF for ALE's Breakout with 36 processes (AWS EC2 c4.8xlarge) for 80 million training steps, which took about 17 hours. The mean and median of scores of test runs along training are plotted below. Ten test runs for every 1 million training steps (counted by the global shared counter). The results seems slightly worse than theirs.

The trained model is uploaded at trained_model/breakout_ff/80000000_finish.h5, so you can make it to play Breakout by the following command:

python demo_a3c_ale.py <path-to-rom> trained_model/breakout_ff/80000000_finish.h5

The animation gif above is the episode I cherry-picked from 10 demo runs using that model.

A3C LSTM

I also trained A3C LSTM for ALE's Space Invaders in the same manner with A3C FF. Training A3C LSTM took about 24 hours for 80 million training steps.

The trained model is uploaded at trained_model/space_invaders_lstm/80000000_finish.h5, so you can make it to play Space Invaders by the following command:

python demo_a3c_ale.py <path-to-rom> trained_model/space_invaders_lstm/80000000_finish.h5 --use-lstm

The animation gif above is the episode I cherry-picked from 10 demo runs using that model.

Implementation details

I received a confirmation about their implementation details and some hyperparameters by e-mail from Dr. Mnih. I summarized them in the wiki: https://github.com/muupan/async-rl/wiki

Requirements

  • Python 3.5.1
  • chainer 1.8.1
  • cached-property 1.3.0
  • h5py 2.5.0
  • Arcade-Learning-Environment

Training

python a3c_ale.py <number-of-processes> <path-to-atari-rom> [--use-lstm]

a3c_ale.py will save best-so-far models and test scores into the output directory.

Unfortunately it seems this script has some bug now. Please see the issues #5 and #6. I'm trying to fix it.

Evaluation

python demo_a3c_ale.py <path-to-atari-rom> <trained-model> [--use-lstm]

Similar Projects