Work In Progress: Crossed out items have been partially implemented.
Prioritised experience replay  persistent advantage learning  bootstrapped  dueling  double  deep recurrent  Q-network  for the Arcade Learning Environment  (and custom environments). Or PERPALB(triple-D)RQN for short...
Additional asynchronous agents :
- One-step Sarsa
- One-step Q-learning
- N-step Q-learning
- Advantage actor-critic
th main.lua to run headless, or
qlua main.lua to display the game. The main options are
-game to choose the ROM (see the ROM directory for more details) and
-mode as either
eval. Can visualise saliency maps , optionally using guided  or "deconvnet"  backpropagation. Saliency map modes are applied at runtime so that they can be applied retrospectively to saved models.
To run experiments based on hyperparameters specified in the individual papers, use
./run.sh <paper> <game> <args>.
<args> can be used to overwrite arguments specified earlier (in the script); for more details see the script itself. By default the code trains on a demo environment called Catch - use
./run.sh demo to run the demo with good default parameters. Note that this code uses CUDA if available, but the Catch network is small enough that it runs faster on CPU. If cuDNN is available, it can be enabled using
-cudnn true; note that by default cuDNN is nondeterministic, and its deterministic modes are slower than cutorch.
In training mode if you want to quit using
Ctrl+C then this will be caught and you will be asked if you would like to save the agent. Note that for non-asynchronous agents the experience replay memory will be included, totalling ~7GB. The main script also automatically saves the last weights (
last.weights.t7) and the weights of the best performing DQN (according to the average validation score) (
In evaluation mode you can create recordings with
-record true (requires FFmpeg); this does not require using
qlua. Recordings will be stored in the videos directory.
Requires Torch7, and can use CUDA and cuDNN if available. Also requires the following extra luarocks packages:
xitari, alewrap and rlenvs can be installed using the following commands:
luarocks install https://raw.githubusercontent.com/lake4790k/xitari/master/xitari-0-0.rockspec luarocks install https://raw.githubusercontent.com/Kaixhin/alewrap/master/alewrap-0-0.rockspec luarocks install https://raw.githubusercontent.com/Kaixhin/rlenvs/master/rocks/rlenvs-scm-1.rockspec
You can use a custom environment (as the path to a Lua file/
rlenvs-namespaced environment) using
-env, as long as the class returned respects the
rlenvs API. One restriction is that the state must be represented as a single tensor (with arbitrary dimensionality), and only a single discrete action must be returned. To prevent massive memory consumption for agents that use experience replay memory, states are discretised to integers ∈ [0, 255], assuming the state is comprised of reals ∈ [0, 1] - this can be disabled with
-discretiseMem false. Visual environments can make use of explicit
-colorSpace options to perform preprocessing for the network.
If the environment has separate behaviour during training and testing it should also implement
evaluate methods - otherwise these will be added as empty methods during runtime. The environment can also implement a
getDisplay method (with a mandatory
getDisplaySpec method for determining screen size) which will be used for displaying the screen/computing saliency maps, where
getDisplay must return a RGB (3D) tensor; this can also be utilised even if the state is not an image (although saliency can only be computed for states that are images). This must be implemented to have a visual display/computing saliency maps. The
-zoom factor can be used to increase the size of small displays.
Environments are meant to be ephemeral, as an instance is created in order to first extract environment details (e.g. state representation), which will later be automatically garbage collected (not under the control of this code).
You can also use a custom model (body) with
-modelBody, which replaces the usual DQN convolutional layers with a custom Torch neural network (as the path to a Lua file/
models-namespaced environment). The class must include a
createBody method which returns the custom neural network. The model will receive a stack of the previous states (as determined by
-histLen), and must reshape them manually if needed. The DQN "heads" will then be constructed as normal, with
-hiddenSize used to change the size of the fully connected layer if needed.
For an example on a GridWorld environment, run
./run.sh demo-grid - the demo also works with
qlua and experience replay agents. The custom environment and network can be found in the examples folder.
- @GeorgOstrovski for confirmation on network usage in advantage operators + note on interaction with Double DQN.
- @schaul for clarifications on prioritised experience replay + dueling DQN hyperparameters.
 Prioritized Experience Replay
 Increasing the Action Gap: New Operators for Reinforcement Learning
 Deep Exploration via Bootstrapped DQN
 Dueling Network Architectures for Deep Reinforcement Learning
 Deep Reinforcement Learning with Double Q-learning
 Deep Recurrent Q-Learning for Partially Observable MDPs
 Playing Atari with Deep Reinforcement Learning
 The Arcade Learning Environment: An Evaluation Platform for General Agents
 Asynchronous Methods for Deep Reinforcement Learning
 Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
 Striving for Simplicity: The All Convolutional Net
 Visualizing and Understanding Convolutional Networks