Replication of Ape-X (Distributed Prioritized Experience Replay)

This repo replicates the results Horgan et al obtained:

[1] Distributed Prioritized Experience Replay

Our code is based off of code from OpenAI baselines. The original code and related paper from OpenAI can be found here. Their implementation of DQN was modified to use Tensorflow custom ops.

Although Ape-X was originally a distributed algorithm, this implementation was meant to maximize throughput on a single machine. It was optimized for 2 GPUs (data gathering + optimization) but could be modified to use only one. With 2 GPUs and 20~40 CPUs you should be able to achieve human median performance in about 2 hours.

How to run

clone repo

git clone https://github.com/uber-research/ape-x.git

create python3 virtual env

python3 -m venv env
. env/bin/activate

install requirements

pip install tensorflow-gpu gym

Follow the setup under gym_tensorflow/README.md and run ./make to compile the custom ops.

launch experiment

python apex.py --env video_pinball --num-timesteps 1000000000 --logdir=/tmp/agent

Monitor your results with tensorboard

tensorboard --logdir=/tmp/agent

visualize results

python demo.py --env video_pinball --logdir=/tmp/agent