Replicate Prioritized Experience Replay's reported performance improvements

[muupan]

Missing details

• "all weights w_i were scaled so that max_i w_i = 1". Is max_i w_i computed over a minibatch or the whole buffer?
• What is the value of epsilon that is added to absolute TD errors?

[muupan]

I asked the author via email and confirmed

• max_i w_i is computed over a minibatch
• epsilon=0.01

[muupan]

breakout: 🆗

[muupan]

space invaders: 🆗

[muupan]

seaquest: a bit worse

[muupan]

beam rider: a bit worse

[muupan]

asterix: 🆗

[muupan]

qbert: 🆗

[muupan]

I compared "Double DQN tuned prioritized lr/4" vs "proportional" in the paper.
similar results: breakout, space invaders, asterix, qbert
a bit worse than the paper: seaquest, beam rider

They seem to use 500,000 frames instead of 108,000 frames for evaluation (B.2.3 of http://arxiv.org/abs/1511.05952), so trying 500,000 frames may fill the gap.

[prabhatnagarajan]

Tested 7 games: Breakout, Space Invaders, Seaquest, Asterix, Beam Rider, Qbert.

Results:

Breakout: 🆗
Space Invaders: 🆗
Seaquest: worse
Beam Rider: slightly worse
Asterix:🆗
Qbert: 🆗

@muupan suggested that it appears that the evaluations used in the paper permitted longer episodes during evaluations, potentially explaining our slightly worse performance in 2 domains.