DQN: Optimizer

[Kautenja]

• Adam
• Nadam
• RMSprop

[Kautenja]

RMSprop with default DeepMind parameters is complete garbage. After 5,000,000 frames it raised the average score per episode to only -15 for Pong. For reference, Adam can converge to nearly perfect games (average score of +16) in the same amount of time. Long story short, RMSprop in Keras is either different from what they used, or Adam is just plain better. No more exploration will be done with RMSprop.

EDIT: fix some spelling, grammar, etc.

[Kautenja]

Nadam and Adam produce similar results. Nadam seems to take a small amount of extra time. Adam will be used from here on out. Notebooks are searching for a solid learning rate to lock for remaining experiments.

EDIT: Nadam just achieved a high average score of 18.1. Rethinking this with more notebooks

[Kautenja]

high learning rates seem to cause an explosion of gradients in the early stages. (i.e. 1e-4, 1e-3, 2e-3, etc.). something stable like 2e-5 might be the best learning rate

[Kautenja]

Further experiments confirm that Adam running at 1e-4 produces unstable results. 2e-5 will be in place from here on out.