Adds N-step learning for DQN-based agents.

[prabhatnagarajan]

Known affected agents:
-DQN

• Double DQN
• PAL
• Advantage Learning
• DPP
• SARSA
• Categorical DQN (TODO), waiting on bug fix

[prabhatnagarajan]

Results:
These are the results for 7 domains, where each domain and algorithm had at least three runs.

N-step performs equal or slightly better than DQN in two domains: Qbert and Pong. Otherwise, N-step DQN performs worse. Unfortunately, we don't have a baseline to compare against.

Space Invaders

Breakout

Seaquest

Qbert

Asterix

Pong

BeamRider

[prabhatnagarajan]

Just eyeballed this PR's DQN (which is actually double DQN) results and compared them to the Tuned DQN results from:
#302 (comment).

The results look similar, so it suggests that the n-step DQN implementation of 1-step DQN does not adversely affect performance.

[prabhatnagarajan]

Just eyeballed this PR's DQN (which is actually double DQN) results and compared them to the Tuned DQN results from:
#302 (comment).

The results look similar, so it suggests that the n-step DQN implementation of 1-step DQN does not adversely affect performance.

[prabhatnagarajan]

Just eyeballed this PR's DQN (which is actually double DQN) results and compared them to the Tuned DQN results from:
#302 (comment).

The results look similar, so it suggests that the n-step DQN implementation of 1-step DQN does not adversely affect performance.

[prabhatnagarajan]

Just eyeballed this PR's DQN (which is actually double DQN) results and compared them to the Tuned DQN results from:
#302 (comment).

The results look similar, so it suggests that the n-step DQN implementation of 1-step DQN does not adversely affect performance.

[prabhatnagarajan]

Just eyeballed this PR's DQN (which is actually double DQN) results and compared them to the Tuned DQN results from:
#302 (comment).

The results look similar, so it suggests that the n-step DQN implementation of 1-step DQN does not adversely affect performance.

[prabhatnagarajan]

Just eyeballed this PR's DQN (which is actually double DQN) results and compared them to the Tuned DQN results from:
#302 (comment).

The results look similar, so it suggests that the n-step DQN implementation of 1-step DQN does not adversely affect performance.

[prabhatnagarajan]

Just eyeballed this PR's DQN (which is actually double DQN) results and compared them to the Tuned DQN results from:
#302 (comment).

The results look similar, so it suggests that the n-step DQN implementation of 1-step DQN does not adversely affect performance.

[prabhatnagarajan]

Just eyeballed this PR's DQN (which is actually double DQN) results and compared them to the Tuned DQN results from:
#302 (comment).

The results look similar, so it suggests that the n-step DQN implementation of 1-step DQN does not adversely affect performance.

[muupan]

Can you resolve the conflicts?

[prabhatnagarajan]

The previous version of N-step DQN did not add an additional N-1 transitions when an episode finishes. These are the updated results after adding this feature to N-step learning.

[muupan]

Looks good!