Implement recurrent policy

[kngwyu]

What's done:

• Implement GRU Policy
• Check a2c_cart_poly and ppo_cart_pole works

TODO:

• Implement LSTM Policy
• Check a2c_atari and ppo_atari works
  • I failed to make a2c_atari work, as also reported in other repo
• Check it works for HalfCheetah
• Check it works for flickering Atari

[kngwyu]

[kngwyu]

Wow it really overfits.
Note
This is the result of HalfCheetah-BulletEnv and not compatible with the result in the one on mujoco.

[kngwyu]

[kngwyu]

I noticed lr_cooler and clip_cooler doesn't work correctly, which possibly decreases scores in ppo_atari.
I'm going to bench it again after fixing it.

[kngwyu]

Looks decreasing clip_eps is really infulential for RNNs.

[kngwyu]

And lr_decay looks also infulential, especially for CNN

[kngwyu]

The result of flicker breakout.
The difference from the above picture is decreasing ppo_clip or not(Though in the above picture, ppo_clip is decreased only (0.1 / max_steps) * (max_steps // (8 * 128)

[kngwyu]

LSTM results here use incorrect bias initialization.
I'll upload new results using #17 after the experiments get over.

[kngwyu]

I found an another bug(#18).
So I think all above results are not reliable, though they looks working correctly....

[kngwyu]

I uploaded some results with bug fix in #18