sum vs. mean

[mimoralea]

Hi, Sam.

First of all, thank so much for this code and making it available. Having such a short implementation definitely helps understanding the algorithm.

I was hoping you can give me some intuition behind your use of sum instead of mean. In my implementation of REINFORCE, A3C, GAE, A2C I use mean and things work fine. Equations in online resources seem to suggest mean is the right approach. Other implementations, also use the mean.

Now, your implementation works very well, too! I tested it myself with ATARI games and other environments, and got rock solid results.

Can you share some insights on the use of sum instead of mean?

Again, thanks so much in advance!