ranking is not done correctly

[adam-katona]

In ES, compute_ranks() does an argsort, which will give different ranks to individuals with the same fitness.
This introduces a noise in the gradient estimate. This is not a big issue since the expected value of the noise is zero, but it can slow down convergence.
This is really only a problem in environments, where rewards are sparse, so a lot of individuals will have the same fitness.
Solution: Average ranks for individuals with equal fitness.

[adam-katona]

I just realized, that this might be intentional, so there is some extra exploration in flat regions of the fitness function.