Shuffle samples to generate each mini-batch at Replay.get_batch.

[nagachika]

Previous implementation randomize only the offset in the memory and the mini-batch contains a series of samples in original experienced order.
I believe the experiences should be sampled at random to get rid of bias of experiences.

I've realized np.random.permutation with whole memory capacity is not memory efficient. Because I'm newbee on Python and numpy, I cannot find the equivalent way to do this more efficiently. Any hint or suggestions are welcome.

[michaelschaarschmidt]

Hi,

thanks for helping! These are 2 separate strategies and both can be desired. One is to sample each entry separately (not with np.permutaton but with np.random.shuffle and re-assignment of keys or with just picking many indices), one is to sample continuous ranges. Hence, I wont merge this but rather suggest you create an issue and then we will see how we can make this optional.

Strategy 1: Batch size n, select n random indices
Strategy 2: Batch size n, select one index, continuous range, but reshuffle the data

1 sounds intuitively more performant to me but we will have a discussion

[nagachika]

Thank you for your comment!
I didn't know that there's the case that the continuous experiences are desirable. I will file new issue to discuss optional strategies.