Memory Leak when using np.random.choice with replace = False

[GrimLefourbe]

When using choice with an integer arg, the choice function creates a randomised array with np.permutations then returns a slice of this array. This keeps the big array in memory instead of just size elements. The choice function should return a copy of the slice to decrement the ref count of the temporary array.

import numpy as np
a = np.random.choice(a=10**8, size=100, replace=False)

Numpy/Python version information:

Tested in 1.16 but I did not find a fix in the master.

[GCC 4.2.1 Compatible Apple LLVM 10.0.0 (clang-1000.10.44.4)]```

[seberg]

In case you are interested in that, but np.random.default_rng.choice would have a much better behaviour here. (The new random number generator API, it is worth a look!).

[charris]

Is this still a problem?

[WarrenWeckesser]

@charris, yes, for the legacy random interface, it still is:

numpy/numpy/random/mtrand.pyx

Line 979 in ba4e4b6

idx = self.permutation(pop_size)[:size]

The recommended way to avoid the problem is to do what @seberg suggested in a comment last year: use the new API that was introduced in 1.17. It doesn't have this problem:

In [8]: rng = np.random.default_rng()

In [9]: a = rng.choice(10**8, size=100, replace=False)

In theory, we could fix the old code by making a copy after slicing, but at this point, I don't think we should bother. If it isn't crashing Python, our policy has been to not touch the old code.

[mattip]

Closing as a "won't fix" and removing the milestone. Please undo if I am mistaken.