ENH: Add `out` parameter to all Generator distributions

[anerokhi]

Random distribution functions like numpy.random.binomial allocate a new out array if the input parameters are arrays. This may lead to high memory consumption. Please add the out parameter to such functions.

[mattip]

Please use the new api instead. Some of the distribution methods of Generator accept an out argument, indeed binomial does not. @rkerns, @bashtage was there a reason we did not add an out argument to all of them?

[bashtage]

Mostly due to the doc cruft of adding it to add distributions and the extra code. Out only makes sense for fast distributions since the overhead from array allocation is negligible when generating from the slower distributions

…

On Mon, Feb 17, 2020, 08:55 Matti Picus ***@***.***> wrote: Please use the new api <https://numpy.org/doc/1.18/reference/random/> instead. Some of the distribution methods of Generator <https://numpy.org/doc/1.18/reference/random/generator.html#distributions> accept an out argument, indeed binomial does not. @rkerns <https://github.com/rkerns>, @bashtage <https://github.com/bashtage> was there a reason we did not add an out argument to all of them? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#15581?email_source=notifications&email_token=ABKTSRP7SQUUBBJJOIFF27LRDJGJ3A5CNFSM4KWJCVJ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEL5R7WI#issuecomment-586883033>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABKTSRMFX6KN3MUC55E6VBLRDJGJ3ANCNFSM4KWJCVJQ> .

[anerokhi]

Please use the new api instead.

I need get_state/set_state.

Out only makes sense for fast distributions since the overhead from array allocation is negligible

Performance overhead may be negligible, but not memory.

[mattip]

I need get_state/set_state.

Could you elaborate your use case? The new API does not use state, it uses a SeedSequence which will seed each BitGenerator with the same seed, and has a spawn(n) method to create n new SeedSequences to be used in separate threads.

[bashtage]

Performance overhead may be negligible, but not memory.

You can always copy from a smaller temporary array to your array your self. Ultimately it is a tradeoff between maintenance/complexity and performance. I (and IIRC @rkern) felt that the API expansion needed to exable this feature wasn't worth theit for NumPy.

The beauty of the new Generator is that you can pretty easily roll your own out generator using either Cython or numba (if you don't want to compile). One of the goals of the new structure was to reduce the need for NumPy to directly provide all features needed by endusers.

I need get_state/set_state.

Nothing stops you from using

from numpy.random import PCG64, Generator

entropy = 43248230958734908675674398
pcg = PCG64(entropy)
state = pcg.state
gen = Generator(pcg)
print(gen.random())
pcg.state = state
print(gen.random())

which outputs

0.6775867162804486
0.6775867162804486

[anerokhi]

@mattip, @bashtage I think I see how I can rewrite my code to use the new API and reset a generator state (or create a new one with the same initial state) at some points in the program (haven't tried yet, though).

You can always copy from a smaller temporary array to your array your self.

Ok, using (assigning through) slices is a possible (temporary?) solution.

[bashtage]

@anerokhi that is the easy way. While this isn't quite as performant as out since it requires allocating an array to hold the values, I suspect that for many distributions this is noise from the pov of performance.

[bashtage]

FWIW if someone implemented #8833 then out would be standard.

[TonyCongqianWang]

The beauty of the new Generator is that you can pretty easily roll your own out generator using either Cython or numba (if you don't want to compile). One of the goals of the new structure was to reduce the need for NumPy to directly provide all features needed by endusers.

Can someone please elaborate? I don't want to compile and haven't used numba yet. How would one do this?