API: provide examples of use of numpy random API via user stories

[mattip]

Gathered from gh-14604, gh-14517 and the discussions.

• "To summarize I need to draw random ints of a given C type from continually changing ranges, either one-by-one or small batch-by-small batch."

• Someone asked how to use random in a ufunc.

• "An ideal API would allow projects like https://github.com/deepmind/torch-randomkit/tree/master/randomkit or numba to consume the code in NumPy without vendoring it."

• "There are c++ applications which use boost::random, would be nice to be able to swap it for numpy.random."

• "Using the existing distributions from Cython was a requested feature and an explicit goal, yes. There are users waiting for this." (mattip: but isn't this supported via Generator?)

• "Numba would definitely appreciate C functions to access the random distribution implementations, and

• have a side-project (numba-scipy) that is making the Cython wrapped functions in SciPy visible to Numba" (mattip: I think we do this after API: rearrange the cython files in numpy.random #14608 from _generator.so via these cdef extern declarations, but that requires the H file.

• Someone who wants to write a new BitGenerator, i.e., the numpy/bitgenerator repo without needing to have the numpy code as a git submodule

Some of these are already be handled by gh-14604, but I am putting them here for completeness

Edit: turned into a checklist

[rkern]

"Using the existing distributions from Cython was a requested feature and an explicit goal, yes. There are users waiting for this." (mattip: but isn't this supported via Generator?)

No. In the context of the discussion this statement was drawn from, this was referring to the functions in distributions.c: efficient C-level functions for drawing numbers one-by-one without all of the overhead that the Generator methods have for cramming these numbers into numpy arrays. That's what people have been waiting for.

[mattip]

@rkern the way the code is currently structured in #14604 we do not ship distributions.c directly, rather we expose all the funcitons in _generator.pyx from the shared object _generator.*.so. If we wish to officially support them from C we should either add a _generator.pxd which will expose them via cimport _generator or ship a _generator.h header file and a link library for MSVC. I think a pxd file would be better, what do you think?

[bashtage]

Coming from #14951, I think the C functions represent are a lot of the value add of a low-level interface. While some are pretty trival, many are not.

I don't think it adds much to maintenance since these functions are all used by Generator. IMO the simple thing to do is to always have the set of random_ functions exactly match the set required in Generator. Functions that are not in Generator should probably not be exposed, and if not used in Generator or RandomState, then they should be removed (don't think there are many if any like this).

Did you change the export of the distributions in #14604? I couldn't figure out where I might find them.

[mattip]

I removed unused functions and renamed some others more consistently (mainly gauss_zig -> normal) in #14604 which also

• moved distributions.h to numpy/core/include/numpy/random/distributions.h
• moved (only the needed) function declarations from distributions.pxd to mtrand.pyx and _generator.pyx
• removed distributions.pxd

The functions are exported from the _generator c-extension shared object, so the issue is how best to access them from C/Cython/CFFI. xref #14954 (the first commit there shows there is a problem on windows)

[bashtage]

The Windows .pyd files only export a single symbol, PyInit_generator. If this is fixable, then the numba examples will work well.

So that leaves Cython. I've always compiled the C source directly into Cython modules. Is there a reasonable alternative? Most examples that I found searching for Cython dll assume that you have the source of the DLL code, and don't actually rely on the DLL when the Cython extension module is built.

[mattip]

you have the source of the DLL code

The numpy code is only a click away. This has the added advantage of compiling the code directly, rather than calling into a DLL which not only adds runtime overhead but allows for compile-time optimizations.

[rkern]

We need to expose the functions in the DLL/SO so that numba can use them. They have made an unsupported hack into the mtrand DLL/SO, and we need to move them off onto something supported. Copying C source code from the numpy source dist doesn't help them.

[mattip]

The functions in distributions.h are now exposed from the _generator DLL/SO. Can you point to a numba issue or PR so we can help them work out how to do this moving forward?

[rkern]

I'm only reacting to @bashtage's comment: "The Windows .pyd files only export a single symbol, PyInit_generator. If this is fixable, then the numba examples will work well."