NEP: propose new nep for allocator policies

[mattip]

Related to PR gh-17582. Once this gets some review I will propose it on the mailing list.

[leofang]

cc: @pentschev @jakirkham

[leofang]

Silly question: Is there a webpage for this NEP rendered somewhere by the CI?

[mattip]

The NEP is rendered at https://19289-908607-gh.circle-artifacts.com/0/doc/neps/_build/html/nep-0049.html

[seberg]

Thanks, just some comments. While I always see that there is a lot of interest in this, I am a bit fuzzy on the gains (and maybe also if there are alternatives such as subclassing, which doesn't sound like a particlarly good alternative). It would be nice to add a (very short) paragraph about which end-user scenarios can gain a lot (I assume these are things where we ensure that arrays are allocated in memory shared between CPU and GPU/FPGA?)

One thing I also wonder if, it might make sense at some point to have an array creation function where the handler is passed explicitly?

[seberg]

This is not quite true. The old strategy was backward compatible. The worst thing that could happen is that you allocate a huge chunk, create an array with it, set OWNDATA and then the free will put it into a "too large" bucket (wasting memory). But overall, there was never any API change the way the cache was implemented.

This PR has a tiny chance of that, because if an external project would replace ndarray->data (i.e. realloc or use free/malloc), that would have worked before but it will stop working if the allocation strategy was modified.

Now, I am not worried about that. Nobody should be doing that, and it can only be a problem if you use one project modifying the allocation strategy AND a project modifying ndarray->data together.

Your PR makes sure that if the user provides the data (and may have allocated it), we will end up using the normal free strategy, so no issue exists for arrays created from passed in data.

[seberg]

(same also applies to ->shape, but that is even crazier, since the shape and strides were always allocated together, so you have to dig into NumPy code to get it right. Overall, it makes me almost wonder if we could get away with breaking ABI and replacing the allocation strategy there with PyMem_Malloc() by default, but I guess it doesn't matter.)

[mattip]

I didn't change the shape/strides allocation in the PR. Should I make it explicit in the NEP that the policy only touches the data and not the shape/strides ?

[seberg]

Yeah, I think I misread the abstract a bit and got sidetracked. Maybe rearrange the second paragraph of the abstract slightly would set the track clearer (brainstorming here):

This NEP proposes a mechanism to override the memory management strategy used for ndarray->data with user-provided alternatives. This allocation holds the arrays data and is can be very large. As this is the data that operations work on it can benefit from custom allocation strategies to guarantee data alignment or placement in memory shared with the GPU.

EDIT: To be clear, I think its fine, but maybe could be a bit clearer there.

[mattip]

Updated.

[seberg]

Seems fine here, in line with my questions below, it might be nice to have one paragraph that explains how users will benefit (that is probably the users of the libraries that modify the allocation strategy).

I assume this are things like: Alignment guarantees or NUMA core pinning can speed up their code. Allocating in memory shared by GPU and CPU speeds up certain operations?

[mattip]

Alignment guarantees or NUMA core pinning can speed up their code

Yes. There are libraries like pnumpy that would like to keep data on a specific core or hardware where unaligned memory is expensive (like ARMv7).

[tylerjereddy]

Can you guarantee "pinned" memory in the CUDA sense as well? I believe that means that the memory is banned from swapping so the CPU doesn't have to be used. I've seen this special memory type used for passing seamlessly between CPU and GPU; however, the allocation of "formally" pinned memory in this sense normally uses the CUDA C API.

[mattip]

I think using pinned memory would hit some snags due to memcpy. A future enhancement might be to enable user-supplied memcpy routines and using them inside numPy when creating a copy of an ndarray.

[leofang]

Just FYI. CuPy recently added a few APIs to allocate pinned memory that backs a NumPy array:

For using pinned memory more conveniently, we also provide a few high-level APIs in the cupyx namespace, including cupyx.empty_pinned(), cupyx.empty_like_pinned(), cupyx.zeros_pinned(), and cupyx.zeros_like_pinned(). They return NumPy arrays backed by pinned memory. If CuPy’s pinned memory pool is in use, the pinned memory is allocated from the pool.

https://docs.cupy.dev/en/stable/user_guide/memory.html#memory-management. I think PyCUDA also have similar APIs.

Indeed, memcpy could be an issue for D2H transfers, but not H2D (at least in CuPy, arr_cp[:] = arr_np_on_pinned would handle this properly). It's something I am working to circumvent (by calling arr_np_on_pinned = cp.asnumpy(arr_cp, out=arr_np_on_pinned)).

[seberg]

Hmmm, not sure I follow. Is this that you want to distinguish between small/large arrays but that would be difficult or impossible with a global "please align to XY" setting?

[mattip]

Yes

[mattip]

One thing I also wonder if, it might make sense at some point to have an array creation function where the handler is passed explicitly?

Maybe. I think that is out of scope for the NEP.

[mattip]

The NEP has been published to the mailing list http://numpy-discussion.10968.n7.nabble.com/NEP-49-Data-allocation-strategies-td49185.html

[leofang]

As per #18805 (comment), can we rephrase/remove this paragraph please? I misread this too and thought the new allocator interface would change how shape and strides are allocated, which I couldn't care less.

[leofang]

nitpick: we need to link to PR 5457.

[leofang]

nitpick: we need to link to issue 14177

[leofang]

The hyperlinks are rendered incorrectly. I think this is the correct syntax:

Suggested change

	_`issue 5312`: https://github.com/numpy/numpy/issues/5312
	.. _issue 5312: https://github.com/numpy/numpy/issues/5312

(Same applies to the links below.)

[leofang]

nitpick

Suggested change

	included later in the NEP. The added ``struct`` will increase the size of the
	included later in this NEP. The added ``struct`` will increase the size of the

[leofang]

Suggested change

	creation/destruction numpy objects, or creating/destroying Python
	creation/destruction NumPy objects, or creating/destroying Python

[leofang]

Better be explicit:

Suggested change

	objects which might cause a gc)
	objects which might cause a garbage collection).

[leofang]

Suggested change

	The NEP is being tracked by the pnumpy_ project and a `comment in the PR`_
	This NEP is being tracked by the pnumpy_ project and a `comment in the PR`_

[leofang]

Suggested change

	The NEP has been implemented in `PR 17582`_.
	This NEP has been implemented in `PR 17582`_.

[leofang]

Question: is there a link to PyArray_malloc_aligned, or is it internal?

Suggested change

	``PyArray_malloc_aligned`` and friends were added to NumPy with the random API
	refactor. and are used there for performance.
	``PyArray_malloc_aligned`` and friends were added to NumPy with the ``random`` API
	refactor and are used there for performance.

[mattip]

Question: is there a link to PyArray_malloc_aligned, or is it internal?

It is internal

[mattip]

@leofang thanks for the careful review. I did not respond to each comment, but did take them into account in 823cbfb

[mattip]

Here is the rendered NEP. It would be nice to merge this as a draft now, we can fix more before it reaches final status.

[pentschev]

Sorry, I only had a chance to look at this now. It looks good, I left a few minor suggestions on typo and formatting fixes, as well as comment on struct alignment.

[pentschev]

Suggested change

	:c:func:`PyDataMem_EventHookFunc` mechanism already present in NumPy for
	:c:func:`PyDataMem_EventHookFunc` mechanism already present in NumPy for

[mattip]

this was removed, are you looking at an older version?

[pentschev]

Sorry, this may have been from an unfinished prior review.

[pentschev]

Should this read "... keep the struct aligned", or am I misunderstanding something here?

[mattip]

fixed

[pentschev]

Suggested change

	the arrays data and is can be very large. As accessing this data often becomes
	the arrays data and can be very large. As accessing this data often becomes

[mattip]

fixed

[pentschev]

Suggested change

	hardware-specific optimizations. The other allocatations remain unchanged.
	hardware-specific optimizations. The other allocations remain unchanged.

[mattip]

fixed

[pentschev]

Suggested change

	allocations to certain NUMA cores. This desire for alignment was disucssed
	allocations to certain NUMA cores. This desire for alignment was discussed

[mattip]

fixed

[pentschev]

Suggested change

	Sets a new allocation policy. If the input value is NULL, will reset
	the policy to the default. Returns the previous policy, NULL if the
	previous policy was the default. We wrap the user-provided functions
	so they will still call the Python and NumPy memory management callback
	hooks. All the function pointers must be filled in, NULL is not accepted.
	Sets a new allocation policy. If the input value is ``NULL``, will reset
	the policy to the default. Returns the previous policy, ``NULL`` if the
	previous policy was the default. We wrap the user-provided functions
	so they will still call the Python and NumPy memory management callback
	hooks. All the function pointers must be filled in, ``NULL`` is not accepted.

[mattip]

fixed

[seberg]

Thanks, Matti, lets put it in. Editing a NEP later if necessary is easier. Anyone can continue commenting here for a while, especially for in-line comments.

[leofang]

@mattip Thanks for putting this together! Not sure if we should continue the discussion here or in a new thread, but do you think a Python API for this NEP is in scope? At least an API for querying the handler name (i.e. wrapping PyDataMem_GetHandlerName) in Python would be very useful IMHO. If I have NumPy arrays backed by CUDA pinned memory or managed memory, for example, I would like to know so I can act differently to explore potential hardware optimization.

[seberg]

In the spirit of minimality, I would say we can probably start of with cupy (or whichever library installs the custom handler) to provide that functionality. If cupy has the feature of cupy.set_numpy_uses_pinned_allocator() it seems very naturaly to also add a function cupy.is_pinned_allocated(array).

A NumPy function could return the name itself more naturally, or check whether it was allocated with the default allocator of course.

[leofang]

Thanks, @seberg. I had my own pet projects in mind and was not speaking for CuPy 😅 I had in mind something as simple as numpy.get_allocator_name() that returns a str.

For CuPy, we actually have to figure out a way to use this, because currently we do not rely on any NumPy C API and I think it's preferable if we stay with the status quo. This discussion has yet to happen.

[mattip]

We could expose a python function from _multiarray_umath, I am not sure what namespace it should live in but I wouldn't want it exposed at the top level as numpy.get_allocator_name

[seberg]

I am not opposed, just wasn't sure it actually adds much.

Agreed about the namespace, if we can put it into some submodule namespace I am happy about just adoing it! We also could have more Capsule Python side APIs, i.e. expose an np.something.set_allocator(python_c_capsule_with_specific_name).

[eliaskoromilas]

You might want to check our integration. It is based on CFuncPtrs (it was more fun than PyCapsules). We will publish Linux wheels on PyPI as soon as the PR gets merged.

https://github.com/inaccel/numpy-allocator

This is how we use it:
https://github.com/inaccel/coral-api/blob/master/src/main/python/inaccel/coral/_shm.py

[leofang]

Thanks, @mattip @seberg. I don't have any preference for the namespace as long as it's guaranteed to be a public API 🙂

I think relying on dlopen as is done by @eliaskoromilas is probably the easiest solution for projects like CuPy that wish to access C APIs but do not want NumPy to become a build-time dependency. @seberg What's the advantage of the capsule approach? Doesn't it also require a library to implement some snippets in C to generate a capsule?

[seberg]

@leofang, a good idea/proposal of where to put such API would help a lot in getting it in smoothly, whether you care about where that is or not.

Not sure that capsules are useful here, they are probably more useful for special attributes.

[eric-wieser]

I wasn't aware this had any effect - doesn't an array in a struct only contribute it's element size to the alignment?

[seberg]

@eric-wieser I understood it as relying on the assumption that the struct itself is 64 byte (cache-line) aligned. In that case the interesting part of the struct will remain aligned if the name is a multiple of 64 byte.

But, I might be misunderstanding it and I think you are right that it may make sense to add some attributes/pragma to ensure the struct itself is cache-line aligned. (Although, that is not part of the header.)

[eric-wieser]

On another note - Python doesn't use static char buffers for any of its object names, as I remember. Is there a reason for doing so here rather than using a char*?

[mattip]

How can we make it clear who allocates and frees the content of the char * and prevent corruption?

[eric-wieser]

I just posted to the mailing list with a related comment, which I'll repeat here for ease.
It's easy to think of allocators that have other state besides the name, for which we need to solve this problem anyway:

Consider an allocator that aligns to N bytes, where N is configurable from a python call in someone else's extension module. Where do they store N?
They can hide it in PyDataMem_Handler::name but that's obviously an abuse of the API.
They can store it as a global variable, but then obviously the idea of tracking the allocator used to construct an array doesn't work, as the state ends up changing with the global allocator.

The easy way out here would be to add a void* context field to the structure, and pass it into all the methods.
This doesn't really solve the problem though, as now there's no way to cleanup any allocations used to populate context, or worse decrement references to python objects stored within context.
I think we want to bundle PyDataMem_Handler in a PyObject somehow, either via a new C type, or by using the PyCapsule API which has the cleanup and state hooks we need.
PyDataMem_GetHandlerName would then return this PyObject rather than an opaque name.

[eric-wieser]

So I guess more concretely, my suggestion is:

typedef void *(PyDataMem_AllocFunc)(PyObject *self, size_t size);
typedef void *(PyDataMem_ZeroedAllocFunc)(PyObject *self, size_t nelems, size_t elsize);
typedef void (PyDataMem_FreeFunc)(PyObject *self, void *ptr, size_t size);
typedef void *(PyDataMem_ReallocFunc)(PyObject *self, void *ptr, size_t size);

typedef struct {
    PyObject_HEAD
    PyDataMem_AllocFunc *alloc;
    PyDataMem_ZeroedAllocFunc *zeroed_alloc;
    PyDataMem_FreeFunc *free;
    PyDataMem_ReallocFunc *realloc;
} PyDataMem_HandlerObject;

// I haven't used the CAPI in a while, but there's not much boilerplate here anyway
PyType_Spec PyDataMem_HandlerTypeSpec =
{
    .name = "numpy.core.datamemhandler",
    .basicsize = sizeof(PyDataMem_HandlerObject),
    .itemsize = 0,
    .flags = Py_TPFLAGS_BASETYPE
};

Users who are fine with a stateless allocator would use:

PyDataMem_HandlerObject *handler = PyObject_New(PyDataMem_HandlerObject, &PyDataMem_HandlerType)

and then can set the fields as appropriate.

Users who want a stateful allocator can then just create a derived type with some extra state, eg:

typedef struct {
    PyDataMem_HandlerObject handler;
    npy_uintp alignment;
} MyAlignedHandlerObject;

// similar boilerplate to above
PyType_Spec MyAlignedHandlerObject_HandlerTypeSpec = ...;

[mattip]

Responded on the mailing list.

[eric-wieser]

I assume by gcc you mean "the c runtime"?

[mattip]

fixed

[mattip]

At least an API for querying the handler name (i.e. wrapping PyDataMem_GetHandlerName) in Python would be very useful IMHO

Added a mention of this in the NEP.