Feature request: Symmetric operation for unsafe_wrap(A, p, dims, own=true)

[rayegun]

I don't believe this operation exists (in previous discussions about implementing a feature it wasn't mentioned) but I'm hoping it can be added:

Some C libraries would like to take ownership of arrays from Julia. In particular SuiteSparse:GraphBLAS would benefit immensely from this operation. The opposite function already exists:
unsafe_wrap(Array, pointer::Ptr{T}, dims; own = true)

What I'd like to request is unwrap!(Array) which would give me a pointer to the underlying storage, but importantly relinquish ownership. This would probably only work in cases where the C library which receives ownership is using the Julia memory functions like :jl_free.

It's understandably niche (and possibly unworkable), but would make Julia's interop with at least one C library faster and less memory intensive by avoiding copies.

[Gnimuc]

Some C libraries would like to take ownership of arrays from Julia.

I don't think it is the ownership that a C library would like to take. Instead, the C libraries usually expect a bubble of heap memory in the C world (e.g. allocated by Libc.malloc). If the C library does not expose an API for allocating such an array, you probably need to use Libc.malloc to get the pointer that can be safely passed to C and wrap the pointer using unsafe_wrap and use it as a Julia array.

In most cases, C libraries will expose a "free" API for releasing the resources created by themselves. But there are cases where one needs to dive into the C code to figure out the lifetime of certain resources.

For example, say, there is a structure called Foo in a C library that has an array pointer field called arr. The C library provides both create_Foo(Foo * x) and destroy_Foo(Foo * x) for allocating and deallocating Foo, but there is no API for manipulating the content of arr, and by default, the array field is initialized as a NULL pointer.

As we know the structure of Foo and we can get the handle on the Julia side via create_Foo, so even there is no API for editing this array, we can create a new array via Libc.malloc, do the necessary initialization we need, and reset Foo.arr to it. Note that, we need to check the C code to see which function is for releasing arr, and we should make sure that the function will be called directly or indirectly by certain APIs we need to use on the Julia side.

[rayegun]

I should say rather than "some" libraries, I know of at least one. SuiteSparse:GraphBLAS for which I would like to do this rather than copying data from Julia.

On initialization I pass SuiteSparse:GraphBLAS :jl_malloc, :jl_free, :jl_realloc, etc. This does help make Julia aware of memory pressure. But the primary purpose of this is actually to allow importing an array to SuiteSparse:GraphBLAS, and exporting it back to Julia so that I can do no-copy computations in C on data from Julia.

Concretely I want to take a SparseMatrixCSC, import the underlying 3 vectors to SuiteSparse:GraphBLAS, do some computations and then export the (potentially different, modified using the jl_ functions above) 3 vectors to Julia.

This can technically be done without any copying (if this were C for instance). Unfortunately right now I have to copy. I do this by ccalling :jl_malloc (Libc.malloc doesn't work with :jl_free) and then doing unsafe_copyto! from the Julia vector. I'd like to do unwrap on this Julia vector, and pass that without any copying.

The memory is sort of still owned by Julia's garbage collector (indeed any matrices created within SuiteSparse:GraphBLAS will be visible to GC since it will use :jl_malloc), but can be :jl_free'd by SuiteSparse:GraphBLAS.

E: This should probably be a second issue but memory created by :jl_malloc cannot be unsafe_wrap'd without segfaulting.

[Gnimuc]

What's the expected behavior when this Julia vector(dropped the ownership) is keeping being used on the Julia side but the C library would like to free it? Should the usage of a Julia vector that has transferred the ownership to C triggers a compilation error? If not, we may need to add a reference-counting mechanism to delay the invocation of GraphBLAS's free APIs in the implementation of finalizers.

[rayegun]

What's the expected behavior when this Julia vector(dropped the ownership) is keeping being used on the Julia side but the C library would like to free it?

The GraphBLAS function states that you shouldn't do this, it sets the pointer you give it to NULL, in the hopes you will not access the memory again until explicitly given back.

Should the usage of a Julia vector that has transferred the ownership to C triggers a compilation error?

If I did something like GBMatrix(X::SparseMatrixCSC) which does the exporting of X's 3 vectors to GraphBLAS/C I would probably set X's internal vectors to []. I as the package author essentially have to promise not to touch that memory again until its given back (unsafe_unwrap would probably be better in that case).

If not, we may need to add a reference-counting mechanism to delay the invocation of GraphBLAS's free APIs in the implementation of finalizers.

My hope would be that unsafe_unwrap would make its input behave like memory allocated by :jl_malloc (unregister finalizer?). It's visible to GC, but doesn't have a finalizer attached, and won't be freed without a call to :jl_free.

[Gnimuc]

The GraphBLAS function states that you shouldn't do this, it sets the pointer you give it to NULL, in the hopes you will not access the memory again until explicitly given back.

Do you mean the C library will set the data field of the Julia vector to NULL firstly and then give it a new address?

Somehow this operation should work like an "atomic" operation, no memory access is allowed until it is finished.

[rayegun]

Do you mean the C library will set the data field of the Julia vector to NULL firstly and then give it a new address?

No, I haven't (yet) integrated that deeply.

This is the ccall: ccall((:GxB_Matrix_import_CSC, libgraphblas), GrB_Info, (Ptr{GrB_Matrix}, GrB_Type, GrB_Index, GrB_Index, Ptr{Ptr{GrB_Index}}, Ptr{Ptr{GrB_Index}}, Ptr{Ptr{Cvoid}}, GrB_Index, GrB_Index, GrB_Index, Bool, Bool, GrB_Descriptor), A, type, nrows, ncols, Ap, Ai, Ax, Ap_size, Ai_size, Ax_size, is_uniform, jumbled, desc).

This function will set Ap, Ai and Ax to C_NULL. This obviously does nothing in Julia, but if were to give GraphBLAS access to the actual underlying pointer in Julia's internals that would be correct.

I'd prefer to do this all within Julia. I could create a shim in C, that calls Julia C code and GraphBLAS, which solves pretty much all these problems I think, but I'd like to avoid that.