-
Notifications
You must be signed in to change notification settings - Fork 1
Cool! #4
Comments
Hi Paleolimbot :) |
The use-case I imagine may help you (or may not!) would be that you'd get |
I'm very open to try out and expose other conversions from polars/arrow to data.frame. rpolars including the nanoarrow dist seems workable. py-polars also here and there have two different "arrow-transfer"-implementations and it is possible to switch between them. I just started learning Rust last March, feel free to write rust PR part in any style you like, the ideomatics can be sorted out later. The current implementation(Robj->polars_series; polars_series->Robj ) is just what I could come up with (relying heavily on extendr and rust-polars), and this impl requires re-allocation. extendr have enabled to use the altRep-vector, which allows zero-copy immutable views. Another extendr contributor made a proof of concept it. I have not gotten to try implementing any of it. some zero-copy view might bring a smoother user experience such that rpolars-'DataFrame' could be become a subclass of R "data.frame", just like "data.table" is a subclass thereof. |
Cool cool! I suppose you could use 'nanoarrow' the C library, although 'nanoarrow' the R package is what I had in mind. I think the path would be something like:
At this point, you'd need to figure out how to get the ArrowArray pointer address into R. I'd have to do some reading up on Rust's ffi interface to learn how to do that (or maybe you know already). Then you can use: # remotes::install_github("apache/arrow-nanoarrow/r")
array <- nanoarrow::nanoarrow_allocate_array()
nanoarrow::nanoarrow_pointer_move("the address of the Polars exported arrow array", array)
as.vector(array)
# or
arrow::as_arrow_array(array) # (which would be zero copy!) The For data frames as a whole, it looks like those are represented by something like stream <- nanoarrow::nanoarrow_allocate_array_stream()
nanoarrow::nanoarrow_pointer_move("the address of the exported arrow array stream", stream)
as.data.frame(stream)
# or
arrow::as_record_batch_reader(stream) # (again, zero copy!) I'm not sure I have time to get into this over the holidays but I do promise to have a look at some point! Polars is awesome and getting the zero-copy thing to work for interchange with nanoarrow/the Arrow R package/DuckDB/other arrow-based things. |
py-polars has this method to export arrow-arrays. I have adopted the method to just print the raw ptrs instead in this branch(link fixed). will provide a link to R installation binaries soon. |
something works, but something with the schemas...
|
I forgot a step! You have to attach the data type or else there's no way to know how long each of the buffers is supposed to be: https://github.com/apache/arrow-nanoarrow/blob/main/r/R/pkg-arrow.R#L122 |
For lifecycle, the C data interface is supposed to be self contained (i.e., as soon as you call nanoarrow_pointer_move(), you shouldn't need to keep any other objects in scope to keep the array valid) |
It seems the py-polars internal functions, that I used, clones before export such that pyarrow is owned by it self. I have made a new Nanoarrow Release 3 hours ago I can move the ptr and it nanoarrow recognizes the length of the array. But I could not get the as.vector() to work due to some schema thing. |
Will try out if I can import the schema as of your link |
|
Can nanoarrow read schemas from a raw schema ptr also? |
What are the ownership rules for nanoarrow ptrs in conjunction with R? For polars it is the combination of immutability and reference counting, that makes rust memory management work well together with the R or python garbage collector.
I guess it would be better to pass a smart pointer which nanoarrow can take ownership of. As long as smart pointer is not deleted the arrowarray is guranteed to persist. If nanoarrow drops the smart pointer and there are no other references to the array it will be dropped by polars as usual. I'm quite ignorrant if memory safety is baked in arrowArrays themself or it is the responsibility of the implementation. |
Cool! I will look into the dependency loop thing, that sounds like it's my fault. The C Data interface has pretty strict rules for lifecycle management of its structs ( https://arrow.apache.org/docs/format/CDataInterface.html#release-callback-semantics-for-producers )...I would be surprised if Rust deviated from that in any way. Basically, as soon as you get that pointer, it's up to nanoarrow to manage the lifecycle of the object. Once you do get a valid nanoarrow array, you can call the release callback manually using If for some reason you do need to keep some other object in scope for the lifecycle of the array pointer, there's a mechanism for that, too, I just didn't expose it at the R level. For now you can do something like |
I tinkered a bit more with exporting the ArrowArray and pointers and monitoring memory usage. The exported array and ptrs are not a reallocation but pointing to the same arrays as polars is using. So it is just smart pointers to memory owned by rust. The ArrowArray-objects ( It seems nanoarrow can create an array pointing directly into the polars memory. A reallocation does ocour when calling
|
The interface between rpolars and nanoarrow could as you suggested also be on R level. rpolars can have functions to export to nanoarrow readable format and a user could use the nanoarrow package thereafter. Inspired by |
Again, this is so cool and I'm excited to play with this a little bit. I'm mostly away-from-keyboard during the holidays but should have time this week to properly read the code for this branch.
I think this is functionally what is supposed to happen although the terminology might be slightly different because it's C. You won't be able to avoid pointers into Polars memory for the array data itself (or else you may as well copy the data, which is what we're trying to avoid here I think). R provides a number of mechanisms to make sure this data is not mutated (although because it's C, it's possible to ignore all of them for those who choose to ignore the safeguards). For the
I wrote nanoarrow to make this easy to do (hopefully)...there are some S3 generics you can implement ( |
Just a note that this is a really cool project!
I don't know how useful it is here, but I've been working on a 'nanoarrow' R package that provides conversion to and from Arrow C data interface structures from R data frames for the full spectrum of Arrow and R types: https://github.com/apache/arrow-nanoarrow/tree/main/r . I don't know if polars supports input/output of Arrow C data interface, but if it does, nanoarrow could potentially be useful!
The text was updated successfully, but these errors were encountered: