-
Notifications
You must be signed in to change notification settings - Fork 3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Introduce enif_alloc_list / enif_make_alloc_list / enif_release_alloc_list #6293
Comments
I see two problems with the API:
Nr 1 is more about API consistency than a practical problem I think. What do you think about a callback driven API, something like this:
This would make a |
API wise that looks great to me, although I am unsure if this is something we could leverage from Rust. @hansihe, do you have thoughts here?
If that ever becomes a need, we could also have |
I think an API like that would work very well with Rust iterators. Allowing the callback to decide when the list terminates is a good idea as well, as it would enable us to build lists from iterators of unknown length. I don't know the internals on the OTP side, but if it could be beneficial, allowing the caller to specify an optional list size hint would also be an option. |
Proposal:
To terminate the list, the callback would do It should probably be named something else as all other term creating functions are named
|
A revised API with simple implementation proposal.
Features
I thought about adding some way to abort and signal error back to caller, but the user can do that themself with |
LGTM and i think it would also work with Rust iterators, right @hansihe? |
Yep, this looks like a really nice API that would fit in with Rust iterators very well. |
When working with NIFs and you have to allocate large lists, we typically have two options:
enif_make_list_from_array
but that requires allocating the data into an array, which will then be discardedenif_make_list_cell
but that requires traversing from the back and allocating the cells one at a timeFor large lists (32 elements over), 1 ends up being faster because the whole list is allocated at once but it uses more memory. 2 uses less memory but ends up slower (at least on Apple M1).
It would be nice if we had a similar API to binaries, where the list is preallocated once and then we can set the list elements one by one:
The above will allocate a list of 10 elements, where each element points to an empty list (the third argument). Now, we can update the given list element with:
Eventually,
enif_make_alloc_list
orenif_release_alloc_list
needs to be invoked, similar to the binary API. The hope is that this API will put the performance closer toenif_make_list_from_array
(the cost is one extra pass setting the CARs) while avoiding allocating an intermediate array.I am not sure if this proposal makes sense, therefore feedback is welcome. :)
Is your feature request related to a problem? Please describe.
This came up here: rusterlium/rustler#486
The text was updated successfully, but these errors were encountered: