You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
While one of the advertised point of Arrow Java is that it comes with an efficient allocator that is based on jemalloc, we only provide jemalloc as an alternative allocator in C++. I would propose to switch to it as the default allocator on all platforms jemalloc is available (i.e. MacOS and Linux) as it provides the following benefits:
in-place resizing while ensuring 64byte alignment, for allocations >4kb this won't touch the data but rather only changes some pointers in kernel internal structs pointing to the respecitive memory pages.
Especially for the builders, simply changing the allocator shows speedups between 2x and 10x. In the optimal case, this puts builders nearly on the same level as if one would have pre-allocated the final size.
This change should also include a blog post that outlines the benefits.
While one of the advertised point of Arrow Java is that it comes with an efficient allocator that is based on jemalloc, we only provide jemalloc as an alternative allocator in C++. I would propose to switch to it as the default allocator on all platforms jemalloc is available (i.e. MacOS and Linux) as it provides the following benefits:
Especially for the builders, simply changing the allocator shows speedups between 2x and 10x. In the optimal case, this puts builders nearly on the same level as if one would have pre-allocated the final size.
This change should also include a blog post that outlines the benefits.
Reporter: Uwe Korn / @xhochy
Assignee: Uwe Korn / @xhochy
Note: This issue was originally created as ARROW-1123. Please see the migration documentation for further details.
The text was updated successfully, but these errors were encountered: