-
Notifications
You must be signed in to change notification settings - Fork 50
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Update IPC code to remove Cython #148
Conversation
See if this can be pared down far enough to not need shm at all
Code works locally, removing the need for Cython to do the IPC. After this commit, will go back and refactor
Haven't looked too deeply at the new code yet, but CPU-only Thanks! |
Awesome @andrewseidl! Should be ready for a full review tomorrow. |
Before trying to do IPC, check if libc is found
This needs validation @andrewseidl (along with the CPU deallocation). The GDF examples work on my local machine, so feels like it's either correct or a no-op
When this passes CI, it's ready for review. Looking for someone who understands shared memory better than I do to validate that I call both of these memory-freeing methods after a pandas dataframe is created, so that the pandas df result will then be managed by the Python garbage collector, and no other IPC memory references should exist. Using the Python debugger, |
Goal of this PR is to remove the Cython build process completely, which should improve compatibility across package install tools.
WIP as I'm still trying to figure out if we can release shared memory; original attempt to callshmdt
duringload_buffer()
killed the kernel, suggesting that pyarrow is still using that memory at least up to_load_schema
if not all the way through_load_data()
.Consider passingptr
value out ofload_buffer()
function, then callingshmdt
twice after data has been made into pandas dataframe (once for each original handle passed from Thrift)