Skip to content

ARROW-1216: [Python] Fix creating numpy array from arrow buffers on python 2 #844

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
pcmoritz wants to merge 2 commits into from
Closed

ARROW-1216: [Python] Fix creating numpy array from arrow buffers on python 2 #844

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
22bb6b5
fix creating numpy buffer from arrow buffer on python 2
pcmoritz Jul 15, 2017
a891971
test that base object is set correctly
pcmoritz Jul 15, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions python/pyarrow/io.pxi
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -528,6 +528,18 @@ cdef class Buffer:
buffer.strides = self.strides
buffer.suboffsets = NULL

def __getsegcount__(self, Py_ssize_t *len_out):
if len_out != NULL:
len_out[0] = <Py_ssize_t>self.size
return 1

def __getreadbuffer__(self, Py_ssize_t idx, void **p):
if idx != 0:
raise SystemError("accessing non-existent buffer segment")
if p != NULL:
p[0] = <void*> self.buffer.get().data()
return self.size


cdef shared_ptr[PoolBuffer] allocate_buffer(CMemoryPool* pool):
cdef shared_ptr[PoolBuffer] result
Expand Down
10 changes: 10 additions & 0 deletions python/pyarrow/tests/test_io.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -160,6 +160,16 @@ def test_buffer_bytearray():
assert result == val


def test_buffer_numpy():
# Make sure creating a numpy array from an arrow buffer works
byte_array = bytearray(20)
byte_array[0] = 42
buf = pa.frombuffer(byte_array)
array = np.frombuffer(buf, dtype="uint8")
assert array[0] == byte_array[0]
Copy link
Member

@wesm wesm Jul 15, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you verify that this sets the array's base member?

Copy link
Contributor Author

@pcmoritz pcmoritz Jul 15, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@wesm Thanks for the note, I added the check and the tests have passed.

assert array.base == buf


def test_buffer_memoryview_is_immutable():
val = b'some data'

Expand Down