[Python] Add convenience function to convert pandas.DataFrame to pyarrow.Buffer containing a file or stream representation

[asfimport]

Reporter: Wes McKinney / @wesm
Assignee: Phillip Cloud / @cpcloud

_{Note: This issue was originally created as ARROW-596. Please see the migration documentation for further details.}

[asfimport]

Matthew Rocklin / @mrocklin:
For network applications a nice interface is something that we can pass to socket.send. This might be something like a bytes, bytearray, memoryview, or sequence of those.

[asfimport]

Wes McKinney / @wesm:
pyarrow.Buffer has a to_pybytes method as a last resort. Converting to bytes is bad because it is a memory copy

Is there a way to do a zero-copy memory handoff to a memoryview? I can dig into how NumPy provides a zero-copy buffer interface but in case you know off hand the right tool to use

[asfimport]

Matthew Rocklin / @mrocklin:
I've never had to construct one myself. I just grab my_numpy_array.data and pass that around. I'll ask Antoine Pitrou to chime in here. I suspect that he would have a better understanding.

[asfimport]

Antoine Pitrou / @pitrou:
Cython allows you to implement the buffer protocol: see https://cython.readthedocs.io/en/latest/src/userguide/buffer.html . I've never used it but it looks similar to what you would do in C.

Note that pyarrow.Buffer needs to be a fixed-size buffer for that operation to make sense. If not, then getbuffer should lock the buffer size until releasebuffer is called.

[asfimport]

Wes McKinney / @wesm:
thanks @pitrou – it is a fixed size buffer, so I think that will work out fine. This is being worked on right now in #369

[asfimport]

Wes McKinney / @wesm:
related to ARROW-376

[asfimport]

Wes McKinney / @wesm:
This is part of ARROW-881 #612

[asfimport]

Wes McKinney / @wesm:
I moved this to 0.4 since we need to come up with a standard spec for the index metadata that other serializers (e.g. fastparquet) can conform to also

[asfimport]

Wes McKinney / @wesm:
This was done in bed0197#diff-c05ae8aa370c62f401b6d29ae2e8ea3e