[Python] Make Buffered* IO classes available to Python, incorporate into input_stream, output_stream factory functions

[asfimport]

Reporter: Wes McKinney / @wesm
Assignee: Krisztian Szucs / @kszucs

PRs and other links:

• GitHub Pull Request #3252

_{Note: This issue was originally created as ARROW-3126. Please see the migration documentation for further details.}

[asfimport]

Antoine Pitrou / @pitrou:
What would that do under the hood exactly? Any benchmarks to watch for?

[asfimport]

Uwe Korn / @xhochy:
As far as I understand the title, I would do the same as https://docs.python.org/3/library/io.html#io.BufferedReader internally does. Simply using the Python class in pyarrow already brought us great improvements in reading Parquet files from Azure.

[asfimport]

Antoine Pitrou / @pitrou:
Ok. The term "read ahead" is a bit misleading, because it implies that I/O is hidden in the background, which is not how a buffering layer works (the buffer is filled up synchronously when empty, it's not fed by a separate thread).

Can we-reuse io.BufferedReader for this or is the intention to have a similar primitive written in C++? Also, does it return a InputStream or a full-blown RandomAccessFile (the latter is quite a bit more difficult to get right and optimize).

[asfimport]

Wes McKinney / @wesm:
open_stream only uses InputStream https://github.com/apache/arrow/blob/master/python/pyarrow/ipc.pxi#L247. So we should implement a buffering InputStream in C++

[asfimport]

Wes McKinney / @wesm:
@kszucs would you be interested in working on this? My thinking is to add a buffer_size argument to both pyarrow.input_stream and output_stream. After a raw reader or writer is created, if this argument it set, it will be wrapped in either a BufferedInputStream or BufferedOutputStream as appropriate

[asfimport]

Wes McKinney / @wesm:
If someone else could pick this up I would be appreciative. If I finish the other work assigned to me this week and this is not done, I will pick it up

[asfimport]

Wes McKinney / @wesm:
Issue resolved by pull request 3252
#3252