[Python] Expose IpcReadOptions to ipc facility

[asfimport]

I would like to be able to read only a subset of columns from a given IPC file. To do this, we need to expose the EXPERIMENTAL (is it still?) IpcReaderOptions.include_fields option. The reason is that the file is on a remote storage and can't mmap thus I want to minimize network transfer.

I do not know the best way to "pythonize" IpcReaderOptions and would need help on this.

Reporter: Francois Saint-Jacques / @fsaintjacques

_{Note: This issue was originally created as ARROW-12681. Please see the migration documentation for further details.}

[asfimport]

David Li / @lidavidm:
FWIW, maybe the better way would be to expose a columns option or similar instead of wholesale exposing the IpcReadOptions struct? At least, I'm thinking of Parquet where we don't expose (Arrow)ReaderProperties but rather various keyword arguments.

(Though of course, as discussed in ARROW-12683, this won't reduce network transfer either way; we always read the entire record batch.)

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
In the context of ARROW-14470 for the Feather reader, we have been looking a bit into the IpcReadOptions.

Some observations / questions:

• For writing, we already expose the IpcWriteOptions in Python (so also exposing IpcReadOptions would be consistent with that), although I agree adding a columns keyword would be more user friendly.

• Typically (for other readers we have), such a columns keyword for only reading a subset is exposed in the "read" function. But for RecordBatchFileReader, the options are passed when opening the reader. So in the Python API it would rather be RecordBatchFileReader(source, columns=...).read_all() instead of RecordBatchFileReader(source).read_all(columns=...). Are we OK with that discrepancy on the Python side?

  In addition to the ability to select fields, the IpcReadOptions also have options for use_threads, memory_pool, max_recursion_depth and ensure_native_endian

[asfimport]

David Li / @lidavidm:
It does seem a little odd that the columns are on the file reader and not on the read itself - but I think we could rectify that easily enough on the C++ side later on.

[asfimport]

Weston Pace / @westonpace:
I agree it is odd and it makes it more difficult to duck type generic collections of readers. It would be nice if users could avoid things like:

for reader in readers:
  if reader instance of ...
  else:

[github-actions]

This issue has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this issue will be closed in 14 days. If this improvement is still desired but has no current owner, please add the 'Status: needs champion' label.