[Python] Memorymapped arrow file conversion to parquet loads everything into RAM

[asfimport]

When converting a memory mapped arrow file into parquet file, it loads the whole table into RAM. This effectively negates the point of memory mapping.

If this is not a bug,  perhaps there is a proper way of converting the memorymapped arrow file to parquet without using excessive memory?

 

Example code:

    source = pa.memory_map(path_to_arrow_file, 'r')
    table = pa.ipc.RecordBatchFileReader(source).read_all()
    # The followlng line will load the whole thing into RAM
    pq.write_table(table, path_to_parquet)

Reporter: Sep Dehpour

_{Note: This issue was originally created as ARROW-9526. Please see the migration documentation for further details.}

[asfimport]

Antoine Pitrou / @pitrou:
When memory mapping a file, it's the OS which makes decisions about whether to keep things in memory. Chances are that, if the file is kept entirely in memory, it simply means there's no other application requesting that memory.

Do you have reasons to believe something different is happening?