Skip to content

Commit

Permalink
ARROW-8307: [Python] Add memory_map= option to pyarrow.feather.read_t…
Browse files Browse the repository at this point in the history
…able

Having this option makes it easier to do apples-to-apples performance tests of compressed versus uncompressed files (there may be other reasons not to memory map but either way doesn't hurt to have the option)

Closes #6832 from wesm/ARROW-8307

Authored-by: Wes McKinney <wesm+git@apache.org>
Signed-off-by: Wes McKinney <wesm+git@apache.org>
  • Loading branch information
wesm committed Apr 4, 2020
1 parent c87f148 commit a2a2475
Show file tree
Hide file tree
Showing 2 changed files with 7 additions and 2 deletions.
6 changes: 4 additions & 2 deletions python/pyarrow/feather.py
Original file line number Diff line number Diff line change
Expand Up @@ -204,7 +204,7 @@ def read_feather(source, columns=None, use_threads=True):
return read_table(source, columns=columns).to_pandas(use_threads=True)


def read_table(source, columns=None):
def read_table(source, columns=None, memory_map=True):
"""
Read a pyarrow.Table from Feather format
Expand All @@ -214,14 +214,16 @@ def read_table(source, columns=None):
columns : sequence, optional
Only read a specific set of columns. If not provided, all columns are
read.
memory_map : boolean, default True
Use memory mapping when opening file on disk
Returns
-------
table : pyarrow.Table
"""
_check_pandas_version()
reader = ext.FeatherReader()
reader.open(source)
reader.open(source, use_memory_map=memory_map)

if columns is None:
return reader.read()
Expand Down
3 changes: 3 additions & 0 deletions python/pyarrow/tests/test_feather.py
Original file line number Diff line number Diff line change
Expand Up @@ -156,7 +156,10 @@ def test_read_table(version):
table = pa.Table.from_pandas(data)

result = read_table(path)
assert_frame_equal(table.to_pandas(), result.to_pandas())

# Test without memory mapping
result = read_table(path, memory_map=False)
assert_frame_equal(table.to_pandas(), result.to_pandas())


Expand Down

0 comments on commit a2a2475

Please sign in to comment.