PyArrow ArrowMemoryError in 0.10

[dmadeka]

When I try to read a table and convert it to pandas, I get a strange error I haven't seen before:

  File "pyarrow/table.pxi", line 1326, in pyarrow.lib.Table.to_pandas
  File "/home/ubuntu/src/anaconda3/envs/keras1.2_p3/lib/python3.6/site-packages/pyarrow/pandas_compat.py", line 577, in table_to_blockmanager
    blocks = _table_to_blocks(options, block_table, memory_pool, categories)
  File "/home/ubuntu/src/anaconda3/envs/keras1.2_p3/lib/python3.6/site-packages/pyarrow/pandas_compat.py", line 742, in _table_to_blocks
    categories)
  File "pyarrow/table.pxi", line 971, in pyarrow.lib.table_to_blocks
  File "pyarrow/error.pxi", line 85, in pyarrow.lib.check_status
pyarrow.lib.ArrowMemoryError

[dmadeka]

The pyarrow version is 0.10.0

[xhochy]

This means that your system does not have enough memory left to convert the Table to a Pandas DataFrame.

[wesm]

@dmadeka if you get the error reproducibly in 0.10.0 but not 0.8.0 could you please open a JIRA issue

[dmadeka]

@wesm sounds good. But I also updated a bunch of C++ libraries to install some new simd stuff. I need to backtrace that in my virtual image

[dmadeka]

Or it could be a system thing - I need to deep dive it a bit to reproduce it meaningfully

[ghost]

Hello, do you fixed the problem? i have the same problem. (pyarrow.lib.ArrowMemoryError)

[wesm]

@Warfront if you can offer a reproducible example please open a JIRA issue

[ghost]

@wesm the problem was that EC2 instance has not enought RAM memory.. i upgraded ram memory and Ready!

(I forget that i was working on a EC2 instance and i saw the ram memory of my pc and I did not find any problem)

Sorry for my english
Regards!!