[Python] Segmentation fault when loading total 2GB of parquet files

[asfimport]

We are trying to load 100 parquet files, and each of them is around 20MB. Before we port [ARROW-1830] into our pyarrow distribution, we use glob to list all the files, and then load them as pandas dataframe through pyarrow.

The schema of the parquet files is like

root
 |-- dateint: integer (nullable = true)
 |-- profileid: long (nullable = true)
 |-- time: long (nullable = true)
 |-- label: double (nullable = true)
 |-- weight: double (nullable = true)
 |-- features: array (nullable = true)
 |    |-- element: double (containsNull = true)

If we only load couple of them, it works without any issue. However, when loading 100 of them, we got segmentation fault as the following. FYI, if we flatten features: array[double] into top level, the file sizes are around the same, and work fine too.

Is there anything we can try to eliminate this issue? Thanks.

>>> import glob
>>> files = glob.glob("/home/dbt/data/*")
>>> data = pq.ParquetDataset(files).read().to_pandas()
[New Thread 0x7fffe8f84700 (LWP 23769)]
[New Thread 0x7fffe3b93700 (LWP 23770)]
[New Thread 0x7fffe3392700 (LWP 23771)]
[New Thread 0x7fffe2b91700 (LWP 23772)]
[Thread 0x7fffe2b91700 (LWP 23772) exited]
[Thread 0x7fffe3b93700 (LWP 23770) exited]

Thread 4 "python" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffe3392700 (LWP 23771)]
0x00007ffff270fc94 in arrow::Status arrow::VisitTypeInline<arrow::py::ArrowDeserializer>(arrow::DataType const&, arrow::py::ArrowDeserializer*) ()
   from /home/dbt/miniconda3/lib/python3.6/site-packages/pyarrow/../../../libarrow_python.so.0
(gdb) backtrace
#0  0x00007ffff270fc94 in arrow::Status arrow::VisitTypeInline<arrow::py::ArrowDeserializer>(arrow::DataType const&, arrow::py::ArrowDeserializer*) ()
   from /home/dbt/miniconda3/lib/python3.6/site-packages/pyarrow/../../../libarrow_python.so.0
#1  0x00007ffff2700b5a in arrow::py::ConvertColumnToPandas(arrow::py::PandasOptions, std::shared_ptr<arrow::Column> const&, _object*, _object**) ()
   from /home/dbt/miniconda3/lib/python3.6/site-packages/pyarrow/../../../libarrow_python.so.0
#2  0x00007ffff2714985 in arrow::Status arrow::py::ConvertListsLike<arrow::DoubleType>(arrow::py::PandasOptions, std::shared_ptr<arrow::Column> const&, _object**) () from /home/dbt/miniconda3/lib/python3.6/site-packages/pyarrow/../../../libarrow_python.so.0
#3  0x00007ffff2716b92 in arrow::py::ObjectBlock::Write(std::shared_ptr<arrow::Column> const&, long, long) ()
   from /home/dbt/miniconda3/lib/python3.6/site-packages/pyarrow/../../../libarrow_python.so.0
#4  0x00007ffff270a489 in arrow::py::DataFrameBlockCreator::WriteTableToBlocks(int)::{lambda(int)#1}::operator()(int) const ()
   from /home/dbt/miniconda3/lib/python3.6/site-packages/pyarrow/../../../libarrow_python.so.0
#5  0x00007ffff270a67c in std::thread::_Impl<std::_Bind_simple<arrow::Status arrow::ParallelFor<arrow::py::DataFrameBlockCreator::WriteTableToBlocks(int)::{lambda(int)#1}&>(int, int, arrow::py::DataFrameBlockCreator::WriteTableToBlocks(int)::{lambda(int)#1}&)::{lambda()#1} ()> >::_M_run() ()
   from /home/dbt/miniconda3/lib/python3.6/site-packages/pyarrow/../../../libarrow_python.so.0
#6  0x00007ffff1e30c5c in std::execute_native_thread_routine_compat (__p=<optimized out>)
    at /opt/conda/conda-bld/compilers_linux-64_1505664199673/work/.build/src/gcc-7.2.0/libstdc++-v3/src/c++11/thread.cc:110
#7  0x00007ffff7bc16ba in start_thread (arg=0x7fffe3392700) at pthread_create.c:333
#8  0x00007ffff78f73dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Reporter: DB Tsai
Assignee: Wes McKinney / @wesm

Related issues:

• [Python] Add function to "flatten" structs within tables (relates to)

PRs and other links:

• GitHub Pull Request #1404

_{Note: This issue was originally created as ARROW-1873. Please see the migration documentation for further details.}

[asfimport]

Wes McKinney / @wesm:
If I had to guess, the features inner array is overflowing 2 billion elements. I marked this for 0.8.0 to take a look

[asfimport]

DB Tsai:
There are only 14 elements in the feature inner array, but there are 100M records in total. Anyway I can debug it or provide more information for you? Thanks.

[asfimport]

DB Tsai:
Doing some digging in a beefy machine, and it runs in big machine!

We have two variances of the same data in different schema. One is

root
 |-- dateint: integer (nullable = true)
 |-- profileid: long (nullable = true)
 |-- time: long (nullable = true)
 |-- label: double (nullable = true)
 |-- weight: double (nullable = true)
 |-- features: array (nullable = true)
 |    |-- element: double (containsNull = true)

as previously shown, and it's 2.7G.

The other one is in struct, and the schema is like

root
 |-- dateint: integer (nullable = true)
 |-- profileid: long (nullable = true)
 |-- time: long (nullable = true)
 |-- label: double (nullable = true)
 |-- weight: double (nullable = true)
 |-- features: struct (nullable = true)
 |    |-- f1: double (nullable = true)
 |    |-- f2: double (nullable = true)
 ...........
 |    |-- f14: double (nullable = true)

and the total size is 1.9G.

It does make sense that the files in struct are smaller due to better compression in columnar format.

However, when the first one is loaded into memory, it takes 22.32GB, while the second one takes 127.97GB. In theory, for uncompressed data, it should at most take 93993056 * (5 + 14) * 8 bytes = 14GB. Where is the overhead coming from? In particularly, the second one in struct is resulting almost 10x overhead. Is it possible to load those data in memory with compression like Spark's dataframe cache? Is it possible to lazily load the data, and only deserialize the data when in need?

Regardless of the overhead issue, we should figure out how to return OOM instead of segmentation fault for easier debugging.

They are long questions, and thank you for helping out.

[asfimport]

Wes McKinney / @wesm:
I'll take a look to see where the OOM might not be bubbling up. There are a number of places where we are not checking for Python errors as rigorously as we could, so at high memory pressure the OOM could happen in any number of places

[asfimport]

DB Tsai:
@wesm thanks. I wonder why the overhead of loading parquet files into memory is so huge. Is it technically addressable in pyarrow or in pandas?

[asfimport]

Wes McKinney / @wesm:
I think the overhead is coming from the Python objects that are being created by to_pandas. Try removing the to_pandas() call and then look at pyarrow.total_allocated_bytes(). All memory allocations in Arrow are precisely accounted for, so it should give you an exact accounting of how much data has been loaded into memory.

[asfimport]

Wes McKinney / @wesm:
Note also

In [8]: import sys

In [9]: d = 3.5

In [10]: sys.getsizeof(d)
Out[10]: 24

In [11]: dct = {}

In [12]: sys.getsizeof(dct)
Out[12]: 288

So the "features" struct in the other table takes up a minimum of 624 bytes per record, probably more than that when you account for internal PyObject* which are created inside the hash table

[asfimport]

Wes McKinney / @wesm:
I just created https://issues.apache.org/jira/browse/ARROW-1886 as a possible way to help with this

[asfimport]

Wes McKinney / @wesm:
I'm going to add a few missing null checks to help catch the OOM, will put up patch soon. I'm not sure how else to help with this

[asfimport]

Wes McKinney / @wesm:
Issue resolved by pull request 1404
#1404