Running into heap out of memory issues reading a 25 mb parquet files.

[PulkitChadhaAdobe]

Hi All.

I intend to read parquet files up to 1GB in size but I'm hitting heap out of memory reading a 25 MB parquet files with 6 columns. I'm not able to find a read buffer method. Is there any other alternative to be able to read large Parquet files?

Thanks,
Pulkit

[SeanBarry]

Hi there, did you ever manage to solve this issue? I'm facing a similar issue and struggling to work out how to consume larger files.

[PulkitChadhaAdobe]

Hey Sean, we have not gotten a workaround yet. We ended up writing a Python function using pyarrow to convert parquet into JSON. I don't like the duck-tape solution but atleast it unblocked our project. I'm interested to know what you have learned while trying to fix this issue at your end.

[SeanBarry]

Hey, thanks for the reply. Ha, funnily enough we are doing the exact same thing. I'm spinning up a python script using node's 'child_process' module to ingest the parquet file. It's considerably faster!