Parquet support

[dselivanov]

Any plans to add parquet support?

[jorgemarsal]

Hi @dselivanov,

No plans to add Parquet support in the short term.
Shouldn't be too hard though. We just need to add a ParquetRecordParser that delegates to the official parquet-cpp implementation https://github.com/apache/parquet-cpp.

I can help you if you want to give it a shot.

[dselivanov]

@jorgemarsal, thanks for clarification. At the moment I also don't have time for porting parquet-cpp, so will use SparkR in short term. Will try to return back and have a closer look on parquet-cpp integration.

[dselivanov]

@jorgemarsal, FYI - SparkR parquet reading totally unusable for any real problem due to very inefficient serialization/deserialization. Collection even tiny data.frame of 100mb takes more than 2 minutes...

[jorgemarsal]

Parquet support is on the top of my list. Will get to that when I have some free time.

[dselivanov]

FYI introducing-apache-arrow.
Also seems, cloudera developers started actively refactor parquet-cpp.

[DheerajAgarwal]

Any updates on this one?