DataFrame.read_parquet and DataFrame.write_parquet

[cigrainger]

Docs for Polars are here: reading and writing.

I've held back on implementing these as almost certainly a pure elixir backend won't be able to use it. I do think it's worth getting in though. Not sure what the best practice is here: just raising that it's not supported for a given backend?

[arpieb]

I had actually done some research on this a while back trying to leverage Parquet files from within Elixir. Here are some resources I found but never got a chance to dig deeply into beyond generating the Erlang bindings for the Thrift definition (I was more focused on getting the data processed than tool-building at the time):

• https://github.com/apache/parquet-format
• https://thrift.apache.org/docs/Languages.html
  • https://github.com/apache/thrift/blob/master/lib/erl/README.md

[arpieb]

FWIW, I think raising an exception on unimplemented for a specific backend would be fine until it is implemented, but doesn't hold up the progression of the explorer package.

For example, Nx didn't hold back on implementing features based on XLA in order to keep libtorch on parity, which is fine.

[cigrainger]

Thanks for that @arpieb! Those are great resources to come back to for the pure Elixir implementation.