There was an interesting discussion on the Parquet mailing list about writing large values into Parquet files. Summary doc: https://docs.google.com/document/d/1nntcYM98PFSkHT70RexSBPtCnWqg1uRJ5_7m--ZgbsA/edit?tab=t.0#heading=h.k4r8orckhbx0
This did make me think about the situation in arrow-rs. The idea of offloading large completed pages to a temporary file and then reading them back when writing out sounds very interesting to me as a pragmatic solution to the problem.
Is this something we could / should support in arrow-rs?
There was an interesting discussion on the Parquet mailing list about writing large values into Parquet files. Summary doc: https://docs.google.com/document/d/1nntcYM98PFSkHT70RexSBPtCnWqg1uRJ5_7m--ZgbsA/edit?tab=t.0#heading=h.k4r8orckhbx0
This did make me think about the situation in arrow-rs. The idea of offloading large completed pages to a temporary file and then reading them back when writing out sounds very interesting to me as a pragmatic solution to the problem.
Is this something we could / should support in arrow-rs?