Support nested parquet lists for files added through addFiles api #4555
Conversation
|
@rdblue would you be able to provide some thoughts on this? Thanks |
kbendick
left a comment
kbendick
left a comment
There was a problem hiding this comment.
You might want to add tests for 3 level lists.
I know that there were some past PRs around 3 level lists, as parquet uses a different structure for them (or maybe certain engines did with certain versions). If you search closed PRs for 3 level you should find it.
|
@SinghAsDev, does reading 2-level lists work after this PR? Could you write an end-to-end test that validates it? |
|
Hi @rdblue @kbendick thanks for taking a look. This change works for both 2-level and 3-level lists. As part of adding support for 2-level lists, we added quite a few tests which work fine with this change. We have been running some workloads with this change at Pinterest as well. I will add an end-to-end test specifically for this edge case. |
|
Thanks, @SinghAsDev! |
3 participants
No description provided.