You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The common concern with Parquet's Arrow readers and writers is that they're slow.
My diagnosis is that we rely on a chain of processes, which introduces overhead.
For example, writing an Arrow RecordBatch involves the following:
Iterate through arrays to create def/rep levels
Extract Parquet primitive values from arrays using these levels
Write primitive values, validating them in the process (when they already should be validated)
Split the already materialised values into small batches for Parquet chunks (consider where we have 1e6 values in a batch)
Write these batches, computing the stats of each batch, and encoding values
The above is as a side-effect of convenience, as it would likely require a lot more effort to bypass some of the steps.
I have ideas around going from step 1 to 5 directly, but won't know if it's better if there aren't performance benchmarks. I also struggle to see if I'm making improvements while I clean up the writer code, especially removing the allocations that I created to reduce the complexity of the level calculations.
With ARROW-12120 (random array & batch generator), it becomes more convenient to benchmark (and test many combinations of) the Arrow writer.
I would thus like to start adding benchmarks for the Arrow writer.
The common concern with Parquet's Arrow readers and writers is that they're slow.
My diagnosis is that we rely on a chain of processes, which introduces overhead.
For example, writing an Arrow RecordBatch involves the following:
The above is as a side-effect of convenience, as it would likely require a lot more effort to bypass some of the steps.
I have ideas around going from step 1 to 5 directly, but won't know if it's better if there aren't performance benchmarks. I also struggle to see if I'm making improvements while I clean up the writer code, especially removing the allocations that I created to reduce the complexity of the level calculations.
With ARROW-12120 (random array & batch generator), it becomes more convenient to benchmark (and test many combinations of) the Arrow writer.
I would thus like to start adding benchmarks for the Arrow writer.
Reporter: Neville Dipale / @nevi-me
Assignee: Neville Dipale / @nevi-me
PRs and other links:
Note: This issue was originally created as ARROW-12121. Please see the migration documentation for further details.
The text was updated successfully, but these errors were encountered: