-
Notifications
You must be signed in to change notification settings - Fork 3.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
GH-41159: [Go][Parquet] Improvement Parquet BitWriter WriteVlqInt Performance #41160
Conversation
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for this! It's a great find. Do you think you could add a benchmark into the test file to track this so that we can make sure we don't introduce any performance regressions in the future?
thanks for checking @zeroshade And here's the Benchmark Test Before vs After code change
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fantastic, thanks! I'll merge once CI completes
After merging your PR, Conbench analyzed the 5 benchmarking runs that have been run so far on merge-commit ec2d7cb. There were no benchmark performance regressions. 🎉 The full Conbench report has more details. It also includes information about 1 possible false positive for unstable benchmarks that are known to sometimes produce them. |
…nt Performance (apache#41160) [apacheGH-41159](apache#41159) ### Rationale for this change This change improves Parquet FileWriter performance while writing parquets from arrow Records. We saw a speed improvement from writing 320k rows/sec -> 650 rows/sec after making this change. ### What changes are included in this PR? This PR reuses the `buf` variable being used by the bitWriter when writing parquet files. ### Are these changes tested? Yes ### Are there any user-facing changes? No Authored-by: @ hhoughgg * GitHub Issue: apache#41159 Lead-authored-by: Andy Fan <duan-wei@cloudflare.com> Co-authored-by: andyfan <52736754+DuanWeiFan@users.noreply.github.com> Signed-off-by: Matt Topol <zotthewizard@gmail.com>
…nt Performance (apache#41160) [apacheGH-41159](apache#41159) ### Rationale for this change This change improves Parquet FileWriter performance while writing parquets from arrow Records. We saw a speed improvement from writing 320k rows/sec -> 650 rows/sec after making this change. ### What changes are included in this PR? This PR reuses the `buf` variable being used by the bitWriter when writing parquet files. ### Are these changes tested? Yes ### Are there any user-facing changes? No Authored-by: @ hhoughgg * GitHub Issue: apache#41159 Lead-authored-by: Andy Fan <duan-wei@cloudflare.com> Co-authored-by: andyfan <52736754+DuanWeiFan@users.noreply.github.com> Signed-off-by: Matt Topol <zotthewizard@gmail.com>
…nt Performance (apache#41160) [apacheGH-41159](apache#41159) ### Rationale for this change This change improves Parquet FileWriter performance while writing parquets from arrow Records. We saw a speed improvement from writing 320k rows/sec -> 650 rows/sec after making this change. ### What changes are included in this PR? This PR reuses the `buf` variable being used by the bitWriter when writing parquet files. ### Are these changes tested? Yes ### Are there any user-facing changes? No Authored-by: @ hhoughgg * GitHub Issue: apache#41159 Lead-authored-by: Andy Fan <duan-wei@cloudflare.com> Co-authored-by: andyfan <52736754+DuanWeiFan@users.noreply.github.com> Signed-off-by: Matt Topol <zotthewizard@gmail.com>
GH-41159
Rationale for this change
This change improves Parquet FileWriter performance while writing parquets from arrow Records.
We saw a speed improvement from writing 320k rows/sec -> 650 rows/sec after making this change.
What changes are included in this PR?
This PR reuses the
buf
variable being used by the bitWriter when writing parquet files.Are these changes tested?
Yes
Are there any user-facing changes?
No
Authored-by: @hhoughgg