-
Notifications
You must be signed in to change notification settings - Fork 87
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closes #2855: Read batches for Parquet strings #2856
Conversation
I believe the test that is failing is because of the I think it will require some modifications to the |
2d0871c
to
f28c2ce
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good to me!! Thanks @bmcdonald3!
In the first implementation of Parquet, strings were the only data type being read one at a time, rather than in batches since we can't read directly into a
uint(8)
buffer, so we were reading one value at a time and then copying it into the Chapel buffer. Rather than doing that one a time, we can switch to reading batches and then bulk copying to improve performance. Here are some performance numbers collected on 2 nodes of a Cray XC with varying batch sizes:So, we see about a 3x improvement.
Closes #2855