New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
✨ Implement dynamic sizing for SQL and Collection readers #848
Comments
Do a statistical analysis, from the 500 don't just mean the items, work out the standard deviation and assuming the 500 are representative, calculate how many records fit in 64mb with a 90% probability |
we're going to start with a simple average until that's not good enough |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Download an initial chunk of records (say 500), from that estimate how many records will fit in 64Mb and then download that many on subsequent batches.
The text was updated successfully, but these errors were encountered: