Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

✨ Implement dynamic sizing for SQL and Collection readers #848

Closed
joocer opened this issue Feb 9, 2023 · 2 comments
Closed

✨ Implement dynamic sizing for SQL and Collection readers #848

joocer opened this issue Feb 9, 2023 · 2 comments

Comments

@joocer
Copy link
Contributor

joocer commented Feb 9, 2023

Download an initial chunk of records (say 500), from that estimate how many records will fit in 64Mb and then download that many on subsequent batches.

@joocer
Copy link
Contributor Author

joocer commented Feb 12, 2023

Do a statistical analysis, from the 500 don't just mean the items, work out the standard deviation and assuming the 500 are representative, calculate how many records fit in 64mb with a 90% probability

joocer added a commit that referenced this issue Feb 26, 2023
@joocer
Copy link
Contributor Author

joocer commented Feb 26, 2023

we're going to start with a simple average until that's not good enough

joocer added a commit that referenced this issue Feb 26, 2023
joocer added a commit that referenced this issue Feb 26, 2023
joocer added a commit that referenced this issue Feb 27, 2023
joocer added a commit that referenced this issue Feb 27, 2023
joocer added a commit that referenced this issue Feb 27, 2023
joocer added a commit that referenced this issue Feb 27, 2023
@joocer joocer closed this as completed Mar 13, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant