mysql: Batch select queries when required. #1605
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #1608
This commit updates the mysql package to batch SELECT queries if the
number of records being requested exceedes the MySQL limit for the
maximum number placeholders that can be included in a prepared
statement (65,535 placeholders).
As a side note, manually testing this commit required loading a tlog
tree with >65,535 leaves. This was surprisingly difficult to do and
highlights how we're using tlog in a way that it's not really built for.
We will need to write our own append-only log implementation at some
point in order to remove the performance bottlenecks that tlog has.
Log statements from the manual testing are shown below.