Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

mysql: Batch select queries when required. #1605

Merged
merged 6 commits into from Dec 20, 2021

Conversation

lukebp
Copy link
Member

@lukebp lukebp commented Dec 16, 2021

Closes #1608

This commit updates the mysql package to batch SELECT queries if the
number of records being requested exceedes the MySQL limit for the
maximum number placeholders that can be included in a prepared
statement (65,535 placeholders).

As a side note, manually testing this commit required loading a tlog
tree with >65,535 leaves. This was surprisingly difficult to do and
highlights how we're using tlog in a way that it's not really built for.
We will need to write our own append-only log implementation at some
point in order to remove the performance bottlenecks that tlog has.

Log statements from the manual testing are shown below.

2021-12-20 09:04:55.964 [DBG] STOR: Get 83698 blobs using 2 prepared statements
2021-12-20 09:04:55.965 [DBG] STOR: Executing select statement 1/2
2021-12-20 09:04:56.925 [DBG] STOR: Executing select statement 2/2

This commit updates the mysql package to batch SELECT queries if the
number of records being requested exceedes the MySQL limit for the
maximum number placeholders that can be included in a prepared
statement.
@lukebp lukebp added 91cfcc8 bug A bug that made it into a production enviroment. labels Dec 16, 2021
Copy link
Member

@amass01 amass01 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tACK.

@lukebp lukebp merged commit b7f8a71 into decred:master Dec 20, 2021
@lukebp lukebp deleted the bug-mysql-query branch February 6, 2022 17:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug A bug that made it into a production enviroment.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

politeiad: MySQL max placeholder limit hit.
2 participants