Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor(archive): sqlite query optimization. replace or-ed equal conditions list with in-list #1510

Merged
merged 1 commit into from
Jan 25, 2023

Conversation

LNSD
Copy link
Contributor

@LNSD LNSD commented Jan 25, 2023

A small optimization for the "find messages by" query.

Source: https://stackoverflow.com/a/3074846/1099999

The OR operator needs a much more complex evaluation process than the IN construct because it allows many conditions, not only equals like IN.

Source: https://stackoverflow.com/a/73079261/1099999

IN may be significantly faster. E.g. table with ~23M rows.
OR -> Execution Time: 4540.829 ms
IN -> Execution Time: 855.743 ms

With 20 million message archive sizes, I think this query optimization will have a significant impact helping us to squeeze the Sqlite lemon a little bit more 🍋

@LNSD LNSD self-assigned this Jan 25, 2023
Copy link
Contributor

@jm-clius jm-clius left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh! Great!

Copy link
Contributor

@rymnc rymnc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! great find. Not sure if sqlite will convert the in query to an or query under the hood.

@LNSD
Copy link
Contributor Author

LNSD commented Jan 25, 2023

All CI test failures are unrelated to the PR changes. All Waku archive tests are passing. Ubuntu CI jobs are passing.

I have re-run the jobs, but the tests keep failing. cc #1357

@LNSD LNSD merged commit 8c7a931 into master Jan 25, 2023
@LNSD LNSD deleted the refactor-archive-sqlite-query branch January 25, 2023 15:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants