Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update query planner statistics of database after library scan finished #4199

Merged
merged 4 commits into from Aug 16, 2021
Merged

Update query planner statistics of database after library scan finished #4199

merged 4 commits into from Aug 16, 2021

Conversation

uklotzde
Copy link
Contributor

This operation turned out to be crucial for achieving an acceptable performance after a mass import into aoide. Otherwise some queries took 30-90s on a large library or just stalled at 100% CPU! Though the issue might not affect Mixxx which uses a simpler schema that requires less JOINs.

The operation typically finishes after ~5-10 sec.

@uklotzde uklotzde added this to the 2.4.0 milestone Aug 14, 2021
@coveralls
Copy link

coveralls commented Aug 14, 2021

Pull Request Test Coverage Report for Build 1133670393

  • 0 of 22 (0.0%) changed or added relevant lines in 1 file are covered.
  • No unchanged relevant lines lost coverage.
  • Overall coverage decreased (-0.006%) to 25.99%

Changes Missing Coverage Covered Lines Changed/Added Lines %
src/library/scanner/libraryscanner.cpp 0 22 0.0%
Totals Coverage Status
Change from base Build 1133524699: -0.006%
Covered Lines: 20015
Relevant Lines: 77009

💛 - Coveralls

VACUUM;
-- Update statistics for the query planner
-- https://www.sqlite.org/lang_analyze.html
ANALYZE;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the rational for Analyze before Vacuum? I would swap it intuitively.
Can you add a comment?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know which order is correct and why to choose one over the other:

https://stackoverflow.com/questions/58178062/in-what-order-should-i-call-analyze-and-vacuum-in-an-sqlite3-database

Copy link
Contributor Author

@uklotzde uklotzde Aug 14, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Found it:

https://sqlite.org/forum/forumpost/62fb63a29c5f7810?t=h

Richard Hipp himself recommends to run VACCUM before ANALYZE.

src/library/scanner/libraryscanner.cpp Outdated Show resolved Hide resolved
tools/mixxxdb_cleanup.sql Show resolved Hide resolved
@uklotzde uklotzde changed the title Analyze and optimize database after library scan finished Update query planner statistics of database after library scan finished Aug 15, 2021
Copy link
Member

@Holzhaus Holzhaus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thank you!

@Holzhaus Holzhaus requested a review from daschuer August 15, 2021 23:47
@Holzhaus Holzhaus merged commit 86613c3 into mixxxdj:main Aug 16, 2021
@uklotzde uklotzde deleted the analyze-and-optimize-database branch August 16, 2021 19:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants