Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Fix][Backport]Music cleanup hanging on large MySQL databases #11801

Merged
merged 1 commit into from Mar 6, 2017

Conversation

Projects
None yet
3 participants
@DaveTBlake
Copy link
Member

commented Mar 3, 2017

Music library cleanup, both on library update, and the overall facility avilable from settings menu, is hanging on very large MySQL (or MariaDB) databases. Smaller db can benefit for a more efficient process too.

The issue is CMusicDatabase::CleanupArtists which creates a temp table tmp_delartists of artist ids to keep, but this table contains duplicates and does not have an index. If few artists need removing then on a large collection, tmp_delartists can be very large (e.g. ~12 x number of artists). The query
DELETE FROM artist WHERE idArtist NOT IN (SELECT idArtist FROM tmp_delartists)
then hangs.

It is unsurprizing that doing a "NOT IN" on an non-indexed table with 370k rows is unsuccessful!

In comparison testing has proven that an additional step of creating a temp table of unique artist ids to keep with an index (primary key), and using that in the "NOT IN" works reasonably with both few and many records on both MySQL and SQLite.

Look at having a lot of artists, then removing a small number of them by changes to music files.
Creating a test build for additional testing. Backport of #11803

@MilhouseVH any change of testing this on your MySQL system?

@DaveTBlake DaveTBlake added this to the Krypton 17.1-RC2 milestone Mar 3, 2017

@DaveTBlake DaveTBlake force-pushed the DaveTBlake:CleanSQLKrypton branch 2 times, most recently from e0c61f1 to bcc4807 Mar 3, 2017

@DaveTBlake DaveTBlake force-pushed the DaveTBlake:CleanSQLKrypton branch from bcc4807 to 57e5468 Mar 3, 2017

@MilhouseVH

This comment has been minimized.

Copy link
Contributor

commented Mar 3, 2017

Tested on my RPi2 with MySQL 5.5 and it's working OK (confirmed the new query is being executed by enabling the MySQL query log). Unfortunately my music library isn't particularly large (8,000 songs or so) and is not in need of cleaning so from a performance point of view I can't really give an indication of the benefit but it looks sane.

I'll include it in tonight's build. Thanks Dave.

@DaveTBlake

This comment has been minimized.

Copy link
Member Author

commented Mar 3, 2017

I'll include it in tonight's build. Thanks Dave.

Thanks @MilhouseVH that will be really useful, I have a test user with RPi and 150k of songs, 30k artists. The SQL works manually, but good to try it in Kodi. His big MySQL db has been turning up some issues for me to fix.

@Uatschitchun

This comment has been minimized.

Copy link

commented Mar 6, 2017

I confirm this one working well with my large source. Tested with Milhouse #303 on Rpi3 and a build from Daves branch on Linux. Without this no cleaning (neither after update nor explicit from settings) was successful!

@DaveTBlake DaveTBlake merged commit 5d5bd01 into xbmc:Krypton Mar 6, 2017

2 checks passed

continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details

@DaveTBlake DaveTBlake deleted the DaveTBlake:CleanSQLKrypton branch Mar 6, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.