Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[JSON]Faster access to music library #14160

Merged
merged 1 commit into from Jul 15, 2018
Merged

Conversation

DaveTBlake
Copy link
Member

Rework the way AudioLibrary.GetArtists, GetAlbums and GetSongs are implemented to be many times faster.

Looking at up to a 500% increase in speed for processing JSON requests, attained by avoiding the use of fileitem lists, making inefficient queries per row for additional data and by sorting at the db rather than in memory.

Just a minor patch bump to JSON API version despite being a significant speed improvement and fix of several bugs.

@DaveTBlake DaveTBlake added Type: Fix non-breaking change which fixes an issue Type: Improvement non-breaking change which improves existing functionality Component: Music API change: JSON-RPC v18 Leia labels Jul 7, 2018
@DaveTBlake DaveTBlake added this to the Leia 18.0-alpha3 milestone Jul 7, 2018
@garbear
Copy link
Member

garbear commented Jul 7, 2018

Had a look at the code and it seems sane. The music database cpp got bumped by over 1,500 lines. It actually overtook VideoDatabase.cpp as the longest file. Maybe it's time to split into separate files so it can better scale to your future work?

@DaveTBlake
Copy link
Member Author

Thanks for looking @garbear. The length of the .cpp file is not really the issue, for example some of those extra lines are just long comments to document the design. What matters is how clear it is what each method does and why it does it like it does. So hound me for more Doxygen comments I guess.

In my view it is better to have all the specific SQL built in one place, than dotted about, it encourages common queries. And to be honest I am really not sure how I would split it. Despite the initial "that's a lot of lines", once you read through the module it is more about string concantentation than complex code.

Meanwhile it seems I have to tidy up some warnings to get the cross platform builds happy.

@DaveTBlake DaveTBlake force-pushed the JSONFastMusic branch 2 times, most recently from 5b8871b to 3f3d766 Compare July 8, 2018 07:59
: fetch(fetch),
output(output),
recno(recno),
strField()

This comment was marked as spam.

{
DatasetFieldInfo field(false, false, -1);
fields.emplace_back(field);
}

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

const std::string CDatabase::DatasetLayout::GetFields()
{
std::string strSQL;
for (auto field : fields)

This comment was marked as spam.


bool CDatabase::DatasetLayout::HasFilterFields()
{
for (auto field : fields)

This comment was marked as spam.

bool HasFilterFields();

private:
std::vector<DatasetFieldInfo> fields;

This comment was marked as spam.

itemUrl.AppendPath(path);
CThumbLoader *thumbLoader = NULL;
thumbLoader = new CMusicThumbLoader();
if (thumbLoader != NULL)

This comment was marked as spam.

{
CThumbLoader *thumbLoader = NULL;
thumbLoader = new CMusicThumbLoader();
if (thumbLoader != NULL)

This comment was marked as spam.


// Count number of artists that satisfy selection criteria
//(includes xsp limits from filter, but not sort limits)
total = (int)strtol(GetSingleValue("SELECT COUNT(1) FROM artist " + strSQLExtra, m_pDS).c_str(), NULL, 10);

This comment was marked as spam.


// Count number of albums that satisfy selection criteria
//(includes xsp limits from filter, but not sort limits)
total = (int)strtol(GetSingleValue("SELECT COUNT(1) FROM album " + strSQLExtra, m_pDS).c_str(), NULL, 10);

This comment was marked as spam.

// Fields fetched by GetArtistsByWhereJSON, order same as in JSONtoDBArtist
static enum _JoinToArtistFields
{
joinToArtist_isSong = 0,

This comment was marked as spam.

@DaveTBlake
Copy link
Member Author

Unless anyone else wishes to review the code, or is going to do some more JSON testing I would like to merge this Thursday so that it can be part of alpha testing in the wild.

@MilhouseVH
Copy link
Contributor

MilhouseVH commented Jul 11, 2018

@DaveTBlake there's a MySQL (possibly SQLite too) issue with the following JSON query:

curl -s --data-binary '{"jsonrpc": "2.0", "params": {"sort": {"order": "ascending", "method": "title"}, "properties": ["title", "track", "artist", "album", "fanart", "thumbnail", "file"], "limits": {"start": 0, "end": 400}}, "method": "AudioLibrary.GetSongs", "id": "libSongs"}' -H 'content-type: application/json;' http://localhost:8080/jsonrpc

In kodi.log:

00:43:07.105 T:1359635312   DEBUG: CWebServer[8080]: request received for /jsonrpc
00:43:07.118 T:1359635312   DEBUG: GetSongsByWhereJSON query: SELECT sv.*, song_artist.idArtist AS idArtist, song_artist.strArtist AS strArtist FROM (SELECT song.idSong, strTitle, strTitle, path.strPath || strFilename AS strPathFile, album.strAlbum AS strAlbum, (iTrack & 0xffff) AS track FROM song JOIN album ON album.idAlbum = song.idAlbum JOIN path ON path.idPath = song.idPath ORDER BY strTitle LIMIT 400) AS sv JOIN song_artist ON song_artist.idSong = sv.idSong AND song_artist.idRole = 1 GROUP BY sv.idSong, song_artist.idArtist ORDER BY strTitle, song_artist.iOrder
00:43:07.120 T:1359635312   ERROR: SQL: [MyMusic72] Undefined MySQL error: Code (1060)
                                            Query: SELECT sv.*, song_artist.idArtist AS idArtist, song_artist.strArtist AS strArtist FROM (SELECT song.idSong, strTitle, strTitle, path.strPath || strFilename AS strPathFile, album.strAlbum AS strAlbum, (iTrack & 0xffff) AS track FROM song JOIN album ON album.idAlbum = song.idAlbum JOIN path ON path.idPath = song.idPath ORDER BY strTitle LIMIT 400) AS sv JOIN song_artist ON song_artist.idSong = sv.idSong AND song_artist.idRole = 1 GROUP BY sv.idSong, song_artist.idArtist ORDER BY strTitle, song_artist.iOrder
00:43:07.120 T:1359635312   ERROR: GetSongsByWhereJSON failed

Reformatting the query produces:

SELECT sv.*, song_artist.idArtist AS idArtist, song_artist.strArtist AS strArtist
FROM (SELECT song.idSong, strTitle, strTitle, path.strPath || strFilename AS strPathFile, album.strAlbum AS strAlbum, (iTrack & 0xffff) AS track
      FROM song JOIN album ON album.idAlbum = song.idAlbum
                JOIN path ON path.idPath = song.idPath
      ORDER BY strTitle LIMIT 400) AS sv JOIN song_artist ON song_artist.idSong = sv.idSong AND song_artist.idRole = 1
GROUP BY sv.idSong, song_artist.idArtist
ORDER BY strTitle, song_artist.iOrder

Running this query in MySQL Workbench produces the error:

Error Code: 1060. Duplicate column name 'strTitle'

and indeed, in the sv sub query there are two references to the strTitle column - presumably one of them needs to be dropped?

@DaveTBlake
Copy link
Member Author

Thanks @MilhouseVH, while SQLite happily gets same field twice obviously MySQL does not. Fix imminent.

@zag2me
Copy link
Contributor

zag2me commented Jul 12, 2018

Tested in millhouse raspberry pi build and seems to work as expected with the iPhone remote as far as I can tell. Hard to judge the speed but any improvement is most welcome in that area!

@a1rwulf
Copy link
Member

a1rwulf commented Jul 12, 2018

https://github.com/DaveTBlake/xbmc/blob/c5e14e91bddc077250b0b8272162fdaf0deb1da9/xbmc/music/MusicDatabase.cpp#L5802
Imho this is not good - you tie the json schema to the db schema.
They are different subsystems and MusicDatabase.cpp should not need a change when somebody does changes in the json api.

@MilhouseVH
Copy link
Contributor

MilhouseVH commented Jul 12, 2018

Thanks @DaveTBlake that now works (it doesn't crash) but it does have some other issues.

If you run:

curl -s --data-binary '{"jsonrpc": "2.0", "params": {"sort": {"order": "ascending", "method": "title"}, "properties": ["title", "track", "artist", "album", "fanart", "thumbnail", "file"], "limits": {"start": 0, "end": 400}}, "method": "AudioLibrary.GetSongs", "id": "libSongs"}' -H 'content-type: application/json;' http://localhost:8080/jsonrpc
  1. This PR introduces a difference in the response - it now includes the property:
"albumid":null

on every song - including those which are part of an album. albumid is not a requested property, and it is not in the response without this PR. So not only should it not be included in the response, but it seems to have a fixed null value.

  1. This PR is returning "thumbnail":"" for every song, even when a thumbnail is present (without this PR, the thumbnail property is populated correctly).

I have one song in my library which is by two artists, and this PR is returning the same song twice when calling AudioLibrary.GetSongs.

This is the song details as returned by AudioLibrary.GetSongDetails (with this PR):

curl -s --data-binary '{"jsonrpc": "2.0", "params": {"songid": 21791, "properties": ["title", "track", "artist", "album", "fanart", "thumbnail", "file"]}, "method": "AudioLibrary.GetSongDetails", "id": "libSongs"}' -H 'content-type: application/json;' http://localhost:8080/jsonrpc
{"id":"libSongs","jsonrpc":"2.0","result":{"songdetails":{"album":"Ambient Lounge Vol. 4","artist":["Atjazz","Dawne B."],"fanart":"image://http%3a%2f%2fassets.fanart.tv%2ffanart%2fmusic%2f89ad4ac3-39f7-470e-963a-56509c546377%2fartistbackground%2fvarious-artists-5075a5c07a2ac.jpg/","file":"nfs://192.168.0.3/mnt/share/data/Music/MP3/Various Artists/Ambient Lounge Vol. 4/106 - Atjazz feat. Dawne B. - Harmony.mp3","label":"Harmony","songid":21791,"thumbnail":"image://nfs%3a%2f%2f192.168.0.3%2fmnt%2fshare%2fdata%2fMusic%2fMP3%2fVarious%20Artists%2fAmbient%20Lounge%20Vol.%204%2fcover.jpg/","title":"Harmony","track":106}}}

(Same response - pretty format):

{
   "id":"libSongs",
   "jsonrpc":"2.0",
   "result":{
      "songdetails":{
         "album":"Ambient Lounge Vol. 4",
         "artist":[
            "Atjazz",
            "Dawne B."
         ],
         "fanart":"image://http%3a%2f%2fassets.fanart.tv%2ffanart%2fmusic%2f89ad4ac3-39f7-470e-963a-56509c546377%2fartistbackground%2fvarious-artists-5075a5c07a2ac.jpg/",
         "file":"nfs://192.168.0.3/mnt/share/data/Music/MP3/Various Artists/Ambient Lounge Vol. 4/106 - Atjazz feat. Dawne B. - Harmony.mp3",
         "label":"Harmony",
         "songid":21791,
         "thumbnail":"image://nfs%3a%2f%2f192.168.0.3%2fmnt%2fshare%2fdata%2fMusic%2fMP3%2fVarious%20Artists%2fAmbient%20Lounge%20Vol.%204%2fcover.jpg/",
         "title":"Harmony",
         "track":106
      }
   }
}

No issues here - one song, both artists are present (and the thumbnail!)

However, with AudioLibrary.GetSongs (when filtering on title = Harmony):

curl -s --data-binary '{"jsonrpc": "2.0", "params": {"sort": {"order": "ascending", "method": "title"}, "filter": {"operator": "is", "field": "title", "value": "Harmony"}, "properties": ["title", "track", "artist", "album", "fanart", "thumbnail", "file"], "limits": {"start": 0, "end": 400}}, "method": "AudioLibrary.GetSongs", "id": "libSongs"}' -H 'content-type: application/json;' http://localhost:8080/jsonrpc
{"id":"libSongs","jsonrpc":"2.0","result":{"limits":{"end":2,"start":0,"total":2},"songs":[{"album":"Ambient Lounge Vol. 4","albumid":null,"artist":["Atjazz"],"fanart":"","file":"nfs://192.168.0.3/mnt/share/data/Music/MP3/Various Artists/Ambient Lounge Vol. 4/106 - Atjazz feat. Dawne B. - Harmony.mp3","label":"Harmony","songid":21791,"thumbnail":"","title":"Harmony","track":106},{"album":"Pills 'n' Thrills and Bellyaches","albumid":null,"artist":["Happy Mondays"],"fanart":"image://http%3a%2f%2fassets.fanart.tv%2ffanart%2fmusic%2f01f4fb92-6bf0-4de5-83dc-3fe194e763bf%2fartistbackground%2fhappy-mondays-4dda6a5105584.jpg/","file":"nfs://192.168.0.3/mnt/share/data/Music/MP3/Happy Mondays/Pills 'n' Thrills and Bellyaches/Happy Mondays - Harmony.mp3","label":"Harmony","songid":15853,"thumbnail":"","title":"Harmony","track":10},{"album":"Ambient Lounge Vol. 4","albumid":null,"artist":["Dawne B."],"fanart":"","file":"nfs://192.168.0.3/mnt/share/data/Music/MP3/Various Artists/Ambient Lounge Vol. 4/106 - Atjazz feat. Dawne B. - Harmony.mp3","label":"Harmony","songid":21791,"thumbnail":"","title":"Harmony","track":106}]}}

(Same response - pretty format):

{
   "id":"libSongs",
   "jsonrpc":"2.0",
   "result":{
      "limits":{
         "end":2,
         "start":0,
         "total":2
      },
      "songs":[
         {
            "album":"Ambient Lounge Vol. 4",
            "albumid":null,
            "artist":[
               "Atjazz"
            ],
            "fanart":"",
            "file":"nfs://192.168.0.3/mnt/share/data/Music/MP3/Various Artists/Ambient Lounge Vol. 4/106 - Atjazz feat. Dawne B. - Harmony.mp3",
            "label":"Harmony",
            "songid":21791,
            "thumbnail":"",
            "title":"Harmony",
            "track":106
         },
         {
            "album":"Pills 'n' Thrills and Bellyaches",
            "albumid":null,
            "artist":[
               "Happy Mondays"
            ],
            "fanart":"image://http%3a%2f%2fassets.fanart.tv%2ffanart%2fmusic%2f01f4fb92-6bf0-4de5-83dc-3fe194e763bf%2fartistbackground%2fhappy-mondays-4dda6a5105584.jpg/",
            "file":"nfs://192.168.0.3/mnt/share/data/Music/MP3/Happy Mondays/Pills 'n' Thrills and Bellyaches/Happy Mondays - Harmony.mp3",
            "label":"Harmony",
            "songid":15853,
            "thumbnail":"",
            "title":"Harmony",
            "track":10
         },
         {
            "album":"Ambient Lounge Vol. 4",
            "albumid":null,
            "artist":[
               "Dawne B."
            ],
            "fanart":"",
            "file":"nfs://192.168.0.3/mnt/share/data/Music/MP3/Various Artists/Ambient Lounge Vol. 4/106 - Atjazz feat. Dawne B. - Harmony.mp3",
            "label":"Harmony",
            "songid":21791,
            "thumbnail":"",
            "title":"Harmony",
            "track":106
         }
      ]
   }
}

With this PR, the limit values indicate there are 2 songs, but 3 items are actually being returned in the response as songid 21791 is being returned twice, once for each artist. This is the only song entry that does this - I have other multi-artist songs that are not being split, so it's a bit odd. Not sure if it's because there's a different song (by a different artist, "Happy Mondays" in this case) with the same name, and which may be affecting [My]SQL grouping?

@DaveTBlake
Copy link
Member Author

DaveTBlake commented Jul 12, 2018

Thanks for further tests @MilhouseVH
1 & 2 are related and actually nothing to do with the query or db side of things, but how thumbnail was fetched when album Id isn't known. Now fixed

3 is as you surmised, songs with the same name not being correctly grouped or ordered. Took me a while to repeat it, and I want to do some more checks myself

@DaveTBlake
Copy link
Member Author

Like to try one more time @MilhouseVH? I have fixed the issues you found and done quite a lot more testing myself (to ensure I didn't add regressions).

@MilhouseVH
Copy link
Contributor

Thanks, the duplicate song and albumid issues are resolved.

However there's still an issue with thumbnails when calling AudioLibrary.GetSongs:

Query:

curl -s --data-binary '{"jsonrpc": "2.0", "params": {"sort": {"order": "ascending", "method": "title"}, "filter": {"operator": "contains", "field": "title", "value": "Cry like a baby"}, "properties": ["title", "track", "artist", "album", "fanart", "thumbnail", "file"], "limits": {"start": 0, "end": 400}}, "method": "AudioLibrary.GetSongs", "id": "libSongs"}' -H 'content-type: application/json;' http://localhost:8080/jsonrpc

Response before (ie. without PR):

{
   "id":"libSongs",
   "jsonrpc":"2.0",
   "result":{
      "limits":{
         "end":1,
         "start":0,
         "total":1
      },
      "songs":[
         {
            "album":"Sparkle in the Rain",
            "artist":[
               "Simple Minds"
            ],
            "fanart":"image://http%3a%2f%2fassets.fanart.tv%2ffanart%2fmusic%2ff41490ce-fe39-435d-86c0-ab5ce098b423%2fartistbackground%2fsimple-minds-4ffd2f8913a09.jpg/",
            "file":"nfs://192.168.0.3/mnt/share/data/Music/MP3/Simple Minds/Sparkle In The Rain/Simple Minds - 'C' Moon Cry Like A Baby.mp3",
            "label":"\"C\" Moon Cry Like A Baby",
            "songid":17702,
            "thumbnail":"image://nfs%3a%2f%2f192.168.0.3%2fmnt%2fshare%2fdata%2fMusic%2fMP3%2fSimple%20Minds%2fSparkle%20In%20The%20Rain%2fcover.jpg/",
            "title":"\"C\" Moon Cry Like A Baby",
            "track":8
         }
      ]
   }
}

Response after (with latest PR):

{
   "id":"libSongs",
   "jsonrpc":"2.0",
   "result":{
      "limits":{
         "end":1,
         "start":0,
         "total":1
      },
      "songs":[
         {
            "album":"Sparkle in the Rain",
            "artist":[
               "Simple Minds"
            ],
            "fanart":"image://http%3a%2f%2fassets.fanart.tv%2ffanart%2fmusic%2ff41490ce-fe39-435d-86c0-ab5ce098b423%2fartistbackground%2fsimple-minds-4ffd2f8913a09.jpg/",
            "file":"nfs://192.168.0.3/mnt/share/data/Music/MP3/Simple Minds/Sparkle In The Rain/Simple Minds - 'C' Moon Cry Like A Baby.mp3",
            "label":"\"C\" Moon Cry Like A Baby",
            "songid":17702,
            "thumbnail":"",
            "title":"\"C\" Moon Cry Like A Baby",
            "track":8
         }
      ]
   }
}

There's also some other differences (the before/after file sizes are not the same, even when taking the lack of thumbnails into account), so I need to look at that and determine what is causing the difference.

@MilhouseVH
Copy link
Contributor

MilhouseVH commented Jul 14, 2018

I think this is the final set of differences, at least as far as AudioLibrary.GetSongs is concerned.

(all 3 of the following issues are most likely one and the same)

  1. Missing artist now returned as "[Missing Tag]":

Query:

curl -s --data-binary '{"jsonrpc": "2.0", "params": {"sort": {"order": "ascending", "method": "title"}, "filter": {"operator": "contains", "field": "title", "value": "End Credits"}, "properties": ["title", "track", "artist", "album", "fanart", "thumbnail", "file"], "limits": {"start": 0, "end": 400}}, "method": "AudioLibrary.GetSongs", "id": "libSongs"}' -H 'content-type: application/json;' http://localhost:8080/jsonrpc

Response before (without PR):

{
   "id":"libSongs",
   "jsonrpc":"2.0",
   "result":{
      "limits":{
         "end":1,
         "start":0,
         "total":1
      },
      "songs":[
         {
            "album":"Tiësto in Concert (DVD - AC3)",
            "artist":[
               ""
            ],
            "fanart":"",
            "file":"nfs://192.168.0.3/mnt/share/data/Music/MP3/Tiësto/Tiësto in Concert (DVD - AC3)/40. End Credits.flac",
            "label":"End Credits",
            "songid":21631,
            "thumbnail":"image://nfs%3a%2f%2f192.168.0.3%2fmnt%2fshare%2fdata%2fMusic%2fMP3%2fTi%c3%absto%2fTi%c3%absto%20in%20Concert%20(DVD%20-%20AC3)%2fcover.jpg/",
            "title":"End Credits",
            "track":40
         }
      ]
   }
}

Reponse after (with latest PR):

{
   "id":"libSongs",
   "jsonrpc":"2.0",
   "result":{
      "limits":{
         "end":1,
         "start":0,
         "total":1
      },
      "songs":[
         {
            "album":"Tiësto in Concert (DVD - AC3)",
            "artist":[
               "[Missing Tag]"
            ],
            "fanart":"",
            "file":"nfs://192.168.0.3/mnt/share/data/Music/MP3/Tiësto/Tiësto in Concert (DVD - AC3)/40. End Credits.flac",
            "label":"End Credits",
            "songid":21631,
            "thumbnail":"",
            "title":"End Credits",
            "track":40
         }
      ]
   }
}

Response after, when calling AudioLibrary.GetSongDetails:

{
   "id":"libSongs",
   "jsonrpc":"2.0",
   "result":{
      "songdetails":{
         "album":"Tiësto in Concert (DVD - AC3)",
         "artist":[
            ""
         ],
         "fanart":"",
         "file":"nfs://192.168.0.3/mnt/share/data/Music/MP3/Tiësto/Tiësto in Concert (DVD - AC3)/40. End Credits.flac",
         "label":"End Credits",
         "songid":21631,
         "thumbnail":"image://nfs%3a%2f%2f192.168.0.3%2fmnt%2fshare%2fdata%2fMusic%2fMP3%2fTi%c3%absto%2fTi%c3%absto%20in%20Concert%20(DVD%20-%20AC3)%2fcover.jpg/",
         "title":"End Credits",
         "track":40
      }
   }
}
  1. Artist encoding changes:

Query:

curl -s --data-binary '{"jsonrpc": "2.0", "params": {"sort": {"order": "ascending", "method": "title"}, "filter": {"operator": "contains", "field": "title", "value": "You Made Me The Thief Of Your Heart"}, "properties": ["title", "track", "artist", "album", "fanart", "thumbnail", "file"], "limits": {"start": 0, "end": 400}}, "method": "AudioLibrary.GetSongs", "id": "libSongs"}' -H 'content-type: application/json;' http://localhost:8080/jsonrpc

Before:

{
   "id":"libSongs",
   "jsonrpc":"2.0",
   "result":{
      "limits":{
         "end":1,
         "start":0,
         "total":1
      },
      "songs":[
         {
            "album":"",
            "artist":[
               "Sinead O'Connor"
            ],
            "fanart":"",
            "file":"nfs://192.168.0.3/mnt/share/data/Music/MP3/Station Ripper2/Sinéad O'Connor - You Made Me The Thief Of Your Heart.mp3",
            "label":"You Made Me The Thief Of Your Heart",
            "songid":20074,
            "thumbnail":"",
            "title":"You Made Me The Thief Of Your Heart",
            "track":0
         }
      ]
   }
}

After:

{
   "id":"libSongs",
   "jsonrpc":"2.0",
   "result":{
      "limits":{
         "end":1,
         "start":0,
         "total":1
      },
      "songs":[
         {
            "album":"",
            "artist":[
               "Sinéad O'Connor"
            ],
            "fanart":"",
            "file":"nfs://192.168.0.3/mnt/share/data/Music/MP3/Station Ripper2/Sinéad O'Connor - You Made Me The Thief Of Your Heart.mp3",
            "label":"You Made Me The Thief Of Your Heart",
            "songid":20074,
            "thumbnail":"",
            "title":"You Made Me The Thief Of Your Heart",
            "track":0
         }
      ]
   }
}

Sinead -> Sinéad

Response after, when calling AudioLibrary.GetSongDetails:

{
   "id":"libSongs",
   "jsonrpc":"2.0",
   "result":{
      "songdetails":{
         "album":"",
         "artist":[
            "Sinead O'Connor"
         ],
         "fanart":"",
         "file":"nfs://192.168.0.3/mnt/share/data/Music/MP3/Station Ripper2/Sinéad O'Connor - You Made Me The Thief Of Your Heart.mp3",
         "label":"You Made Me The Thief Of Your Heart",
         "songid":20074,
         "thumbnail":"",
         "title":"You Made Me The Thief Of Your Heart",
         "track":0
      }
   }
}

s1

s2

  1. Artist capitalisation changes:

Comparing random artist, before (-) and after (+):

@@ -35310,7 +35310,7 @@
   {
     "album": "Deadringer",
     "artist": [
-      "RJD2"
+      "RjD2"
     ],
     "fanart": "image://http://assets.fanart.tv/fanart/music/1a120ec5-35b3-4f3e-aa49-7201550d9feb/artistbackground/rjd2-4ddc5606c42bc.jpg/",
     "file": "nfs://192.168.0.3/mnt/share/data/Music/MP3/RjD2/Deadringer/RjD2 - The Horror.mp3",

s1

s2

Again, AudioLibrary.GetSongDetails returns the original RJD2 not RjD2.

The only source of "RJD2" appears to be the artist table.


1, 2 and 3 are likely all related. It now appears this PR, when calling AudioLibrary.GetSongs, is returning artist from the song_artist table whereas before it came from the artist table.

However AudioLibrary.GetSongDetails, with this PR, now returns a different artist, presumably from the artist table.

@DaveTBlake
Copy link
Member Author

Thanks for further tests @MilhouseVH
GetSongs

  • yes "artist" needed to come from artist table (1, 2, 3 all related). Fixed
  • missing "thumbnail" was when "albumid" not also requested. Fixed

GetArtists

  • "art" only returned when requested (not accidentally with "thumbnail" or "fanart")
  • Empty art on MySQL correctly shown as "art": { }

@DaveTBlake DaveTBlake force-pushed the JSONFastMusic branch 2 times, most recently from 6a846c0 to b97ce38 Compare July 14, 2018 16:30
…implemented to be many times faster. Minor patch bump despite a significant speed improvement and fix of several bugs
@MilhouseVH
Copy link
Contributor

MilhouseVH commented Jul 15, 2018

Thanks @DaveTBlake - your latest updates now produce 100% identical results when compared with a build without this PR, ie. LibreELEC build #0707 (methods tested: GetArtists, GetAlbums, GetSongs).

I know you're keen to merge this, it's probably the best way to shake out any remaining issues (hopefully there aren't any).


Performance comparison using MySQL 5.5.21 on HP N36L

Albums     : 351
Artists    : 1537
Songs      : 7384

Old/New timings are in seconds, averaged over 3 runs each.

JSON data retrieved iteratively in chunks of 400 items - timings are for the complete data set.

Tested using texturecache.py, eg. texturecache.py j artists >/dev/null

The following JSON calls methods/properties/sort orders were used:

AudioLibrary.GetArtists

{"jsonrpc": "2.0", "params": {"sort": {"order": "ascending", "method": "artist"}, "albumartistsonly": false, "properties": ["fanart", "thumbnail"], "limits": {"start": 0, "end": 400}}, "method": "AudioLibrary.GetArtists", "id": "libArtists"}

AudioLibrary.GetAlbums:

{"jsonrpc": "2.0", "params": {"sort": {"order": "ascending", "method": "label"}, "properties": ["title", "artist", "fanart", "thumbnail"], "limits": {"start": 0, "end": 400}}, "method": "AudioLibrary.GetAlbums", "id": "libAlbums"}

AudioLibrary.GetSongs:

{"jsonrpc": "2.0", "params": {"sort": {"order": "ascending", "method": "title"}, "properties": ["title", "track", "artist", "album", "fanart", "thumbnail", "file"], "limits": {"start": 0, "end": 400}}, "method": "AudioLibrary.GetSongs", "id": "libSongs"}

RPi1 (512MB RAM, 10MB/s ethernet):

Method Old New
AudioLibrary.GetArtists 22.615 5.046
AudioLibrary.GetAlbums 7.055 4.531
AudioLibrary.GetSongs 218.391 50.844

RPi3+ (1GB RAM, 30MB/s ethernet):

Method Old New
AudioLibrary.GetArtists 3.865 1.340
AudioLibrary.GetAlbums 1.204 1.436
AudioLibrary.GetSongs 42.823 15.588

Skylake i5 NUC (8GB RAM, 30MB/s 5GHz WiFi):

Method Old New
AudioLibrary.GetArtists 8.410 0.421
AudioLibrary.GetAlbums 1.318 1.036
AudioLibrary.GetSongs 39.441 22.384

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
API change: JSON-RPC Component: Music Type: Fix non-breaking change which fixes an issue Type: Improvement non-breaking change which improves existing functionality v18 Leia
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants