Scan for empty book series more efficiently

[selfhost-alt]

I've been trying to figure out why my server was starting so slowly after the Sqlite migration, and I found that the scan for empty book series seems to be the place where it spends the most time during startup.

I believe this slowness is due to the fact that it does a sub-query in the WHERE clause, which gets pretty inefficient when you have a large series table (as I do).

This is the original query that gets run before my change:

SELECT `id`, `name`, `nameIgnorePrefix`, `description`, `createdAt`, `updatedAt`, `libraryId`
FROM `series` AS `series`
WHERE (SELECT count(*) FROM bookSeries bs WHERE bs.seriesId = series.id) = 0;

On my dev container, against a snapshot of my DB (10k series entries, 36k bookSeries entries), this takes just under 10 minutes to complete.

With this PR, the query changes to the following:

SELECT
  `series`.`id`,
  `series`.`name`,
  `series`.`nameIgnorePrefix`,
  `series`.`description`,
  `series`.`createdAt`,
  `series`.`updatedAt`,
  `series`.`libraryId`,
  `bookSeries`.`id` AS `bookSeries.id`,
  `bookSeries`.`sequence` AS `bookSeries.sequence`,
  `bookSeries`.`createdAt` AS `bookSeries.createdAt`,
  `bookSeries`.`bookId` AS `bookSeries.bookId`,
  `bookSeries`.`seriesId` AS `bookSeries.seriesId`
FROM `series` AS `series`
LEFT OUTER JOIN `bookSeries` AS `bookSeries`
ON `series`.`id` = `bookSeries`.`seriesId`
WHERE `bookSeries`.`id` IS NULL;

On my dev container against the same DB snapshot, this query takes about 300ms.

[selfhost-alt]

btw there are other similar slow queries that run at startup, like the one that checks for books with no media, but the empty series scan was the most painfully slow one for me, so I started with this.

[selfhost-alt]

Actually, doing a COUNT doesn't quite do what we want. I will take another pass at this and see if I can come up with the right query.

[advplyr]

Thanks, I went and fixed the other 2 happening in that function. There are many more sql queries to optimize as well, unfortunately more complicated then those.

[selfhost-alt]

Great! I think the next optimizations I will try to tackle are some of the recommendations queries for the library home page. On my library the home page takes 30+ seconds to load, so I'm sure there are some optimization opportunities.

…

On Sun, Sep 17, 2023, 1:54 PM advplyr ***@***.***> wrote: Merged #2103 <#2103> into master. — Reply to this email directly, view it on GitHub <#2103 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AV3GPLFU3LP5GTFWTSA2NK3X25PRJANCNFSM6AAAAAA4ZKD5ME> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[advplyr]

Your help with that would be great. I know that one of the main issues with the home page & library queries is they are not using the proper indexes. I've been using the EXPLAIN QUERY PLAN command to check the indexes

[BlackHoleFox]

@selfhost-alt You seem to be running into the same thing I am on the library home page if its that slow. Wrote about it here since it seemed most relevant but I suspect something is wrong with how cached images are getting served.

[thomcc]

I wonder if this could have also been fixed by just adding an index on seriesId (I suspect so, but it's hard to say. It's plausible that this change is faster than that regardless).

I've been using the EXPLAIN QUERY PLAN command to check the indexes

I used to work on Firefox, and the expert extension https://www.sqlite.org/src/dir?ci=trunk&name=ext/expert was a great help for optimizing the history/bookmarks (places) database.

I believe the extension is now integrated in the SQLite CLI via the .expert command, but I haven't had a lot of experience using it that way. Given that compiling the tool I linked was always kind of a pain, I'd recommend investigating this.