Fix slow/unindexed queries

[shtrom]

Q	A
Bug fix?	yes
New feature?	no
BC breaks?	no
Deprecations?	no
Tests pass?	NO
Documentation	no
Translation	no
CHANGELOG.md	yes
License	MIT

Fixes #4079, #4080

@j0k3r I updated the entities and generated the migrations with doctrine:migrations:diff, but I did struggle and have to go back and forth fiddling with the files (for some reason the first diff would generate a whole lot of unrelated migrations).

Now, the tests fail on the change of the the language column, which I suspect is due to SQLite3 being used for the test, and not supporting this sort of change. I'm not sure how to proceed with that particular migration (save from dumping/restoring all the data, but it feels quite overkill).

[j0k3r]

That's a start.
Migration in SQLite are quite complex (but well handled by doctrine) but yes, when changing a column type in SQLite require some big query. I'll try to help here

[virtadpt]

If you like, I have slow query logging active on the server I run Wallabag on. I can check the MySQL log and see if anything from Wallabag wound up in there.

[shtrom]

Yeah, my guess is that the migration need to move away from raw SQL, and use Doctrine instead, but I'm not familiar enough with it just yet :

[shtrom]

If you like, I have slow query logging active on the server I run Wallabag on. I can check the MySQL log and see if anything from Wallabag wound up in there.

Yeah, same; that's how I keep finding slow queries in a few projects I use. Mainly because my server is so underpowered 😬

[j0k3r]

I've fixed migrations.

[shtrom]

Cool. Yeah SQLite3 is messy.

[shtrom]

Damn, hold on, I hit a new slow query now.

[shtrom]

mariadb root@localhost:wallabag> explain SELECT w0_.id AS id_0 FROM `wallabag_entry` w0_ WHERE w0_.user_id = 1 AND w0_.is_archived = 1 GROUP BY w0_.id ORDER BY w0_.archived_at DESC\G           
***************************[ 1. row ]***************************                                                                                                                                 
id            | 1                                                                                                                                                                                
select_type   | SIMPLE                                                                                                                                                                           
table         | w0_
type          | ref
possible_keys | IDX_F4D18282A76ED395,IDX_entry_archived,hashed_url_user_id,idx_user_starred_at
key           | hashed_url_user_id
key_len       | 5
ref           | const
rows          | 2310
Extra         | Using where; Using filesort
1 row in set
Time: 0.137s

Same fix as tagged:

mariadb root@localhost:wallabag> CREATE INDEX test_user_archived ON wallabag_entry (user_id, is_archived, archived_at)                                                                           
Query OK, 0 rows affected
Time: 50.549s
mariadb root@localhost:wallabag> explain SELECT w0_.id AS id_0 FROM `wallabag_entry` w0_ WHERE w0_.user_id = 1 AND w0_.is_archived = 1 GROUP BY w0_.id ORDER BY w0_.archived_at DESC\G           
***************************[ 1. row ]***************************
id            | 1
select_type   | SIMPLE
table         | w0_
type          | ref
possible_keys | IDX_F4D18282A76ED395,IDX_entry_archived,hashed_url_user_id,idx_user_starred_at,test_user_archived
key           | test_user_archived
key_len       | 6
ref           | const,const
rows          | 2310
Extra         | Using where
1 row in set
Time: 1.544s

I'll add that to the PR.

[shtrom]

Same for created_at. This looks like the end of it, though.

[j0k3r]

wallabag will be really faster after that 🚀

[j0k3r]

@virtadpt did you find some other slow queries?

[virtadpt]

I have not.
My server rotates its logs daily and persists them for one calendar week. For the last seven days, my mysql-slow.log file has been empty (I've been running Wallabag on that server for just under three months now).

[shtrom]

wallabag will be really faster after that rocket

Yeah. tbh, it's an issue I have with ~2k entries, but mainly because my server is so small.

One thing I wonder, though, is how wallabag.it and other shared instances with many users handle this.

[shtrom]

My server rotates its logs daily and persists them for one calendar week. For the last seven days, my mysql-slow.log file has been empty (I've been running Wallabag on that server for just under three months now).

You can tell MySQL to also log queries that are not using indices. This makes the slow_log substantially noisier, but this might help find problematic queries before they are actually slow.

[j0k3r]

One thing I wonder, though, is how wallabag.it and other shared instances with many users handle this.

I should try to have a look at those log on wallabag.it

[j0k3r]

Checking the mariadb-slow.log on app.wallabag.it for yesterday (only) and found some other slow queries:

MariaDB [app_wallabag_it]> explain SELECT **** FROM `wallabag_tag` t0 WHERE t0.label = 'zap' LIMIT 1\G
*************************** 1. row ***************************
           id: 1
  select_type: SIMPLE
        table: t0
         type: ALL
possible_keys: NULL
          key: NULL
      key_len: NULL
          ref: NULL
         rows: 63841
        Extra: Using where
1 row in set (0.00 sec)

That one is easy to fix with an index on the label of the tag.

CREATE INDEX tag_label ON wallabag_tag (label (255));

Then I found that one:

MariaDB [app_wallabag_it]> explain SELECT **** FROM `wallabag_user` w0_ LEFT JOIN `wallabag_config` w1_ ON w0_.id = w1_.user_id WHERE w1_.rss_token = '****' AND w0_.username = '****'\G
*************************** 1. row ***************************
           id: 1
  select_type: SIMPLE
        table: w1_
         type: ALL
possible_keys: UNIQ_87E64C53A76ED395
          key: NULL
      key_len: NULL
          ref: NULL
         rows: 9319
        Extra: Using where
*************************** 2. row ***************************
           id: 1
  select_type: SIMPLE
        table: w0_
         type: eq_ref
possible_keys: PRIMARY
          key: PRIMARY
      key_len: 4
          ref: app_wallabag_it.w1_.user_id
         rows: 1
        Extra: Using where
2 rows in set (0.00 sec)

Which is also easy to fix:

CREATE INDEX config_rss_token ON wallabag_config (rss_token (255));

[j0k3r]

Good job @shtrom 👏
If you found more slow queries, don't hesitate to open a PR directly (same goes for @virtadpt)