Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix slow/unindexed queries #4081

Merged
merged 8 commits into from Aug 8, 2019

Conversation

@shtrom
Copy link
Contributor

commented Aug 6, 2019

Q A
Bug fix? yes
New feature? no
BC breaks? no
Deprecations? no
Tests pass? NO
Documentation no
Translation no
CHANGELOG.md yes
License MIT

Fixes #4079, #4080

@j0k3r I updated the entities and generated the migrations with doctrine:migrations:diff, but I did struggle and have to go back and forth fiddling with the files (for some reason the first diff would generate a whole lot of unrelated migrations).

Now, the tests fail on the change of the the language column, which I suspect is due to SQLite3 being used for the test, and not supporting this sort of change. I'm not sure how to proceed with that particular migration (save from dumping/restoring all the data, but it feels quite overkill).

shtrom added some commits Aug 5, 2019

Update language in Entry model, and add index
Fixes #4080

Signed-off-by: Olivier Mehani <shtrom@ssji.net>
Add index on starred entries
Fixes #4079

Signed-off-by: Olivier Mehani <shtrom@ssji.net>

@shtrom shtrom force-pushed the shtrom:fix-slow-queries branch from a593e9c to f5527fc Aug 6, 2019

@j0k3r j0k3r added this to To Do in 2.4 via automation Aug 6, 2019

@j0k3r j0k3r added this to the 2.4.0 milestone Aug 6, 2019

@j0k3r
Copy link
Member

left a comment

That's a start.
Migration in SQLite are quite complex (but well handled by doctrine) but yes, when changing a column type in SQLite require some big query. I'll try to help here

CHANGELOG.md Outdated Show resolved Hide resolved
@virtadpt

This comment has been minimized.

Copy link

commented Aug 6, 2019

If you like, I have slow query logging active on the server I run Wallabag on. I can check the MySQL log and see if anything from Wallabag wound up in there.

@shtrom

This comment has been minimized.

Copy link
Contributor Author

commented Aug 6, 2019

Yeah, my guess is that the migration need to move away from raw SQL, and use Doctrine instead, but I'm not familiar enough with it just yet :

@shtrom

This comment has been minimized.

Copy link
Contributor Author

commented Aug 6, 2019

If you like, I have slow query logging active on the server I run Wallabag on. I can check the MySQL log and see if anything from Wallabag wound up in there.

Yeah, same; that's how I keep finding slow queries in a few projects I use. Mainly because my server is so underpowered 😬

@j0k3r j0k3r force-pushed the shtrom:fix-slow-queries branch from f5527fc to 24acf2e Aug 7, 2019

@j0k3r

This comment has been minimized.

Copy link
Member

commented Aug 7, 2019

I've fixed migrations.

@shtrom

This comment has been minimized.

Copy link
Contributor Author

commented Aug 7, 2019

Cool. Yeah SQLite3 is messy.

@shtrom

This comment has been minimized.

Copy link
Contributor Author

commented Aug 7, 2019

Damn, hold on, I hit a new slow query now.

@shtrom

This comment has been minimized.

Copy link
Contributor Author

commented Aug 7, 2019

mariadb root@localhost:wallabag> explain SELECT w0_.id AS id_0 FROM `wallabag_entry` w0_ WHERE w0_.user_id = 1 AND w0_.is_archived = 1 GROUP BY w0_.id ORDER BY w0_.archived_at DESC\G           
***************************[ 1. row ]***************************                                                                                                                                 
id            | 1                                                                                                                                                                                
select_type   | SIMPLE                                                                                                                                                                           
table         | w0_
type          | ref
possible_keys | IDX_F4D18282A76ED395,IDX_entry_archived,hashed_url_user_id,idx_user_starred_at
key           | hashed_url_user_id
key_len       | 5
ref           | const
rows          | 2310
Extra         | Using where; Using filesort
1 row in set
Time: 0.137s

Same fix as tagged:

mariadb root@localhost:wallabag> CREATE INDEX test_user_archived ON wallabag_entry (user_id, is_archived, archived_at)                                                                           
Query OK, 0 rows affected
Time: 50.549s
mariadb root@localhost:wallabag> explain SELECT w0_.id AS id_0 FROM `wallabag_entry` w0_ WHERE w0_.user_id = 1 AND w0_.is_archived = 1 GROUP BY w0_.id ORDER BY w0_.archived_at DESC\G           
***************************[ 1. row ]***************************
id            | 1
select_type   | SIMPLE
table         | w0_
type          | ref
possible_keys | IDX_F4D18282A76ED395,IDX_entry_archived,hashed_url_user_id,idx_user_starred_at,test_user_archived
key           | test_user_archived
key_len       | 6
ref           | const,const
rows          | 2310
Extra         | Using where
1 row in set
Time: 1.544s

I'll add that to the PR.

shtrom added some commits Aug 7, 2019

Add user_archived index
Signed-off-by: Olivier Mehani <shtrom@ssji.net>
Add user_created index
Signed-off-by: Olivier Mehani <shtrom@ssji.net>
@shtrom

This comment has been minimized.

Copy link
Contributor Author

commented Aug 7, 2019

Same for created_at. This looks like the end of it, though.

@shtrom shtrom changed the title Fix slow queries Fix slow/unindexed queries Aug 7, 2019

@j0k3r

This comment has been minimized.

Copy link
Member

commented Aug 7, 2019

wallabag will be really faster after that 🚀

@j0k3r

This comment has been minimized.

Copy link
Member

commented Aug 7, 2019

@virtadpt did you find some other slow queries?

@virtadpt

This comment has been minimized.

Copy link

commented Aug 7, 2019

I have not.
My server rotates its logs daily and persists them for one calendar week. For the last seven days, my mysql-slow.log file has been empty (I've been running Wallabag on that server for just under three months now).

@shtrom

This comment has been minimized.

Copy link
Contributor Author

commented Aug 7, 2019

wallabag will be really faster after that rocket

Yeah. tbh, it's an issue I have with ~2k entries, but mainly because my server is so small.

One thing I wonder, though, is how wallabag.it and other shared instances with many users handle this.

@shtrom

This comment has been minimized.

Copy link
Contributor Author

commented Aug 7, 2019

My server rotates its logs daily and persists them for one calendar week. For the last seven days, my mysql-slow.log file has been empty (I've been running Wallabag on that server for just under three months now).

You can tell MySQL to also log queries that are not using indices. This makes the slow_log substantially noisier, but this might help find problematic queries before they are actually slow.

@j0k3r

This comment has been minimized.

Copy link
Member

commented Aug 8, 2019

One thing I wonder, though, is how wallabag.it and other shared instances with many users handle this.

I should try to have a look at those log on wallabag.it

@j0k3r

This comment has been minimized.

Copy link
Member

commented Aug 8, 2019

Checking the mariadb-slow.log on app.wallabag.it for yesterday (only) and found some other slow queries:

MariaDB [app_wallabag_it]> explain SELECT **** FROM `wallabag_tag` t0 WHERE t0.label = 'zap' LIMIT 1\G
*************************** 1. row ***************************
           id: 1
  select_type: SIMPLE
        table: t0
         type: ALL
possible_keys: NULL
          key: NULL
      key_len: NULL
          ref: NULL
         rows: 63841
        Extra: Using where
1 row in set (0.00 sec)

That one is easy to fix with an index on the label of the tag.

CREATE INDEX tag_label ON wallabag_tag (label (255));

Then I found that one:

MariaDB [app_wallabag_it]> explain SELECT **** FROM `wallabag_user` w0_ LEFT JOIN `wallabag_config` w1_ ON w0_.id = w1_.user_id WHERE w1_.rss_token = '****' AND w0_.username = '****'\G
*************************** 1. row ***************************
           id: 1
  select_type: SIMPLE
        table: w1_
         type: ALL
possible_keys: UNIQ_87E64C53A76ED395
          key: NULL
      key_len: NULL
          ref: NULL
         rows: 9319
        Extra: Using where
*************************** 2. row ***************************
           id: 1
  select_type: SIMPLE
        table: w0_
         type: eq_ref
possible_keys: PRIMARY
          key: PRIMARY
      key_len: 4
          ref: app_wallabag_it.w1_.user_id
         rows: 1
        Extra: Using where
2 rows in set (0.00 sec)

Which is also easy to fix:

CREATE INDEX config_rss_token ON wallabag_config (rss_token (255));

@j0k3r j0k3r added the Improvement label Aug 8, 2019

CS

@j0k3r j0k3r requested review from Kdecherf and tcitworld Aug 8, 2019

@j0k3r

This comment has been minimized.

Copy link
Member

commented Aug 8, 2019

Good job @shtrom 👏
If you found more slow queries, don't hesitate to open a PR directly (same goes for @virtadpt)

@j0k3r j0k3r merged commit 97c678a into wallabag:master Aug 8, 2019

2 checks passed

Scrutinizer No new issues
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details

2.4 automation moved this from To Do to Done Aug 8, 2019

@shtrom shtrom deleted the shtrom:fix-slow-queries branch Aug 9, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
4 participants
You can’t perform that action at this time.