Add index on delayed_jobs (locked_at, locked_by) #71

Open
wants to merge 2 commits into
from

Conversation

Projects
None yet
3 participants

spajus commented Aug 14, 2013

This index avoids an increasingly growing performance penalty when delayed_jobs table tends to grow bigger.

Before index:

mysql> explain SELECT  `delayed_jobs`.* FROM `delayed_jobs`  WHERE `delayed_jobs`.`locked_at` = '2013-08-13 13:03:06' AND `delayed_jobs`.`locked_by` = 'delayed_job host:private.internal pid:12345' LIMIT 1;
+----+-------------+--------------+------+---------------+------+---------+------+--------+-------------+
| id | select_type | table        | type | possible_keys | key  | key_len | ref  | rows   | Extra       |
+----+-------------+--------------+------+---------------+------+---------+------+--------+-------------+
|  1 | SIMPLE      | delayed_jobs | ALL  | NULL          | NULL | NULL    | NULL | 260546 | Using where |
+----+-------------+--------------+------+---------------+------+---------+------+--------+-------------+

After:

mysql> explain SELECT  `delayed_jobs`.* FROM `delayed_jobs`  WHERE `delayed_jobs`.`locked_at` = '2013-08-13 13:03:06' AND `delayed_jobs`.`locked_by` = 'delayed_job host:private.internal pid:12345' LIMIT 1;
+----+-------------+--------------+------+-------------------+-------------------+---------+-------------+------+-------------+
| id | select_type | table        | type | possible_keys     | key               | key_len | ref         | rows | Extra       |
+----+-------------+--------------+------+-------------------+-------------------+---------+-------------+------+-------------+
|  1 | SIMPLE      | delayed_jobs | ref  | delayed_jobs_lock | delayed_jobs_lock | 312     | const,const |    1 | Using where |
+----+-------------+--------------+------+-------------------+-------------------+---------+-------------+------+-------------+

Visual performance impact after this migration was applied to a production app during rush hour:
preview

Kudos to @tomas-didziokas for suggesting this fix.

@spajus spajus Add index on delayed_jobs (locked_at, locked_by)
This index avoids an increasingly growing performance penalty when delayed_jobs table tends to
grow bigger.
1f41fe0

Coverage Status

Changes Unknown when pulling 1f41fe0 on spajus:lock_index into * on collectiveidea:master*.

@spajus spajus Add index on delayed_jobs (run_at, failed_at, priority)
When lots of failed jobs occur, this index gives huge performance benefit on high loads with big
delayed_jobs table.
41bbfc2

Coverage Status

Changes Unknown when pulling 41bbfc2 on spajus:lock_index into * on collectiveidea:master*.

please

sorry looking at my query log, I see a lot of
SELECT COUNT(*) FROMdelayed_jobsWHERE (failed_at is not NULL) and SELECT COUNT(*) FROMdelayed_jobsWHERE (locked_by is not NULL)

shouldn't those be top level indeces?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment