add index on statuses for /api/v1/accounts/:account_id/statuses

[takayamaki]

pghero shows me bellow slow query.

SELECT  "statuses"."id", "statuses"."updated_at" FROM "statuses" WHERE "statuses"."account_id" = ? AND "statuses"."visibility" IN (?, ?) AND "statuses"."id" IN (SELECT DISTINCT "media_attachments"."status_id" FROM "media_attachments" WHERE "media_attachments"."account_id" = ? AND ("media_attachments"."status_id" IS NOT NULL)) AND (statuses.reply = ? OR statuses.in_reply_to_account_id = statuses.account_id) ORDER BY "statuses"."id" DESC LIMIT ?

SELECT  "statuses"."id", "statuses"."updated_at" FROM "statuses" WHERE ("statuses"."account_id" = ? AND "statuses"."visibility" IN (?, ?) OR "statuses"."account_id" = ? AND "statuses"."id" IN (SELECT "mentions"."status_id" FROM "mentions" WHERE "mentions"."account_id" = ?)) AND "statuses"."id" IN (SELECT DISTINCT "media_attachments"."status_id" FROM "media_attachments" WHERE "media_attachments"."account_id" = ? AND ("media_attachments"."status_id" IS NOT NULL)) ORDER BY "statuses"."id" DESC LIMIT ?

These statements is part of /api/v1/accounts/:account_id/statuses and /api/v1/accounts/:account_id/statuses?only_media=true.
This PR improve speed of these statements.

before

mastodon_production=> explain analyze SELECT "statuses"."id", "statuses"."updated_at" FROM "statuses" WHERE ("statuses"."account_id" = xxxxx AND "statuses"."visibility" IN (0, 1) OR "statuses"."account_id" = xxxxx AND "statuses"."id" IN (SELECT "men
tions"."status_id" FROM "mentions" WHERE "mentions"."account_id" = yyyyy)) ORDER BY "statuses"."id" DESC LIMIT 20;
                                                                                QUERY PLAN                                                                                 
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Limit  (cost=65.89..142.35 rows=20 width=16) (actual time=2.038..3698.471 rows=20 loops=1)
   ->  Index Scan Backward using index_statuses_on_account_id_id on statuses  (cost=65.89..228203.94 rows=59672 width=16) (actual time=2.036..3698.456 rows=20 loops=1)
         Index Cond: (account_id = xxxxx)
         Filter: ((visibility = ANY ('{0,1}'::integer[])) OR (hashed SubPlan 1))
         Rows Removed by Filter: 13448
         SubPlan 1
           ->  Index Only Scan using index_mentions_on_account_id_and_status_id on mentions  (cost=0.42..65.32 rows=55 width=8) (actual time=0.065..1.482 rows=25 loops=1)
                 Index Cond: (account_id = yyyyy)
                 Heap Fetches: 22
 Planning time: 0.258 ms
 Execution time: 3698.526 ms
(11 rows)

mastodon_production=>

mastodon_production=> explain analyze SELECT "statuses"."id", "statuses"."updated_at" FROM "statuses" WHERE ("statuses"."account_id" = xxxxx AND "statuses"."visibility" IN (0, 1, 2) OR "statuses"."account_id" = xxxxx AND "statuses"."id" IN (SELECT "mentions"."status_id" FROM "mentions" WHERE "mentions"."account_id" = yyyyy)) AND "statuses"."id" IN (SELECT DISTINCT "media_attachments"."status_id" FROM "media_attachments" WHERE "media_attachments"."account_id" = xxxxx AND ("media_attachments"."status_id" IS NOT NULL)) ORDER BY "statuses"."id" DESC LIMIT 40;
                                                                                      QUERY PLAN                                                                                       
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Limit  (cost=82655.91..82655.93 rows=5 width=16) (actual time=6340.909..6340.935 rows=40 loops=1)
   ->  Sort  (cost=82655.91..82655.93 rows=5 width=16) (actual time=6340.907..6340.919 rows=40 loops=1)
         Sort Key: statuses.id DESC
         Sort Method: top-N heapsort  Memory: 26kB
         ->  Hash Semi Join  (cost=6382.12..82655.85 rows=5 width=16) (actual time=23.130..6338.864 rows=1316 loops=1)
               Hash Cond: (statuses.id = media_attachments.status_id)
               ->  Bitmap Heap Scan on statuses  (cost=1153.00..77364.93 rows=23522 width=16) (actual time=8.729..6308.794 rows=26106 loops=1)
                     Recheck Cond: (account_id = xxxxx)
                     Filter: ((visibility = ANY ('{0,1,2}'::integer[])) OR (hashed SubPlan 1))
                     Rows Removed by Filter: 53
                     Heap Blocks: exact=21873
                     ->  Bitmap Index Scan on index_statuses_on_account_id_id  (cost=0.00..1081.66 rows=23631 width=0) (actual time=4.716..4.716 rows=26273 loops=1)
                           Index Cond: (account_id = xxxxx)
                     SubPlan 1
                       ->  Index Only Scan using index_mentions_on_account_id_and_status_id on mentions  (cost=0.42..65.32 rows=55 width=8) (actual time=0.054..0.079 rows=18 loops=1)
                             Index Cond: (account_id = yyyyy)
                             Heap Fetches: 12
               ->  Hash  (cost=5214.83..5214.83 rows=1144 width=8) (actual time=8.292..8.292 rows=1316 loops=1)
                     Buckets: 2048  Batches: 1  Memory Usage: 68kB
                     ->  HashAggregate  (cost=5191.95..5203.39 rows=1144 width=8) (actual time=7.517..7.877 rows=1316 loops=1)
                           Group Key: media_attachments.status_id
                           ->  Bitmap Heap Scan on media_attachments  (cost=52.98..5187.90 rows=1619 width=8) (actual time=0.503..6.752 rows=1827 loops=1)
                                 Recheck Cond: (account_id = xxxxx)
                                 Filter: (status_id IS NOT NULL)
                                 Heap Blocks: exact=1500
                                 ->  Bitmap Index Scan on index_media_attachments_on_account_id  (cost=0.00..52.57 rows=1620 width=0) (actual time=0.306..0.306 rows=1830 loops=1)
                                       Index Cond: (account_id = xxxxx)
 Planning time: 1.188 ms
 Execution time: 6341.246 ms
(29 rows)

mastodon_production=>

after

mastodon_production=> explain analyze SELECT "statuses"."id", "statuses"."updated_at" FROM "statuses" WHERE ("statuses"."account_id" = xxxxx AND "statuses"."visibility" IN (0, 1) OR "statuses"."account_id" = xxxxx AND "statuses"."id" IN (SELECT "mentions"."status_id" FROM "mentions" WHERE "mentions"."account_id" = yyyyy)) ORDER BY "statuses"."id" DESC LIMIT 20;
                                                                                 QUERY PLAN                                                                                 
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Limit  (cost=66.01..76.77 rows=20 width=16) (actual time=10.460..373.748 rows=20 loops=1)
   ->  Index Only Scan Backward using index_20180106 on statuses  (cost=66.01..25782.60 rows=47811 width=16) (actual time=10.459..373.738 rows=20 loops=1)
         Index Cond: (account_id = xxxxx)
         Filter: ((visibility = ANY ('{0,1}'::integer[])) OR (hashed SubPlan 1))
         Rows Removed by Filter: 13450
         Heap Fetches: 8115
         SubPlan 1
           ->  Index Only Scan using index_mentions_on_account_id_and_status_id on mentions  (cost=0.42..65.32 rows=55 width=8) (actual time=0.698..10.262 rows=25 loops=1)
                 Index Cond: (account_id = yyyyy)
                 Heap Fetches: 22
 Planning time: 0.379 ms
 Execution time: 373.795 ms
(12 rows)

mastodon_production=>

mastodon_production=> explain analyze SELECT "statuses"."id", "statuses"."updated_at" FROM "statuses" WHERE ("statuses"."account_id" = xxxxx AND "statuses"."visibility" IN (0, 1, 2) OR "statuses"."account_id" = xxxxx AND "statuses"."id" IN (SELECT "mentions"."status_id" FROM "mentions" WHERE "mentions"."account_id" = yyyyy)) AND "statuses"."id" IN (SELECT DISTINCT "media_attachments"."status_id" FROM "media_attachments" WHERE "media_attachments"."account_id" = xxxxx AND ("media_attachments"."status_id" IS NOT NULL)) ORDER BY "statuses"."id" DESC LIMIT 40;
                                                                                   QUERY PLAN                                                                                    
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Limit  (cost=5338.95..18393.71 rows=5 width=16) (actual time=292.109..293.233 rows=40 loops=1)
   ->  Merge Semi Join  (cost=5338.95..18393.71 rows=5 width=16) (actual time=292.107..293.210 rows=40 loops=1)
         Merge Cond: (statuses.id = media_attachments.status_id)
         ->  Index Only Scan using index_20180106 on statuses  (cost=66.01..13056.18 rows=23525 width=16) (actual time=0.031..0.776 rows=505 loops=1)
               Index Cond: (account_id = xxxxx)
               Filter: ((visibility = ANY ('{0,1,2}'::integer[])) OR (hashed SubPlan 1))
               Rows Removed by Filter: 1
               Heap Fetches: 506
               SubPlan 1
                 ->  Index Only Scan using index_mentions_on_account_id_and_status_id on mentions  (cost=0.42..65.32 rows=55 width=8) (actual time=0.017..0.030 rows=18 loops=1)
                       Index Cond: (account_id = yyyyy)
                       Heap Fetches: 12
         ->  Sort  (cost=5272.94..5275.80 rows=1144 width=8) (actual time=292.055..292.066 rows=40 loops=1)
               Sort Key: media_attachments.status_id DESC
               Sort Method: quicksort  Memory: 110kB
               ->  HashAggregate  (cost=5191.95..5203.39 rows=1144 width=8) (actual time=291.063..291.527 rows=1316 loops=1)
                     Group Key: media_attachments.status_id
                     ->  Bitmap Heap Scan on media_attachments  (cost=52.98..5187.90 rows=1619 width=8) (actual time=0.419..289.136 rows=1827 loops=1)
                           Recheck Cond: (account_id = xxxxx)
                           Filter: (status_id IS NOT NULL)
                           Heap Blocks: exact=1500
                           ->  Bitmap Index Scan on index_media_attachments_on_account_id  (cost=0.00..52.57 rows=1620 width=0) (actual time=0.230..0.230 rows=1830 loops=1)
                                 Index Cond: (account_id = xxxxx)
 Planning time: 0.533 ms
 Execution time: 293.312 ms
(25 rows)

mastodon_production=>

essential speed

Below explain analyzes is execution time of the statements when all data is cached by memory.
It is the shortest execution time among multiple trials.

before

mastodon_production=> explain analyze SELECT "statuses"."id", "statuses"."updated_at" FROM "statuses" WHERE ("statuses"."account_id" = xxxxx AND "statuses"."visibility" IN (0, 1) OR "statuses"."account_id" = xxxxx AND "statuses"."id" IN (SELECT "mentions"."status_id" FROM "mentions" WHERE "mentions"."account_id" = yyyyy)) ORDER BY "statuses"."id" DESC LIMIT 20;
                                                                                QUERY PLAN                                                                                 
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Limit  (cost=65.89..143.78 rows=20 width=16) (actual time=0.241..16.070 rows=20 loops=1)
   ->  Index Scan Backward using index_statuses_on_account_id_id on statuses  (cost=65.89..173471.77 rows=44521 width=16) (actual time=0.239..16.056 rows=20 loops=1)
         Index Cond: (account_id = xxxxx)
         Filter: ((visibility = ANY ('{0,1}'::integer[])) OR (hashed SubPlan 1))
         Rows Removed by Filter: 13450
         SubPlan 1
           ->  Index Only Scan using index_mentions_on_account_id_and_status_id on mentions  (cost=0.42..65.32 rows=55 width=8) (actual time=0.013..0.050 rows=25 loops=1)
                 Index Cond: (account_id = yyyyy)
                 Heap Fetches: 22
 Planning time: 0.236 ms
 Execution time: 16.110 ms
(11 rows)

mastodon_production=>

mastodon_production=> explain analyze SELECT "statuses"."id", "statuses"."updated_at" FROM "statuses" WHERE ("statuses"."account_id" = xxxxx AND "statuses"."visibility" IN (0, 1, 2) OR "statuses"."account_id" = xxxxx AND "statuses"."id" IN (SELECT "mentions"."status_id" FROM "mentions" WHERE "mentions"."account_id" = yyyyy)) AND "statuses"."id" IN (SELECT DISTINCT "media_attachments"."status_id" FROM "media_attachments" WHERE "media_attachments"."account_id" = xxxxx AND ("media_attachments"."status_id" IS NOT NULL)) ORDER BY "statuses"."id" DESC LIMIT 40;
                                                                                      QUERY PLAN                                                                                       
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Limit  (cost=82661.86..82661.88 rows=5 width=16) (actual time=54.487..54.512 rows=40 loops=1)
   ->  Sort  (cost=82661.86..82661.88 rows=5 width=16) (actual time=54.486..54.497 rows=40 loops=1)
         Sort Key: statuses.id DESC
         Sort Method: top-N heapsort  Memory: 26kB
         ->  Hash Semi Join  (cost=6382.14..82661.81 rows=5 width=16) (actual time=11.654..53.982 rows=1316 loops=1)
               Hash Cond: (statuses.id = media_attachments.status_id)
               ->  Bitmap Heap Scan on statuses  (cost=1153.01..77370.87 rows=23525 width=16) (actual time=7.523..42.555 rows=26112 loops=1)
                     Recheck Cond: (account_id = xxxxx)
                     Filter: ((visibility = ANY ('{0,1,2}'::integer[])) OR (hashed SubPlan 1))
                     Rows Removed by Filter: 53
                     Heap Blocks: exact=21877
                     ->  Bitmap Index Scan on index_statuses_on_account_id_id  (cost=0.00..1081.68 rows=23633 width=0) (actual time=3.833..3.833 rows=26285 loops=1)
                           Index Cond: (account_id = xxxxx)
                     SubPlan 1
                       ->  Index Only Scan using index_mentions_on_account_id_and_status_id on mentions  (cost=0.42..65.32 rows=55 width=8) (actual time=0.013..0.025 rows=18 loops=1)
                             Index Cond: (account_id = yyyyy)
                             Heap Fetches: 12
               ->  Hash  (cost=5214.83..5214.83 rows=1144 width=8) (actual time=4.050..4.050 rows=1316 loops=1)
                     Buckets: 2048  Batches: 1  Memory Usage: 68kB
                     ->  HashAggregate  (cost=5191.95..5203.39 rows=1144 width=8) (actual time=3.324..3.675 rows=1316 loops=1)
                           Group Key: media_attachments.status_id
                           ->  Bitmap Heap Scan on media_attachments  (cost=52.98..5187.90 rows=1619 width=8) (actual time=0.420..2.616 rows=1827 loops=1)
                                 Recheck Cond: (account_id = xxxxx)
                                 Filter: (status_id IS NOT NULL)
                                 Heap Blocks: exact=1500
                                 ->  Bitmap Index Scan on index_media_attachments_on_account_id  (cost=0.00..52.57 rows=1620 width=0) (actual time=0.244..0.244 rows=1830 loops=1)
                                       Index Cond: (account_id = xxxxx)
 Planning time: 0.417 ms
 Execution time: 54.594 ms
(29 rows)

mastodon_production=>

after

mastodon_production=> explain analyze SELECT "statuses"."id", "statuses"."updated_at" FROM "statuses" WHERE ("statuses"."account_id" = xxxxx AND "statuses"."visibility" IN (0, 1) OR "statuses"."account_id" = xxxxx AND "statuses"."id" IN (SELECT "mentions"."status_id" FROM "mentions" WHERE "mentions"."account_id" = yyyyy)) ORDER BY "statuses"."id" DESC LIMIT 20;
                                                                                QUERY PLAN                                                                                 
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Limit  (cost=66.01..76.83 rows=20 width=16) (actual time=0.240..9.299 rows=20 loops=1)
   ->  Index Only Scan using index_20180106 on statuses  (cost=66.01..24149.02 rows=44518 width=16) (actual time=0.239..9.287 rows=20 loops=1)
         Index Cond: (account_id = xxxxx)
         Filter: ((visibility = ANY ('{0,1}'::integer[])) OR (hashed SubPlan 1))
         Rows Removed by Filter: 13471
         Heap Fetches: 8137
         SubPlan 1
           ->  Index Only Scan using index_mentions_on_account_id_and_status_id on mentions  (cost=0.42..65.32 rows=55 width=8) (actual time=0.012..0.051 rows=25 loops=1)
                 Index Cond: (account_id = yyyyy)
                 Heap Fetches: 22
 Planning time: 0.257 ms
 Execution time: 9.334 ms
(12 rows)

mastodon_production=>

mastodon_production=> explain analyze SELECT "statuses"."id", "statuses"."updated_at" FROM "statuses" WHERE ("statuses"."account_id" = xxxxx AND "statuses"."visibility" IN (0, 1, 2) OR "statuses"."account_id" = xxxxx AND "statuses"."id" IN (SELECT "mentions"."status_id" FROM "mentions" WHERE "mentions"."account_id" = yyyyy)) AND "statuses"."id" IN (SELECT DISTINCT "media_attachments"."status_id" FROM "media_attachments" WHERE "media_attachments"."account_id" = xxxxx AND ("media_attachments"."status_id" IS NOT NULL)) ORDER BY "statuses"."id" DESC LIMIT 40;
                                                                                   QUERY PLAN                                                                                    
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Limit  (cost=5338.95..18393.71 rows=5 width=16) (actual time=4.269..5.074 rows=40 loops=1)
   ->  Merge Semi Join  (cost=5338.95..18393.71 rows=5 width=16) (actual time=4.267..5.053 rows=40 loops=1)
         Merge Cond: (statuses.id = media_attachments.status_id)
         ->  Index Only Scan using index_20180106 on statuses  (cost=66.01..13056.18 rows=23525 width=16) (actual time=0.028..0.683 rows=505 loops=1)
               Index Cond: (account_id = xxxxx)
               Filter: ((visibility = ANY ('{0,1,2}'::integer[])) OR (hashed SubPlan 1))
               Rows Removed by Filter: 1
               Heap Fetches: 506
               SubPlan 1
                 ->  Index Only Scan using index_mentions_on_account_id_and_status_id on mentions  (cost=0.42..65.32 rows=55 width=8) (actual time=0.011..0.025 rows=18 loops=1)
                       Index Cond: (account_id = yyyyy)
                       Heap Fetches: 12
         ->  Sort  (cost=5272.94..5275.80 rows=1144 width=8) (actual time=4.226..4.237 rows=40 loops=1)
               Sort Key: media_attachments.status_id DESC
               Sort Method: quicksort  Memory: 110kB
               ->  HashAggregate  (cost=5191.95..5203.39 rows=1144 width=8) (actual time=3.416..3.771 rows=1316 loops=1)
                     Group Key: media_attachments.status_id
                     ->  Bitmap Heap Scan on media_attachments  (cost=52.98..5187.90 rows=1619 width=8) (actual time=0.411..2.738 rows=1827 loops=1)
                           Recheck Cond: (account_id = xxxxx)
                           Filter: (status_id IS NOT NULL)
                           Heap Blocks: exact=1500
                           ->  Bitmap Index Scan on index_media_attachments_on_account_id  (cost=0.00..52.57 rows=1620 width=0) (actual time=0.233..0.233 rows=1830 loops=1)
                                 Index Cond: (account_id = xxxxx)
 Planning time: 0.417 ms
 Execution time: 5.138 ms
(25 rows)

mastodon_production=>

[takayamaki]

The reason for specifying the name of index is that the automatically generated name was too long.

$ bundle ex rails db:migrate
   (0.3ms)  SELECT pg_try_advisory_lock(5881450594042647960)
   (1.0ms)  SELECT "schema_migrations"."version" FROM "schema_migrations" ORDER BY "schema_migrations"."version" ASC
Migrating to AddIndexOnStatusesForApiV1AccountsAccountIdStatuses (20180106000232)
== 20180106000232 AddIndexOnStatusesForApiV1AccountsAccountIdStatuses: migrating
-- add_index(:statuses, [:account_id, :id, :visibility, :updated_at], {:order=>{:id=>:desc}, :algorithm=>:concurrently})
   (0.3ms)  SELECT pg_advisory_unlock(5881450594042647960)
rails aborted!
StandardError: An error has occurred, all later migrations canceled:

Index name 'index_statuses_on_account_id_and_id_and_visibility_and_updated_at' on table 'statuses' is too long; the limit is 63 characters

[abcang]

Since this index is covered by index_statuses_20180106, it seems unnecessary.

[Gargron]

I am curious what is the purpose of updated_at in the index? Since updated_at is never in the WHERE clause I do not think that ever gets used?

[abcang]

The reason that updated_at is included in the index seems to be to perform Index Only Scan. Index Only Scan is much faster than reading a column. I think we can consider removing updated_at from the index if it is fast enough to read the column.