Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

Be more agressive about purging old room event_push_actions #1873

Merged
merged 1 commit into from Feb 14, 2017

Conversation

Projects
None yet
3 participants
Owner

erikjohnston commented Feb 1, 2017

There's no reason to keep a months worth of notifications after they've been read. While the pushers do use the table, they shouldn't be needing to send anything that has already been read.

We keep a months worth of highlights so that the notification API still works.

Owner

erikjohnston commented Feb 1, 2017

Actually, the API does allow you to paginate on plain notifications, though I don't think anybody uses it.

Owner

ara4n commented Feb 1, 2017

Owner

erikjohnston commented Feb 2, 2017

@ara4n They are just highlights I thought?

@@ -475,7 +479,8 @@ def _remove_old_push_actions_before_txn(self, txn, room_id, user_id,
txn.execute(
"DELETE FROM event_push_actions "
" WHERE user_id = ? AND room_id = ? AND "
- " topological_ordering < ? AND stream_ordering < ?",
+ " topological_ordering < ?"
+ " AND ((stream_ordering < ? AND highlight = 1) or highlight = 0)",
@NegativeMjark

NegativeMjark Feb 2, 2017

Contributor

Capitalise "OR" please.

Contributor

NegativeMjark commented Feb 2, 2017

Apart from the capitalisation LGTM.

The query plans before and after are:

 Delete on event_push_actions  (cost=0.70..8.72 rows=1 width=6)
   ->  Index Scan using event_push_actions_rm_tokens on event_push_actions  (cost=0.70..8.72 rows=1 width=6)
         Index Cond: ((user_id = '?'::text) AND (room_id = '?'::text) AND (topological_ordering < ?) AND (stream_ordering < ?))
 Delete on event_push_actions  (cost=0.70..8.73 rows=1 width=6)
   ->  Index Scan using event_push_actions_rm_tokens on event_push_actions  (cost=0.70..8.73 rows=1 width=6)
         Index Cond: ((user_id = '?'::text) AND (room_id = '?'::text) AND (topological_ordering < ?))
         Filter: (((stream_ordering < ?) AND (highlight = 1)) OR (highlight = 0))

I don't think it matters that we're losing the stream_ordering from the index cond, since it'll need to scan for topological_ordering < ? in either case, and this should help performance because more of the stuff matching topological_ordering < ? will be deleted.

@erikjohnston erikjohnston merged commit 795f8e3 into develop Feb 14, 2017

8 checks passed

Sytest Dendron (Commit) Build #1468 origin/erikj/delete_push_actions succeeded in 11 min
Details
Sytest Dendron (Merged PR) Build finished.
Details
Sytest Postgres (Commit) Build #2288 origin/erikj/delete_push_actions succeeded in 6 min 22 sec
Details
Sytest Postgres (Merged PR) Build finished.
Details
Sytest SQLite (Commit) Build #2356 origin/erikj/delete_push_actions succeeded in 6 min 1 sec
Details
Sytest SQLite (Merged PR) Build finished.
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
continuous-integration/travis-ci/push The Travis CI build passed
Details

erikjohnston added a commit that referenced this pull request Mar 13, 2017

Merge tag 'v0.19.3-rc1' into release-v0.19.3
Changes in synapse v0.19.3-rc1 (2017-03-08)
===========================================

Features:

* Add some administration functionalities. Thanks to morteza-araby! (PR #1784)

Changes:

* Reduce database table sizes (PR #1873, #1916, #1923, #1963)
* Update contrib/ to not use syutil. Thanks to andrewshadura! (PR #1907)
* Don't fetch current state when sending an event in common case (PR #1955)

Bug fixes:

* Fix synapse_port_db failure. Thanks to Pneumaticat! (PR #1904)
* Fix caching to not cache error responses (PR #1913)
* Fix APIs to make kick & ban reasons work (PR #1917)
* Fix bugs in the /keys/changes api (PR #1921)
* Fix bug where users couldn't forget rooms they were banned from (PR #1922)
* Fix issue with long language values in pushers API (PR #1925)
* Fix a race in transaction queue (PR #1930)
* Fix dynamic thumbnailing to preserve aspect ratio. Thanks to jkolo! (PR
  #1945)
* Fix device list update to not constantly resync (PR #1964)
* Fix potential for huge memory usage when getting device that have
  changed (PR #1969)

@erikjohnston erikjohnston deleted the erikj/delete_push_actions branch Mar 29, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment