This repository has been archived by the owner on Apr 26, 2024. It is now read-only.
Reduce user_ips bloat during dedupe background update #4626
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The background update to remove duplicate rows naively deleted and reinserted the duplicates. For large tables with a large number of duplicates this causes a lot of bloat (with postgres), as the inserted rows are appended to the table, since deleted rows will not be overwritten until a VACUUM has happened. This should hopefully also help ensure that the query in the last batch uses the correct index, as inserting a large number of new rows without analyzing will upset the query planner.
erikjohnston
force-pushed
the
erikj/fixup_user_ips_dedupe
branch
from
ed0e9b0
to
218cc07
Compare
richvdh
reviewed
Feb 12, 2019
| @@ -0,0 +1 @@ | |||
| Improve 'user_ips' table deduplication background update | |||
richvdh
approved these changes
Feb 12, 2019
richvdh
added a commit
that referenced
this pull request
Feb 14, 2019
Synapse 0.99.1 (2019-02-14) =========================== Features -------- - Include m.room.encryption on invites by default ([\#3902](#3902)) - Federation OpenID listener resource can now be activated even if federation is disabled ([\#4420](#4420)) - Synapse's ACME support will now correctly reprovision a certificate that approaches its expiry while Synapse is running. ([\#4522](#4522)) - Add ability to update backup versions ([\#4580](#4580)) - Allow the "unavailable" presence status for /sync. This change makes Synapse compliant with r0.4.0 of the Client-Server specification. ([\#4592](#4592)) - There is no longer any need to specify `no_tls`: it is inferred from the absence of TLS listeners ([\#4613](#4613), [\#4615](#4615), [\#4617](#4617), [\#4636](#4636)) - The default configuration no longer requires TLS certificates. ([\#4614](#4614)) Bugfixes -------- - Copy over room federation ability on room upgrade. ([\#4530](#4530)) - Fix noisy "twisted.internet.task.TaskStopped" errors in logs ([\#4546](#4546)) - Synapse is now tolerant of the `tls_fingerprints` option being None or not specified. ([\#4589](#4589)) - Fix 'no unique or exclusion constraint' error ([\#4591](#4591)) - Transfer Server ACLs on room upgrade. ([\#4608](#4608)) - Fix failure to start when not TLS certificate was given even if TLS was disabled. ([\#4618](#4618)) - Fix self-signed cert notice from generate-config. ([\#4625](#4625)) - Fix performance of `user_ips` table deduplication background update ([\#4626](#4626), [\#4627](#4627)) Internal Changes ---------------- - Change the user directory state query to use a filtered call to the db instead of a generic one. ([\#4462](#4462)) - Reject federation transactions if they include more than 50 PDUs or 100 EDUs. ([\#4513](#4513)) - Reduce duplication of ``synapse.app`` code. ([\#4567](#4567)) - Fix docker upload job to push -py2 images. ([\#4576](#4576)) - Add port configuration information to ACME instructions. ([\#4578](#4578)) - Update MSC1711 FAQ to calrify .well-known usage ([\#4584](#4584)) - Clean up default listener configuration ([\#4586](#4586)) - Clarifications for reverse proxy docs ([\#4607](#4607)) - Move ClientTLSOptionsFactory init out of `refresh_certificates` ([\#4611](#4611)) - Fail cleanly if listener config lacks a 'port' ([\#4616](#4616)) - Remove redundant entries from docker config ([\#4619](#4619)) - README updates ([\#4621](#4621))
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The background update to remove duplicate rows naively deleted and
reinserted the duplicates. For large tables with a large number of
duplicates this causes a lot of bloat (with postgres), as the inserted
rows are appended to the table, since deleted rows will not be
overwritten until a VACUUM has happened.
This should hopefully also help ensure that the query in the last batch
uses the correct index, as inserting a large number of new rows without
analyzing will upset the query planner.