/synapse

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

Move to using TCP replication #2097

Merged
merged 8 commits into from Apr 5, 2017

Conversation

Reviewers

@richvdh richvdh

Assignees

@erikjohnston erikjohnston

Labels
None yet
Projects
None yet
Milestone
No milestone
2 participants
@erikjohnston @richvdh
@erikjohnston
Owner

erikjohnston commented Apr 4, 2017 • edited

This is a continuation of #2082

Sytest branch: matrix-org/sytest#351

erikjohnston added some commits Mar 27, 2017

@erikjohnston
Add basic replication client handler and factory
52bfa60
@erikjohnston
Change slave storage to use new replication interface 
As the TCP replication uses a slightly different API and streams than
the HTTP replication.

This breaks HTTP replication.
3a1f3f8
@erikjohnston
Update all the workers and master to use TCP replication

Some checks were not successful

1 failing and 7 successful checks
@matrixbot
Sytest Dendron (Merged PR) — Build finished.
Details
@matrixbot
Sytest Dendron (Commit) — Build #1883 origin/erikj/repl_tcp_client succeeded in 9 min 39 sec
Details
@matrixbot
Sytest Postgres (Commit) — Build #2710 origin/erikj/repl_tcp_client succeeded in 7 min 28 sec
Details
@matrixbot
Sytest Postgres (Merged PR) — Build finished.
Details
@matrixbot
Sytest SQLite (Commit) — Build #2782 origin/erikj/repl_tcp_client succeeded in 7 min 20 sec
Details
@matrixbot
Sytest SQLite (Merged PR) — Build finished.
Details
continuous-integration/travis-ci/pr — The Travis CI build passed
Details
continuous-integration/travis-ci/push — The Travis CI build passed
Details
36c28bc
@erikjohnston
Move where we ack federation
6ce6bbe
@erikjohnston
Shuffle and comment synchrotron presence
3376f16
@erikjohnston
Remove unused worker config option

Some checks were not successful

1 failing and 7 successful checks
@matrixbot
Sytest Dendron (Merged PR) — Build finished.
Details
@matrixbot
Sytest Dendron (Commit) — Build #1891 origin/erikj/repl_tcp_client succeeded in 9 min 7 sec
Details
@matrixbot
Sytest Postgres (Commit) — Build #2719 origin/erikj/repl_tcp_client succeeded in 8 min 22 sec
Details
@matrixbot
Sytest Postgres (Merged PR) — Build finished.
Details
@matrixbot
Sytest SQLite (Commit) — Build #2791 origin/erikj/repl_tcp_client succeeded in 6 min 17 sec
Details
@matrixbot
Sytest SQLite (Merged PR) — Build finished.
Details
continuous-integration/travis-ci/pr — The Travis CI build passed
Details
continuous-integration/travis-ci/push — The Travis CI build passed
Details
d160579

@erikjohnston erikjohnston requested a review from richvdh Apr 4, 2017

@erikjohnston erikjohnston assigned erikjohnston and richvdh and unassigned erikjohnston Apr 4, 2017

synapse/app/federation_sender.py
- pass
+ def __init__(self, db_conn, hs):
+ super(FederationSenderSlaveStore, self).__init__(db_conn, hs)
+ self.federation_out_pos_startup = self._get_federation_out_pos(db_conn)
@richvdh

richvdh Apr 4, 2017

Member

why do we need to store this rather than let the handler just call get_federation_out_pos?

@erikjohnston

erikjohnston Apr 4, 2017

Owner

As we always want to have the position available rather than having to jump through a deferred the first time round. This is the same as what we do with all the ID generators too, we fetch them out of the database during start up

@richvdh

richvdh Apr 4, 2017

Member

might be worth a comment then?

synapse/app/federation_sender.py
- yield self.store.update_federation_out_pos(
- "federation", latest_id
- )
+ self.update_token(token)
@richvdh

richvdh Apr 4, 2017

Member

preserve_fn please

synapse/app/pusher.py
+ super(PusherReplicationHandler, self).on_rdata(stream_name, token, rows)
+ self.poke_pushers(stream_name, token, rows)
+
+ def poke_pushers(self, stream_name, token, rows):
@richvdh

richvdh Apr 4, 2017

Member

might be easier to make this defer.inlineCallbacks and preserve_fn when calling it?

synapse/app/pusher.py
+ def start_pusher(self, user_id, app_id, pushkey):
+ key = "%s:%s" % (app_id, pushkey)
+ logger.info("Starting pusher %r / %r", user_id, key)
+ return self.pusher_pool._refresh_pusher(app_id, pushkey, user_id)
@richvdh

richvdh Apr 4, 2017

Member

this seems to be returning a deferred which is dropped. Should there be a preserve_fn somewhere?

@erikjohnston

erikjohnston Apr 4, 2017

Owner

(done)

@erikjohnston
Add the appropriate amount of preserve_fn

Some checks were not successful

1 failing and 7 successful checks
@matrixbot
Sytest Dendron (Merged PR) — Build finished.
Details
@matrixbot
Sytest Dendron (Commit) — Build #1895 origin/erikj/repl_tcp_client succeeded in 8 min 17 sec
Details
@matrixbot
Sytest Postgres (Commit) — Build #2723 origin/erikj/repl_tcp_client succeeded in 7 min 53 sec
Details
@matrixbot
Sytest Postgres (Merged PR) — Build finished.
Details
@matrixbot
Sytest SQLite (Commit) — Build #2795 origin/erikj/repl_tcp_client succeeded in 6 min 24 sec
Details
@matrixbot
Sytest SQLite (Merged PR) — Build finished.
Details
continuous-integration/travis-ci/pr — The Travis CI build passed
Details
continuous-integration/travis-ci/push — The Travis CI build passed
Details
ac66e11

@richvdh richvdh assigned erikjohnston and unassigned richvdh Apr 4, 2017

@erikjohnston
Add comment

Some checks were not successful

1 failing and 7 successful checks
@matrixbot
Sytest Dendron (Merged PR) — Build finished.
Details
@matrixbot
Sytest Dendron (Commit) — Build #1899 origin/erikj/repl_tcp_client succeeded in 8 min 51 sec
Details
@matrixbot
Sytest Postgres (Commit) — Build #2727 origin/erikj/repl_tcp_client succeeded in 7 min 53 sec
Details
@matrixbot
Sytest Postgres (Merged PR) — Build finished.
Details
@matrixbot
Sytest SQLite (Commit) — Build #2799 origin/erikj/repl_tcp_client succeeded in 5 min 42 sec
Details
@matrixbot
Sytest SQLite (Merged PR) — Build finished.
Details
continuous-integration/travis-ci/pr — The Travis CI build passed
Details
continuous-integration/travis-ci/push — The Travis CI build passed
Details
d1d5362

@erikjohnston erikjohnston assigned richvdh and unassigned erikjohnston Apr 4, 2017

@richvdh

richvdh approved these changes Apr 4, 2017
View changes

I guess it's plausible? Can't promise it won't break things.

@richvdh richvdh assigned erikjohnston and unassigned richvdh Apr 4, 2017

@erikjohnston erikjohnston merged commit a5c401b into develop Apr 5, 2017
7 of 8 checks passed

7 of 8 checks passed

Sytest Dendron (Merged PR) Build finished.
Details
Sytest Dendron (Commit) Build #1899 origin/erikj/repl_tcp_client succeeded in 8 min 51 sec
Details
Sytest Postgres (Commit) Build #2727 origin/erikj/repl_tcp_client succeeded in 7 min 53 sec
Details
Sytest Postgres (Merged PR) Build finished.
Details
Sytest SQLite (Commit) Build #2799 origin/erikj/repl_tcp_client succeeded in 5 min 42 sec
Details
Sytest SQLite (Merged PR) Build finished.
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
continuous-integration/travis-ci/push The Travis CI build passed
Details

psaavedra added a commit to psaavedra/synapse that referenced this pull request May 19, 2017

@psaavedra
Merge tag 'v0.21.0' into v0.21.0_no_federate_by_default 
Changes in synapse v0.21.0 (2017-05-18)
=======================================

No changes since v0.21.0-rc3

Changes in synapse v0.21.0-rc3 (2017-05-17)
===========================================

Features:

* Add per user rate-limiting overrides (PR #2208)
* Add config option to limit maximum number of events requested by ``/sync``
  and ``/messages`` (PR #2221) Thanks to @psaavedra!

Changes:

* Various small performance fixes (PR #2201, #2202, #2224, #2226, #2227, #2228,
  #2229)
* Update username availability checker API (PR #2209, #2213)
* When purging, don't de-delta state groups we're about to delete (PR #2214)
* Documentation to check synapse version (PR #2215) Thanks to @hamber-dick!
* Add an index to event_search to speed up purge history API (PR #2218)

Bug fixes:

* Fix API to allow clients to upload one-time-keys with new sigs (PR #2206)

Changes in synapse v0.21.0-rc2 (2017-05-08)
===========================================

Changes:

* Always mark remotes as up if we receive a signed request from them (PR #2190)

Bug fixes:

* Fix bug where users got pushed for rooms they had muted (PR #2200)

Changes in synapse v0.21.0-rc1 (2017-05-08)
===========================================

Features:

* Add username availability checker API (PR #2183)
* Add read marker API (PR #2120)

Changes:

* Enable guest access for the 3pl/3pid APIs (PR #1986)
* Add setting to support TURN for guests (PR #2011)
* Various performance improvements (PR #2075, #2076, #2080, #2083, #2108,
  #2158, #2176, #2185)
* Make synctl a bit more user friendly (PR #2078, #2127) Thanks @APwhitehat!
* Replace HTTP replication with TCP replication (PR #2082, #2097, #2098,
  #2099, #2103, #2014, #2016, #2115, #2116, #2117)
* Support authenticated SMTP (PR #2102) Thanks @DanielDent!
* Add a counter metric for successfully-sent transactions (PR #2121)
* Propagate errors sensibly from proxied IS requests (PR #2147)
* Add more granular event send metrics (PR #2178)

Bug fixes:

* Fix nuke-room script to work with current schema (PR #1927) Thanks
  @zuckschwerdt!
* Fix db port script to not assume postgres tables are in the public schema
  (PR #2024) Thanks @jerrykan!
* Fix getting latest device IP for user with no devices (PR #2118)
* Fix rejection of invites to unreachable servers (PR #2145)
* Fix code for reporting old verify keys in synapse (PR #2156)
* Fix invite state to always include all events (PR #2163)
* Fix bug where synapse would always fetch state for any missing event (PR #2170)
* Fix a leak with timed out HTTP connections (PR #2180)
* Fix bug where we didn't time out HTTP requests to ASes  (PR #2192)

Docs:

* Clarify doc for SQLite to PostgreSQL port (PR #1961) Thanks @benhylau!
* Fix typo in synctl help (PR #2107) Thanks @HarHarLinks!
* ``web_client_location`` documentation fix (PR #2131) Thanks @matthewjwolff!
* Update README.rst with FreeBSD changes (PR #2132) Thanks @feld!
* Clarify setting up metrics (PR #2149) Thanks @encks!
badd30b

@erikjohnston erikjohnston deleted the erikj/repl_tcp_client branch Oct 26, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment