/synapse

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

Wrap connections in an N minute timeout to ensure they get reaped correctly #1725

Merged
merged 6 commits into from Dec 29, 2016

Conversation

Reviewers

@uhoreg uhoreg

@ara4n ara4n

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
4 participants
@erikjohnston @uhoreg @ara4n @NegativeMjark
@erikjohnston
Owner

erikjohnston commented Dec 29, 2016

No description provided.

erikjohnston added some commits Dec 16, 2016

@erikjohnston
Merge branch 'release-v0.18.5' of github.com:matrix-org/synapse

All checks have failed

4 errored checks
@matrixbot
Sytest Dendron (Commit) — Build #1261 origin/master failed in 37 sec
Details
@matrixbot
Sytest Postgres (Commit) — Build #2103 origin/master failed in 19 sec
Details
@matrixbot
Sytest SQLite (Commit) — Build #2145 origin/master failed in 25 sec
Details
continuous-integration/travis-ci/push — The Travis CI build could not complete due to an error
Details
f5a4001
@erikjohnston
Wrap connections in an N minute timeout to ensure they get reaped cor… 
…rectly

Some checks were not successful

2 errored, 3 failing, and 2 successful checks
@matrixbot
SynapseSytestDendronPR — Build #838 detached failed in 1 min 33 sec
Details
@matrixbot
SynapseSytestSQLitePR — Build #1360 detached failed in 6 min 59 sec
Details
@matrixbot
Sytest Dendron (Merged PR) — Build finished.
Details
@matrixbot
Sytest SQLite (Merged PR) — Build finished.
Details
continuous-integration/travis-ci/push — The Travis CI build failed
Details
@matrixbot
SynapseSytestPostgresPR — Build #1357 detached succeeded in 8 min 14 sec
Details
@matrixbot
Sytest Postgres (Merged PR) — Build finished.
Details
5b6672c
synapse/http/endpoint.py
+
+ # Time this connection out if we haven't send a request in the last
+ # N minutes
+ reactor.callLater(3 * 60, self._time_things_out_maybe)
@ara4n

ara4n Dec 29, 2016

Owner

why is it 3 mins here, but 2 mins in the _time_things_out_maybe above?

synapse/http/endpoint.py
+ return _WrappingEndointFac(transport_endpoint(reactor, domain, port, **endpoint_kw_args))
+
+
+class _WrappingEndointFac(object):
@ara4n

ara4n Dec 29, 2016

Owner

s/Endoint/Endpoint/ probably ;)

@erikjohnston
Clean up

All checks have passed

6 successful checks
@matrixbot
SynapseSytestDendronPR — Build #839 detached succeeded in 13 min
Details
@matrixbot
SynapseSytestSQLitePR — Build #1361 detached succeeded in 8 min 1 sec
Details
@matrixbot
Sytest Dendron (Merged PR) — Build finished.
Details
@matrixbot
Sytest Postgres (Merged PR) — Build finished.
Details
@matrixbot
Sytest SQLite (Merged PR) — Build finished.
Details
continuous-integration/travis-ci/push — The Travis CI build passed
Details
b7336ff
synapse/http/matrixfederationclient.py
@@ -61,6 +61,11 @@
MAX_SHORT_RETRIES = 3
+def test(conn):
@ara4n

ara4n Dec 29, 2016

Owner

hm, is this meant to be here?

@erikjohnston
Spelling and comments

Some checks were not successful

1 errored and 9 successful checks
@matrixbot
Sytest Dendron (Commit) — Build #1274 origin/erikj/timeout_conn failed in 1 min 15 sec
Details
@matrixbot
SynapseSytestDendronPR — Build #841 detached succeeded in 14 min
Details
@matrixbot
SynapseSytestPostgresPR — Build #1358 detached succeeded in 7 min 57 sec
Details
@matrixbot
SynapseSytestSQLitePR — Build #1363 detached succeeded in 7 min 23 sec
Details
@matrixbot
Sytest Dendron (Merged PR) — Build finished.
Details
@matrixbot
Sytest Postgres (Commit) — Build #2117 origin/erikj/timeout_conn succeeded in 9 min 5 sec
Details
@matrixbot
Sytest Postgres (Merged PR) — Build finished.
Details
@matrixbot
Sytest SQLite (Commit) — Build #2159 origin/erikj/timeout_conn succeeded in 7 min 53 sec
Details
@matrixbot
Sytest SQLite (Merged PR) — Build finished.
Details
continuous-integration/travis-ci/push — The Travis CI build passed
Details
68030fd
synapse/http/endpoint.py
+ reactor.callLater(3 * 60, self._time_things_out_maybe)
+ return res
+
+ d.addCallback(update_request_time)
@ara4n

ara4n Dec 29, 2016

Owner

why do we do this by both a callback and a reactor.callLater()?

@erikjohnston

erikjohnston Dec 29, 2016

Owner

We schedule a timeout in 3 minutes both before we send the request, and after we received the response.

@erikjohnston
Respect long_retries param and default to off

Some checks were not successful

1 errored and 9 successful checks
@matrixbot
SynapseSytestDendronPR — Build #842 detached failed in 1 min 33 sec
Details
@matrixbot
SynapseSytestPostgresPR — Build #1359 detached succeeded in 8 min 58 sec
Details
@matrixbot
SynapseSytestSQLitePR — Build #1364 detached succeeded in 7 min 49 sec
Details
@matrixbot
Sytest Dendron (Commit) — Build #1275 origin/erikj/timeout_conn succeeded in 13 min
Details
@matrixbot
Sytest Dendron (Merged PR) — Build finished.
Details
@matrixbot
Sytest Postgres (Commit) — Build #2118 origin/erikj/timeout_conn succeeded in 9 min 29 sec
Details
@matrixbot
Sytest Postgres (Merged PR) — Build finished.
Details
@matrixbot
Sytest SQLite (Commit) — Build #2160 origin/erikj/timeout_conn succeeded in 7 min 18 sec
Details
@matrixbot
Sytest SQLite (Merged PR) — Build finished.
Details
continuous-integration/travis-ci/push — The Travis CI build passed
Details
b4bc6fe
synapse/http/endpoint.py
+
+
+class _WrappedConnection(object):
+ """Wraps a connection and calls abort on it if it hasn't seen any actio
@uhoreg

uhoreg Dec 29, 2016

missing "n" in "action". Also, the comment says 5 minutes, but the code says 3 minutes?

@NegativeMjark
Manually abort the underlying TLS connection. 
The abort() method calls loseConnection() which tries to shutdown the
TLS connection cleanly. We now call abortConnection() directly which
should promptly close both the TLS connection and the underlying TCP
connection.

I also added some TODO markers to consider cancelling the old previous
timeout rather than checking time.time(). But given how urgently we want
to get this code released I'd rather leave the existing code with the
duplicate timeouts and the time.time() check.

Some checks were not successful

1 errored, 1 failing, and 5 successful checks
@matrixbot
Sytest Dendron (Commit) — Build #1276 origin/erikj/timeout_conn failed in 13 min
Details
@matrixbot
Sytest Dendron (Merged PR) — Build finished.
Details
@matrixbot
Sytest Postgres (Commit) — Build #2119 origin/erikj/timeout_conn succeeded in 9 min 25 sec
Details
@matrixbot
Sytest Postgres (Merged PR) — Build finished.
Details
@matrixbot
Sytest SQLite (Commit) — Build #2161 origin/erikj/timeout_conn succeeded in 8 min 22 sec
Details
@matrixbot
Sytest SQLite (Merged PR) — Build finished.
Details
continuous-integration/travis-ci/push — The Travis CI build passed
Details
97ffc56

@NegativeMjark NegativeMjark changed the base branch from develop to release-v0.18.6 Dec 29, 2016

@NegativeMjark NegativeMjark merged commit 828c585 into release-v0.18.6 Dec 29, 2016
5 of 7 checks passed

5 of 7 checks passed

Sytest Dendron (Merged PR) Build finished.
Details
Sytest Dendron (Commit) Build #1276 origin/erikj/timeout_conn in progress...
Details
Sytest Postgres (Commit) Build #2119 origin/erikj/timeout_conn succeeded in 9 min 25 sec
Details
Sytest Postgres (Merged PR) Build finished.
Details
Sytest SQLite (Commit) Build #2161 origin/erikj/timeout_conn succeeded in 8 min 22 sec
Details
Sytest SQLite (Merged PR) Build finished.
Details
continuous-integration/travis-ci/push The Travis CI build passed
Details

This was referenced Dec 30, 2016

Closed
Closed
Closed
Open

@erikjohnston erikjohnston deleted the erikj/timeout_conn branch Mar 29, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment