0.99.5.1 hoards CPU+RAM and gets OOM in ever few minutes

[olmari]

Description

Synapse 0.99.5.1 from deb repo consumes all CPU and RAM and gets OOM every few minutes, accompanied with synapse user postgre thread eatin CPU too

Steps to reproduce

• start synapse
• watch world burn

I don't know is these warnings about db connection related, but other than those log doesn't seem to reveal anything...

synapse.storage._base - 467 - WARNING - - Starting db txn 'set_destination_retry_timings' from sentinel context
May 28 00:01:49 matrix matrix-synapse[7962]: 2019-05-28 00:01:49,500 - synapse.storage._base - 506 - WARNING - - Starting db connection from sentinel context: metrics will be lost
May 28 00:01:49 matrix matrix-synapse[7962]: 2019-05-28 00:01:49,501 - synapse.util.metrics - 111 - WARNING - - Context has unexpectedly changed from 'federation_transaction_transmission_loop@7ffa31d892e8' to 'sentinel'. ('_send_new_transaction')
May 28 00:01:49 matrix matrix-synapse[7962]: 2019-05-28 00:01:49,505 - synapse.federation.sender.per_destination_queue - 310 - WARNING - - TX [matrix.hum.is] Failed to send transaction: Failed to send request: TimeoutError: User timeout caused connection failure.
May 28 00:01:49 matrix matrix-synapse[7962]: 2019-05-28 00:01:49,505 - synapse.util.logcontext - 277 - WARNING - - Expected logging context federation_transaction_transmission_loop@7ffa31d892e8 has been lost
May 28 00:01:50 matrix matrix-synapse[7962]: 2019-05-28 00:01:50,279 - synapse.util.metrics - 111 - WARNING - - Context has unexpectedly changed from 'federation_transaction_transmission_loop@7ffa14fcc668' to 'sentinel'. ('outbound_request')
May 28 00:01:50 matrix matrix-synapse[7962]: 2019-05-28 00:01:50,279 - synapse.http.matrixfederationclient - 500 - WARNING - - {PUT-O-5508} [mental.af] Request failed: PUT matrix://mental.af/_matrix/federation/v1/send/1558989794957: HttpResponseException("502: b'Bad Gateway'",)

Version information

matrix-synapse-py3 | 0.99.5.1+bionic1 | https://packages.matrix.org/debian bionic/main amd64 Packages

Ubuntu 18.04.2 LTS (GNU/Linux 4.15.0-50-generic x86_64)

• Homeserver: hacklab.fi

If not matrix.org:

• Version: 0.9.55.1

• Install method: package manager

• Platform:
  Ubuntu 18.04.2 LTS (GNU/Linux 4.15.0-50-generic x86_64), Digital ocean VM, 2 cpu, 4g ram, postgre database

[aaronraimist]

Lots of people have been reporting problems :(

Have you checked #1760?

[olmari]

Whoa, I think "solution" from #1760 (comment) solved this, at least initially it looks that way. I'll close this ticket in some moments later if situation doesn't come back.

(Still not good thing to happend)

[aaronraimist]

I don't think you should close it until someone from the Synapse team takes a look though. I've helped several people in #synapse:matrix.org who all just started having this problem with 0.99.5.1 so I think it may have gotten worse.

[turt2live]

tbh I'm not sure if it's a problem which got worse, or just increased due to the marketing push for people to upgrade Synapse.

[ejpcmac]

I’m hitting a similar issue on Synapse 0.99.3 for a few days now. If I start it, it works OK for a few minutes, then stops to respond (my client says it is disconnected). Afterwards I can see the memory growing and my swap starting to be full.

In the mean time, I start to get (FreeBSD) Kernel messages like:

May 26 21:08:41 ejpcmac kernel: sonewconn: pcb 0xfffff8000f6d8570: Listen queue overflow: 76 already in queue awaiting acceptance (124 occurrences)

P.S. : I’ve checked #1760, and the greatest number I get is 6, for a matrix.org room, so it should be OK.

[erikjohnston]

I think @olmari problems are caused by #5269, so closing this as a duplicate.

@ejpcmac: That sounds like a separate problem. Can you open a new issue please?