Cannot leave unjoined room in “Hotel California”.

[MacLemon]

This one is likely related to #2395.

Description

I cannot successfully join or leave a #matrix-dev:matrix.org resulting in a room situation like Hotel California where I can neither join, nor leave a room.

Steps to reproduce

• Run my own federated homeserver.
• On Riot/Mac or Riot/iOS (Which is what I use) try to join #matrix-dev:matrix.org.
• This will fail since the server crashes, locks up or otherwise just stops to respond requireing the synapsed to be killed hard. (meaning kill -9, otherwise you cannot terminate the process anymore)
• Upon restarting synapsed, it starts to sync #matrix-dev:matrix.org.
• Any client thinks it never joined the room.
• Leaving the room is not possible, since Riot thinks it isn't joined.
• Joining the room isn't possible either, since synapsed will lock up and stop responding again never resulting in a successfuly join.
• Server is now heavily loaded trying to sync #matrix-dev:matrix.org for no clients and regularly crashes.
• Should, after a gazillion tries, the client be actually “successful” in joining the room before synapsed crashes it will result in having an “Empty Room” with no members and no messages.
• Push messages for postings in #matrix-dev:matrix.org are still sent but cannot be muted on the client either, since I can't get into room settings for a not-joined room.

Describe how what happens differs from what you expected.
Joining a room should be an atomic operation. You are either joined or you're not. Not somthing in between of both at the same time yet none of either as well.

Version information

• Homeserver: Was this issue identified on matrix.org or another homeserver?
  On my homeserver which federates with matrix.org.

If not matrix.org:

• Version: What version of Synapse is running?
  Server: Synapse/0.22.1

• Install method: package manager/git clone/pip
  pkg install py27-matrix-synapse

• Platform:
  FreeBSD

$ freebsd-version
11.1-RELEASE-p1

Python:

$ python -V
Python 2.7.13

Synapse is running in its onw FreeBSD Jail on metal on x86_64/amd64 arch.

Notes:
Doesn't seem to be the #1760 Forward extremities problem.

$ sqlite3 homeserver.db
SQLite version 3.20.0 2017-08-01 13:24:15
Enter ".help" for usage hints.
sqlite> select room_id, count(*) c from event_forward_extremities group by room_id order by c desc limit 20 ;
!*******************:darkfasel.net|2
!*******************:chat.weho.st|2
!*******************:maclemon.at|1
!*******************:matrix.org|1
!*******************:maclemon.at|1
!*******************:maclemon.at|1
!*******************:matrix.org|1
!*******************:matrix.org|1
!*******************:matrix.org|1
!*******************:matrix.org|1
!*******************:maclemon.at|1
!*******************:maclemon.at|1
!*******************:darkfasel.net|1
!*******************:matrix.org|1
!*******************:maclemon.at|1
!*******************:maclemon.at|1
!*******************:matrix.org|1
!*******************:matrix.org|1
!*******************:maclemon.at|1
!*******************:maclemon.at|1

A complete homeserver.log at INFO level is attached from launching synapsed until it stops reacting to requests and has to be killed. During that time, no clients were connected to the server (as far as I can tell.)

from_launch_to_crash userinforedacted.txt

[erikjohnston]

(discussions are happening at: #2395 ftr)

[richvdh]

so can we close this as a dup of #2395?

[richvdh]

I'm assuming so; we can reopen if it's still a problem