Accepting an invite to a room on another HS returns a 404

[ewingrj]

Description

In the matrix-appservice-slackbot, the bot will auto join any rooms it has been invited to. If a user from another HS invites the bot to a room that hasn't been synced to the bots HS, synapse will throw an error 2018-01-18 01:26: [-] POST https://matrix.giveth.io/_matrix/client/r0/join/!TAeakIVVBKXyGipXoY%3Amatrix.org AS) HTTP 404 Error: {"errcode":"M_UNKNOWN","error":"No row found"}.

Version information

• Homeserver: Was this issue identified on matrix.org or another homeserver?

If not matrix.org: giveth.io

• Version: 26
• Install method: package manager
• Platform: VPS w/ debian 9

[richvdh]

is this specific to the matrix-appservice-slackbot?

Are you sure the bot has a pending invitation to the room? This sounds like synapse failing to find a remote server to help with the join request, but it should check if there is pending invitation and use that: https://github.com/matrix-org/synapse/blob/master/synapse/handlers/room_member.py#L287

We'll probably need to see the logs to diagnose further

[ewingrj]

I haven't testing if this is specific to the matrix-appservice-slack bridge or not.

yes, I'm sure about the pending invitation.

here's the invite msg:

{
  "origin_server_ts": 1516886875874,
  "sender": "@perissology:matrix.org",
  "event_id": "$15168868758429051VcQex:matrix.org",
  "unsigned": {
    "age": 772
  },
  "state_key": "@slackbot:giveth.io",
  "content": {
    "membership": "invite"
  },
  "membership": "invite",
  "type": "m.room.member",
  "room_id": "!MPjsKDPOIfQrVFcgUU:matrix.org"
}

and the logs from the giveth.io server from right before the invite to after the failure:
https://gist.github.com/perissology/e18b126b7e1348210ae7e154e0cb81d5

[ara4n]

This feels like a general bug with federation interacting with application services - although I'm surprised @turt2live hasn't encountered it... @turt2live, any idea what might be going on here?

[ara4n]

@perissology thanks for the excellent bug report & details btw

[turt2live]

@ara4n I actually added a retry loop to almost everything I've deployed to combat this. The common failure is the room comes down the /sync stream, but the homeserver isn't aware of the room yet. It can sometimes take minutes for the room the appear.

[turt2live]

Well, this is fun. I just ran into this on my test server. The logs are very unexciting:

2018-02-21 19:07:55,983 - synapse.access.http.8008 - 59 - INFO - POST-247- 172.16.0.1 - 8008 - Received request: POST /_matrix/client/r0/join/!nrZKEcuLPAvzNPfAsH%3At2l.io?access_token=<redacted>
2018-02-21 19:07:55,984 - synapse.util.async - 230 - INFO - POST-247- Acquired uncontended linearizer lock 'member' for key (u'!nrZKEcuLPAvzNPfAsH:t2l.io',)
2018-02-21 19:07:55,984 - synapse.util.async - 230 - INFO - POST-247- Acquired uncontended linearizer lock 'state_resolve_lock' for key frozenset([])
2018-02-21 19:07:55,985 - synapse.state - 455 - INFO - POST-247- Resolving state for !nrZKEcuLPAvzNPfAsH:t2l.io with 0 groups
2018-02-21 19:07:55,985 - synapse.util.async - 237 - INFO - POST-247- Releasing linearizer lock 'state_resolve_lock' for key frozenset([])
2018-02-21 19:07:55,988 - synapse.util.async - 237 - INFO - POST-247- Releasing linearizer lock 'member' for key (u'!nrZKEcuLPAvzNPfAsH:t2l.io',)
2018-02-21 19:07:55,989 - synapse.http.server - 162 - INFO - POST-247- <XForwardedForRequest at 0x7fdb665ea200 method=POST uri=/_matrix/client/r0/join/!nrZKEcuLPAvzNPfAsH%3At2l.io?access_token=<redacted> clientproto=HTTP/1.0 site=8008> SynapseError: 404 - No row found
2018-02-21 19:07:55,989 - synapse.access.http.8008 - 93 - INFO - POST-247- 172.16.0.1 - 8008 - {@_discord_bot:dev.t2bot.io} Processed request: 5ms (0ms, 0ms) (0ms/1ms/1) 46B 404 "POST /_matrix/client/r0/join/!nrZKEcuLPAvzNPfAsH%3At2l.io?access_token=<redacted> HTTP/1.0" "None"

[danilaplee]

this is the bug, i was talking about a month ago, it's really wide, i have a server completely locked out of federation due to this issue

2018-05-10 13:52:50,153 - synapse.util.async - 296 - INFO - PUT-5442287- Releasing limiter lock for key u'!KjedgbiACDMhyAYZzU:matrix-01<redacted>'
2018-05-10 13:52:50,155 - synapse.util.async - 230 - INFO - PUT-5442287- Acquired uncontended linearizer lock 'rules_for_room' for key ()
2018-05-10 13:52:50,156 - synapse.util.async - 237 - INFO - PUT-5442212- Releasing linearizer lock 'member' for key (u'!GCmQbvPeGOnqXNnSsL:matrix-01<redacted>',)
2018-05-10 13:52:50,156 - synapse.handlers.profile - 250 - WARNING - PUT-5442212- Failed to update join event for room !GCmQbvPeGOnqXNnSsL:matrix-01<redacted> - 4484590
2018-05-10 13:52:50,156 - synapse.util.async - 230 - INFO - PUT-5442212- Acquired uncontended linearizer lock 'member' for key (u'!WsJLlVBOlInCGskuzt:matrix-10<redacted>',)
2018-05-10 13:52:50,157 - twisted - 131 - CRITICAL - - Unhandled error in Deferred:
2018-05-10 13:52:50,157 - twisted - 131 - CRITICAL - - KeyError: 4484590

[theotheroracle]

i'm pretty sure i'm experiencing this issue, if i make a chat on the librem.one HS and invite someone from matrix, it gives the M_UNKOWN error.

[richvdh]

@SaturnSoftware can you share the synapse logs for the failing request please

https://github.com/matrix-org/synapse/wiki#how-can-i-find-the-lines-corresponding-to-a-given-http-request-in-my-homeserver-log

[theotheroracle]

would that be in the browser console? or where would i find the synapse

[squahtx]

Is this still happening on recent releases of Synapse?

[turt2live]

Yes. Constantly.