Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

Cannot write to a room(or leave it) with MatrixError: [403] No create event in auth events #16676

Closed
VPaulV opened this issue Nov 22, 2023 · 6 comments
Closed

Cannot write to a room(or leave it) with MatrixError: [403] No create event in auth events #16676

VPaulV opened this issue Nov 22, 2023 · 6 comments
Labels
A-Database DB stuff like queries, migrations, new/remove columns, indexes, unexpected entries in the db A-Retention Retention rules to delete messages after a certain amount of time O-Uncommon Most users are unlikely to come across this or unexpected workflow S-Minor Blocks non-critical functionality, workarounds exist. T-Defect Bugs, crashes, hangs, security vulnerabilities, or other reported issues. X-Needs-Info This issue is blocked awaiting information from the reporter

Comments

@VPaulV
Copy link

VPaulV commented Nov 22, 2023 •
edited
Loading

Description

Hi Guys,

I have a simple docker based setup with workers. Everything worked fine for a year, but recently got an issue with the Synapse Admin room.

When I try to write something there or leave the channel I get the following error:

matrix-nginx            | 172.18.0.2 - - [22/Nov/2023:13:17:41 +0000] "OPTIONS /_matrix/client/v3/rooms/!ehXvUhWNASUkSLvAGP%3Amatrix.org/send/m.room.message/m1700659050405.95 HTTP/1.1" 204 0 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Element/1.11.49 Chrome/118.0.5993.120 Electron/27.0.4 Safari/537.36" "79.248.1.168"
matrix-worker-5         | 2023-11-22 13:17:41,896 - synapse.http.server - 124 - INFO - PUT-10309 - <XForwardedForRequest at 0x7faaf2f6d850 method='PUT' uri='/_matrix/client/v3/rooms/!ehXvUhWNASUkSLvAGP%3Amatrix.org/send/m.room.message/m1700659050405.95' clientproto='HTTP/1.0' site='unix'> SynapseError: 403 - No create event in auth events
matrix-worker-5         | 2023-11-22 13:17:41,897 - synapse.access.http.unix - 465 - INFO - PUT-10309 - 172.18.0.2 - unix - {@user:redacted} Processed request: 0.022sec/0.000sec (0.006sec, 0.001sec) (0.002sec/0.011sec/9) 66B 403 "PUT /_matrix/client/v3/rooms/!ehXvUhWNASUkSLvAGP%3Amatrix.org/send/m.room.message/m1700659050405.95 HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Element/1.11.49 Chrome/118.0.5993.120 Electron/27.0.4 Safari/537.36" [0 dbevts]
matrix-nginx            | 172.18.0.2 - - [22/Nov/2023:13:17:41 +0000] "PUT /_matrix/client/v3/rooms/!ehXvUhWNASUkSLvAGP%3Amatrix.org/send/m.room.message/m1700659050405.95 HTTP/1.1" 403 77 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Element/1.11.49 Chrome/118.0.5993.120 Electron/27.0.4 Safari/537.36" "79.248.1.168"

I tried to delete the room with the admin API and join back, no luck:

/ # synadm room delete-status -d uTabWIzPGlVybpcZ
{
    "delete_id": "uTabWIzPGlVybpcZ",
    "status": "failed",
    "shutdown_room": {
        "kicked_users": [],
        "failed_to_kick_users": [
            "@user:redacted"
        ],
        "local_aliases": [],
        "new_room_id": null
    }
}

Steps to reproduce

Unfortunately, I have no idea how to reproduce it. It just randomly happened today after I tried to write something. Probably the problem is somehow related to retention I enabled 1 week ago, or history purge I did with the admin API (also around 1 week ago).

Homeserver

parlor.chf.house

Synapse Version

1.96.1

Installation Method

Docker (matrixdotorg/synapse)

Database

Single PostgresSQL instance. Didn't port it from SQLite nor did any backup restoration

Workers

Multiple workers

Platform

My private server, everything is running in docker containers

Configuration

No response

Relevant log output

When I try to write something there or leave the channel I get the following error:

matrix-nginx            | 172.18.0.2 - - [22/Nov/2023:13:17:41 +0000] "OPTIONS /_matrix/client/v3/rooms/!ehXvUhWNASUkSLvAGP%3Amatrix.org/send/m.room.message/m1700659050405.95 HTTP/1.1" 204 0 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Element/1.11.49 Chrome/118.0.5993.120 Electron/27.0.4 Safari/537.36" "79.248.1.168"
matrix-worker-5         | 2023-11-22 13:17:41,896 - synapse.http.server - 124 - INFO - PUT-10309 - <XForwardedForRequest at 0x7faaf2f6d850 method='PUT' uri='/_matrix/client/v3/rooms/!ehXvUhWNASUkSLvAGP%3Amatrix.org/send/m.room.message/m1700659050405.95' clientproto='HTTP/1.0' site='unix'> SynapseError: 403 - No create event in auth events
matrix-worker-5         | 2023-11-22 13:17:41,897 - synapse.access.http.unix - 465 - INFO - PUT-10309 - 172.18.0.2 - unix - {@user:redacted} Processed request: 0.022sec/0.000sec (0.006sec, 0.001sec) (0.002sec/0.011sec/9) 66B 403 "PUT /_matrix/client/v3/rooms/!ehXvUhWNASUkSLvAGP%3Amatrix.org/send/m.room.message/m1700659050405.95 HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Element/1.11.49 Chrome/118.0.5993.120 Electron/27.0.4 Safari/537.36" [0 dbevts]
matrix-nginx            | 172.18.0.2 - - [22/Nov/2023:13:17:41 +0000] "PUT /_matrix/client/v3/rooms/!ehXvUhWNASUkSLvAGP%3Amatrix.org/send/m.room.message/m1700659050405.95 HTTP/1.1" 403 77 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Element/1.11.49 Chrome/118.0.5993.120 Electron/27.0.4 Safari/537.36" "79.248.1.168"

Please let me know if I can provide any additional logs
@VPaulV VPaulV changed the title Cannot write to a room/leave it with MatrixError: [403] No create event in auth events Cannot write to a room(or leave it) with MatrixError: [403] No create event in auth events Nov 22, 2023
@VPaulV
Copy link
Author

VPaulV commented Nov 22, 2023 •
edited
Loading

kind of resolved by using room delete API with force_purge, but I am not sure is it a real solution...

@DMRobertson
Copy link
Contributor

No create event in auth events

Odd. My guess is that your server's view of the current room state got corrupted such that it didn't see the create event. That is very odd and should never happen.

kind of resolved by using room delete API with force_purge, but I am not sure is it a real solution...

Unfortunately I don't think we'll be able to diagnose what happened now that the data has been deleted. Do you have any kind of backup of the database while you were stuck unable to send messages?

Probably the problem is somehow related to retention I enabled 1 week ago, or history purge I did with the admin API (also around 1 week ago).

When you enabled retention, what Synapse version were you running?

@DMRobertson DMRobertson added S-Minor Blocks non-critical functionality, workarounds exist. T-Defect Bugs, crashes, hangs, security vulnerabilities, or other reported issues. A-Database DB stuff like queries, migrations, new/remove columns, indexes, unexpected entries in the db O-Uncommon Most users are unlikely to come across this or unexpected workflow A-Retention Retention rules to delete messages after a certain amount of time labels Nov 23, 2023
@DMRobertson
Copy link
Contributor

Per #7642 (comment) it sounds like force-purging and rejoining the room was the right thing to do, however.

@DMRobertson DMRobertson added the X-Needs-Info This issue is blocked awaiting information from the reporter label Nov 23, 2023
@VPaulV
Copy link
Author

VPaulV commented Nov 23, 2023 •
edited
Loading

When you enabled retention, what Synapse version were you running?

I think it was 1.96.1

Do you have any kind of backup of the database while you were stuck unable to send messages?

Yes, I have a backup, so let me know if I can provide any logs to diagnose what happened

@DMRobertson
Copy link
Contributor

Apologies for the delay in responding, and thanks for getting back to me. I think the team isn't going to have much time to track this down right now, so I suggest we consider this issue closed. But: if you see this again in a room that was created on Synapse 1.94 or newer, please re-open and we can take another look.

@DMRobertson
Copy link
Contributor

(and I say 1.94 because that's when #16382 landed)

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
A-Database DB stuff like queries, migrations, new/remove columns, indexes, unexpected entries in the db A-Retention Retention rules to delete messages after a certain amount of time O-Uncommon Most users are unlikely to come across this or unexpected workflow S-Minor Blocks non-critical functionality, workarounds exist. T-Defect Bugs, crashes, hangs, security vulnerabilities, or other reported issues. X-Needs-Info This issue is blocked awaiting information from the reporter
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@VPaulV @DMRobertson