synapse gets stuck in a loop with `Invalid prev_events for <event_id>`

[richvdh]

If inbound federation traffic backs up in a v1 room, synapse starts spamming the logs with messages like:

2022-01-23 00:42:23,593 - synapse.storage.databases.main.event_federation - 1436 - INFO - _process_incoming_pdus_in_room_inner-1197913 - Invalid prev_events for <event_id>

this means that rather than dropping excess traffic as it should, it ends up in a tight loop and fails to process incoming traffic from other sources.

[richvdh]

see 2b9f741 which is patches this on the matrix-org-hotfixes branch

[richvdh]

(dates back to #10390)

[richvdh]

A workaround is to for all users on the server to leave the room containing the events (or to use the delete room admin API to shut it down).

[Cknight70]

Just applied this patch.
Previously my homeserver.log would get to 2gb in 30 minutes, Now it is only a few megabytes.
Ram usage is cut in half, messages are reliably sending again, thanks.

[redlewel]

I have this same issue, with synapse.storage.databases.main.event_federation - 1427 - INFO - _process_incoming_pdus_in_room_inner-4792 - Invalid prev_events for $164289566819600CTQgK:matrix.kiwifarms.net

being the log message. I also have been getting connection interrupts and obscene resource usage(100% CPU and maxing out 4GB mem). The logs were so constant that my server storage was filled up with a 153GB log file. I fixed this by making logging in /etc/matrix-synapse/log.yaml ERROR instead of INFO.

How do I apply the patch mentioned by @richvdh? Sorry I haven't made any changes to synapse besides the installation and setting up email and turnserver. Any help would be greatly appreciated.

[Cknight70]

@Redmauss There is probably a more graceful way to do this with patch but I did the following:

1. I found the location of synapse from the systemd service file /lib/systemd/system/matrix-synapse.service For me this was /opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/
2. cd into the directory and create a backup cp event_federation.py ~
3. Stop matrix synapse sudo systemctl stop matrix-synapse
4. Remove the old version sudo rm event_federation.py
5. Download the patched version sudo wget https://raw.githubusercontent.com/matrix-org/synapse/2b9f741f3a9dee93e9744774b342c35ce60062c4/synapse/storage/databases/main/event_federation.py
6. You can now start synapse.

I did this on Debian 11 with the official Debian Synapse package 1.50.1

[redlewel]

@Cknight70 worked like a charm! I am now using 245mb of RAM instead of over 3GB! My messages send reliably now and there are no connection dropouts. Thanks man, your writeup was super helpful.

I am using Ubuntu Server 20.04, and I installed the official matrix-synapse-py3 package. Steps were the exact same except it was Python version 3.8 instead of 3.9 for me.

[anoadragon453]

Fixed in #11806. The fix will be released in Synapse v1.50.2 and v1.51.0rc2.

Thank you all for testing the patch 🙂