/messages pagination fails with AssertionError: pulled event unexpectedly flagged as outlier if an event has a broken signature
#12584
Comments
anoadragon453
added
S-Major
|
yes, my matrix.cpn.so is down, because key lose. now, i changed the key and domain of my server. i'm constantly getting error messages, but I couldn't server infomation |
|
@andinue If possible, could you remove the https://matrix.cpn.so/.well-known/matrix/server file (or stop it from pointing to |
now, removed |
|
@andinue Thank you. Note that it may take some time (on the order of hours) for remote servers to re-check your well-known details and flush it from their cache. If you still see errors after 48hrs, feel free to create a new issue, or ask in #synapse:matrix.org and we can try and track them down. |
|
I have the same issue on my server from many different other homeservers. |
richvdh
changed the title
AssertionError: pulled event unexpectedly flagged as outlier errors on matrix.orgAssertionError: pulled event unexpectedly flagged as outlier
richvdh
changed the title
AssertionError: pulled event unexpectedly flagged as outlier/messages pagination fails with AssertionError: pulled event unexpectedly flagged as outlier
richvdh
changed the title
/messages pagination fails with AssertionError: pulled event unexpectedly flagged as outlier/messages pagination fails with AssertionError: pulled event unexpectedly flagged as outlier if an event has a broken signature
|
Although the particular problem noted above concerns events from a specific homeserver ( |
|
I think the right solution here is just to clear the outlier flag when we reprocess events in this way, though beware of reintroducing #12201. |
MadLittleMods
added
the
A-Messages-Endpoint
Update `get_pdu` to return the untouched, pristine `EventBase` as it was originally seen over federation (no metadata added). Previously, we returned the same `event` reference that we stored in the cache which downstream code modified in place and added metadata like setting it as an `outlier` and essentially poisoned our cache. Now we always return a copy of the `event` so the original can stay pristine in our cache and re-used for the next cache call. Split out from #13205 As discussed at: - #13205 (comment) - #13205 (comment) Related to #12584. This PR doesn't fix that issue because it hits [`get_event` which exists from the local database before it tries to `get_pdu`](https://github.com/matrix-org/synapse/blob/7864f33e286dec22368dc0b11c06eebb1462a51e/synapse/federation/federation_client.py#L581-L594).
Sentry link (internal access only): https://sentry.matrix.org/sentry/synapse-matrixorg/issues/248052
We are repeatedly returning 403's to
GET /_matrix/client/r0/rooms/{roomId}/messagesrequests due to trying to backfill old events in the room, before then choking on an event with an invalid signature, which we already have in our database (as an outlier).What happens:
A client hit
GET /_matrix/client/r0/rooms/{roomId}/messagesfor a room ID beginning with!lPCpzTqvU...(room version 6).matrix.org tried to gather events to service the request, and in doing so needed to
/backfillfrom another homeserver in the room.The
/backfillresponse included event ID$EysLi3HhoGiuQ142A6VoD8Y7F8qv2okN9sW74lIXV3M, which was created by someone onmatrix.cpn.so. It is am.room.memberevent for a (non-3pid) invite from one user onmatrix.cpn.soto another.This event is signed with
matrix.cpn.so's keyed25519:a_sWZq. We have a copy of this key in our database, but it has ats_valid_until_msof 500000 (Thu Jan 1 01:08:20 1970), so we attempt to fetch a fresher copy of the key.matrix.org attempts to reach out to
matrix.cpn.soto download its server keys, but fails withServerKeyFetcher-2245 - Error looking up keys ['ed25519:a_sWZq'] from matrix.cpn.so: Expected a response for server 'matrix.cpn.so' not 'espr.moe'. matrix.cpn.so's server well-known points toespr.moe:443, but querying https://espr.moe/_matrix/key/v2/server/ed25519:a_sWZq returns"server_name":"espr.moe"(!).Since we can't find a copy of the signing key that was valid at the time the event was created, validation fails.
We then attempt to pull an event with the same event ID from the database(!):
synapse/synapse/federation/federation_client.py
Lines 557 to 567 in 95a038c
(We're hitting the
except SynapseError: passbit.)Now, this event is marked as an outlier in matrix.org's database. Thus, the event gets the
outlier=Truebit of internal metadata.The outlier event is passed to
FederationEventHandler._process_pulled_eventsandFederationEventHandler._process_pulled_event. We then run into the assertion 💥So the problem in short:
matrix.cpn.so's federation is broken (it points to a homeserver with different keys), so attempting to validate the event while backfilling it fails./messagescall fails.full stacktrace and logs
(Note that the
Host not in room.error in the logs is due to #3736, and is unrelated to this issue).This one outlier event is causing many of these stacktraces to pop up every few seconds.
The text was updated successfully, but these errors were encountered: