servers cannot re-join rooms after rotating signing keys and clearing database

[richvdh]

if a server is re-provisioned (so that the signing key is replaced, and the event store is cleared), then it cannot re-join any rooms it was previously participating in, because it cannot validate the auth chain:

2021-11-17 15:45:59,186 - synapse.federation.federation_base - 63 - WARNING - POST-150 - Signature check failed for $16371629950pDraD:synapse-performance-test.lab.element.dev: 403: event id $16371629950pDraD:synapse-performance-test.lab.element.dev: unable to verify signature for sender synapse-performance-test.lab.element.dev: 401: Invalid signature for server synapse-performance-test.lab.element.dev with key ed25519:a_qdcq: Missing signature for synapse-performance-test.lab.element.dev, ed25519:a_qdcq
2021-11-17 15:45:59,187 - synapse.federation.federation_base - 63 - WARNING - POST-150 - Signature check failed for $16371620780tvsmk:synapse-performance-test.lab.element.dev: 403: event id $16371620780tvsmk:synapse-performance-test.lab.element.dev: unable to verify signature for sender synapse-performance-test.lab.element.dev: 401: Invalid signature for server synapse-performance-test.lab.element.dev with key ed25519:a_qdcq: Missing signature for synapse-performance-test.lab.element.dev, ed25519:a_qdcq
2021-11-17 15:45:59,333 - synapse.http.matrixfederationclient - 684 - WARNING - POST-150 - {GET-O-74} [synapse-performance-test.lab.element.dev] Request failed: GET matrix://synapse-performance-test.lab.element.dev/_matrix/federation/v1/event/%2416371629950pDraD%3Asynapse-performance-test.lab.element.dev: HttpResponseException('404: Not Found')
2021-11-17 15:45:59,336 - synapse.http.matrixfederationclient - 684 - WARNING - POST-150 - {GET-O-75} [synapse-performance-test.lab.element.dev] Request failed: GET matrix://synapse-performance-test.lab.element.dev/_matrix/federation/v1/event/%2416371620780tvsmk%3Asynapse-performance-test.lab.element.dev: HttpResponseException('404: Not Found')
2021-11-17 15:45:59,538 - synapse.handlers.federation_event - 466 - INFO - POST-150 - Peristing join-via-remote <FrozenEvent event_id=$16371639590LlOmN:synapse-performance-test.lab.element.dev, type=m.room.member, state_key=@admin:synapse-performance-test.lab.element.dev, outlier=False>
2021-11-17 15:45:59,549 - synapse.handlers.federation_event - 1661 - INFO - POST-150-$16371639590LlOmN:synapse-performance-test.lab.element.dev - Event <FrozenEvent event_id=$16371639590LlOmN:synapse-performance-test.lab.element.dev, type=m.room.member, state_key=@admin:synapse-performance-test.lab.element.dev, outlier=False> refers to unknown auth events {'$16371629950pDraD:synapse-performance-test.lab.element.dev'}: fetching auth chain
2021-11-17 15:45:59,612 - synapse.http.matrixfederationclient - 291 - INFO - POST-150-$16371639590LlOmN:synapse-performance-test.lab.element.dev - {GET-O-77} [matrix.org] Completed request: 200 OK in 0.06 secs, got 5942 bytes - GET matrix://matrix.org/_matrix/federation/v1/event_auth/%21haTBqoGqvUMbmwFMVO%3Amatrix.org/%2416371639590LlOmN%3Asynapse-performance-test.lab.element.dev
2021-11-17 15:45:59,615 - synapse.federation.federation_base - 63 - WARNING - POST-150-$16371639590LlOmN:synapse-performance-test.lab.element.dev - Signature check failed for $16371629950pDraD:synapse-performance-test.lab.element.dev: 403: event id $16371629950pDraD:synapse-performance-test.lab.element.dev: unable to verify signature for sender synapse-performance-test.lab.element.dev: 401: Invalid signature for server synapse-performance-test.lab.element.dev with key ed25519:a_qdcq: Missing signature for synapse-performance-test.lab.element.dev, ed25519:a_qdcq
2021-11-17 15:45:59,616 - synapse.federation.federation_base - 63 - WARNING - POST-150-$16371639590LlOmN:synapse-performance-test.lab.element.dev - Signature check failed for $16371620780tvsmk:synapse-performance-test.lab.element.dev: 403: event id $16371620780tvsmk:synapse-performance-test.lab.element.dev: unable to verify signature for sender synapse-performance-test.lab.element.dev: 401: Invalid signature for server synapse-performance-test.lab.element.dev with key ed25519:a_qdcq: Missing signature for synapse-performance-test.lab.element.dev, ed25519:a_qdcq
2021-11-17 15:45:59,619 - synapse.federation.federation_client - 495 - WARNING - POST-150-$16371639590LlOmN:synapse-performance-test.lab.element.dev - Failed to find copy of $16371629950pDraD:synapse-performance-test.lab.element.dev with valid signature
2021-11-17 15:45:59,619 - synapse.federation.federation_client - 495 - WARNING - POST-150-$16371639590LlOmN:synapse-performance-test.lab.element.dev - Failed to find copy of $16371620780tvsmk:synapse-performance-test.lab.element.dev with valid signature
2021-11-17 15:45:59,619 - synapse.handlers.federation_event - 1718 - INFO - POST-150-$16371639590LlOmN:synapse-performance-test.lab.element.dev - /event_auth returned 4 events
2021-11-17 15:45:59,641 - synapse.handlers.federation_event - 1685 - WARNING - POST-150-$16371639590LlOmN:synapse-performance-test.lab.element.dev - Missing auth events for <FrozenEvent event_id=$16371639590LlOmN:synapse-performance-test.lab.element.dev, type=m.room.member, state_key=@admin:synapse-performance-test.lab.element.dev, outlier=False>: ['$16371629950pDraD:synapse-performance-test.lab.element.dev']
2021-11-17 15:45:59,641 - synapse.http.server - 92 - INFO - POST-150 - <XForwardedForRequest at 0x7f7d70196580 method='POST' uri='/_matrix/client/r0/join/%23vdhtest51%3Amatrix.org' clientproto='HTTP/1.1' site='8008'> SynapseError: 403 - Auth events could not be found
2021-11-17 15:45:59,642 - synapse.access.http.8008 - 400 - INFO - POST-150 - ::1 - 8008 - {@admin:synapse-performance-test.lab.element.dev} Processed request: 0.956sec/0.000sec (0.041sec, 0.004sec) (0.007sec/0.019sec/19) 66B 403 "POST /_matrix/client/r0/join/%23vdhtest51%3Amatrix.org HTTP/1.1" "curl/7.74.0" [9 dbevts]

[H-Shay]

So presumably this is due to the behavior change in #11129, if I understand correctly the newly-provisioned server is attempting to verify it's old signature with the new key, causing a failure?

[richvdh]

oops sorry Shay, I missed your question here and forged on with a fix!

Yes, that's correct - it's not checking which key was used for the signature and tries to validate it with the current key, regardless.