Invalid signature for server matrix.org

[ptman]

Description

Home server can't federate with other home servers. Federation tester showed no problems. Log had:

SynapseError: 401: Invalid signature for server matrix.org with key ed25519:auto

Steps to reproduce

• I installed synapse on a new domain (<new-hs.tld>) using the docker image.

• Tried to start a chat with my matrix.org account: @ptman:matrix.org

• We tried looking into it a bit with @richvdh . Added a patch with custom logging

• Did a bit of digging in the database with select * from server_signature_keys where server_name='...' and key_id='...', but everything seemed to be in order

synapse.access.http.8008 - 92 - INFO - GET-9210- 83.136.249.97 - 8008 - Received request: GET /_matrix/federation/v1/make_leave/!dheBoQuTbKKVQIWLYF:<new-hs.tld>/@ptman:matrix.org
synapse.crypto.keyring - 125 - DEBUG - GET-9210- Verifying for matrix.org with key_ids ['ed25519:auto']
synapse.crypto.keyring - 780 - DEBUG - GET-9210- Got key ed25519:auto ed25519:auto for server matrix.org, verifying
synapse.crypto.keyring - 789 - DEBUG - GET-9210- Error verifying signature for matrix.org:ed25519:auto with key Noi6WqcDj0QmPxCNQqgezwTlBKrfqehY1u2FyWP9uYw: Unable to verify signature for matrix.org
synapse.federation.transport.server - 196 - ERROR - GET-9210- authenticate_request failed
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/synapse/federation/transport/server.py", line 189, in new_func
    origin = yield authenticator.authenticate_request(request, content)
SynapseError: 401: Invalid signature for server matrix.org with key ed25519:auto: Unable to verify signature for matrix.org
synapse.http.server - 84 - INFO - GET-9210- <XForwardedForRequest at 0x7f486413bbd8 method='GET' uri='/_matrix/federation/v1/make_leave/!dheBoQuTbKKVQIWLYF:<new-hs.tld>/@ptman:matrix.org' clientproto='HTTP/1.1' site=8008> SynapseError: 401 - Invalid signature for server matrix.org with key ed25519:auto: Unable to verify signature for matrix.org
synapse.access.http.8008 - 126 - INFO - GET-9210- 83.136.249.97 - 8008 - {None} Processed request: 0.007sec (0.007sec, 0.000sec) (0.000sec/0.000sec/0) 144B 401 "GET /_matrix/federation/v1/make_leave/!dheBoQuTbKKVQIWLYF:<new-hs.tld>/@ptman:matrix.org HTTP/1.1" "Synapse/0.31.0 (b=matrix-org-hotfixes,ba3d015)"

Version information

If not matrix.org:

• Version: e82db24
• Install method: docker
• Platform: docker

[richvdh]

We tried looking into it a bit with @richvdh . Added a patch with custom logging

That was #3372 ftr

I'm afraid I ran out of time/ideas to investigate this. I think we'd have to compare what was being signed with what was being verified to figure out why the signature doesn't match

[richvdh]

a similar report at #2034 (comment) :/

[emdete]

i found a "ed25519" in the config homeserver.yaml under a key perspectives.servers.matrix.org.verify_keys, the key there is Noi6WqcDj0QmPxCNQqgezwTlBKrfqehY1u2FyWP9uYw - is that wrong / was that changed on server side? what is that anyway?

i commented the whole section and the error is gone from the logs. instead i see synapse.http.server - 87 - ERROR - GET-4- 403: Forbidden

[ptman]

The difference between this and my other homeserver is that the old working one is behind nginx and this behind apache. Seems that this may now have fixed my problem:

ProxyPass http://localhost:8008/_matrix retry=0 nocanon

A known-good example apache config would be nice to have in contrib or somewhere

[richvdh]

ugh, sorry not to have thought of this before. That makes this a dup of #3294