This repository was archived by the owner on Apr 26, 2024. It is now read-only.
Repository navigation
This repository was archived by the owner on Apr 26, 2024. It is now read-only.
Federation DNS Lookup fails for healthy domain #5831
Description
Hi,
I am getting significant amount of DNS lookup failures message in my Synapse log.
(excerpt only for readability reason)
2019-08-08 11:21:41,212 - synapse.http.matrixfederationclient - 485 - WARNING - PUT-7679- {POST-O-1905} [matrix.org] Request failed: POST matrix://matrix.org/_matrix/key/v2/query: DNSLookupError("Couldn't find the hostname 'matrix.org.cdn.cloudflare.net'",)
2019-08-08 11:21:41,520 - synapse.http.matrixfederationclient - 485 - WARNING - PUT-7649- {POST-O-1904} [matrix.org] Request failed: POST matrix://matrix.org/_matrix/key/v2/query: DNSLookupError("Couldn't find the hostname 'matrix.org.cdn.cloudflare.net'",)
2019-08-08 11:21:43,668 - synapse.http.matrixfederationclient - 485 - WARNING - PUT-7697- {POST-O-1906} [matrix.org] Request failed: POST matrix://matrix.org/_matrix/key/v2/query: DNSLookupError("Couldn't find the hostname 'matrix.org.cdn.cloudflare.net'",)
2019-08-08 11:21:44,043 - synapse.http.matrixfederationclient - 485 - WARNING - PUT-7708- {POST-O-1907} [matrix.org] Request failed: POST matrix://matrix.org/_matrix/key/v2/query: DNSLookupError("Couldn't find the hostname 'matrix.org.cdn.cloudflare.net'",)
2019-08-08 11:21:46,534 - synapse.http.matrixfederationclient - 485 - WARNING - PUT-7128- {POST-O-1908} [matrix.org] Request failed: POST matrix://matrix.org/_matrix/key/v2/query: DNSLookupError("Couldn't find the hostname 'matrix.org.cdn.cloudflare.net'",)
2019-08-08 11:21:46,581 - synapse.http.matrixfederationclient - 485 - WARNING - PUT-7435- {POST-O-1909} [matrix.org] Request failed: POST matrix://matrix.org/_matrix/key/v2/query: DNSLookupError("Couldn't find the hostname 'matrix.org.cdn.cloudflare.net'",)
2019-08-08 11:21:47,011 - synapse.http.matrixfederationclient - 485 - WARNING - PUT-7347- {POST-O-1910} [matrix.org] Request failed: POST matrix://matrix.org/_matrix/key/v2/query: DNSLookupError("Couldn't find the hostname 'matrix.org.cdn.cloudflare.net'",)
2019-08-08 11:21:47,123 - synapse.http.matrixfederationclient - 485 - WARNING - PUT-7344- {POST-O-1911} [matrix.org] Request failed: POST matrix://matrix.org/_matrix/key/v2/query: DNSLookupError("Couldn't find the hostname 'matrix.org.cdn.cloudflare.net'",)
Matrix.org can be resolved correctly on the host system:
user@host:~$ nslookup matrix.org.cdn.cloudflare.net
Server: 8.8.8.8
Address: 8.8.8.8#53
Non-authoritative answer:
Name: matrix.org.cdn.cloudflare.net
Address: 104.20.20.236
Name: matrix.org.cdn.cloudflare.net
Address: 104.20.21.236
Name: matrix.org.cdn.cloudflare.net
Address: 2606:4700:10::6814:15ec
Name: matrix.org.cdn.cloudflare.net
Address: 2606:4700:10::6814:14ec
user@host~$ host matrix.org.cdn.cloudflare.net
matrix.org.cdn.cloudflare.net has address 104.20.21.236
matrix.org.cdn.cloudflare.net has address 104.20.20.236
matrix.org.cdn.cloudflare.net has IPv6 address 2606:4700:10::6814:15ec
matrix.org.cdn.cloudflare.net has IPv6 address 2606:4700:10::6814:14ec
matrix.org.cdn.cloudflare.net mail is handled by 10 polemos.matrix.org.
matrix.org.cdn.cloudflare.net mail is handled by 20 zelus.matrix.org.
user@host:~$ nslookup bau-ha.us
Server: 8.8.8.8
Address: 8.8.8.8#53
I am not sure if Synapse uses twisted.names for resolving hostname to IP address,
Issue #1002 seem to indicate so, but @hawkowl at #synapse-dev room said this is not the case.
If twisted.names is indeed used, there is some weird behavior I noticed by which gethostbyname() refuses to work with combination of some domain and resolver (https://twistedmatrix.com/trac/ticket/9691).
Is this even reproducible on other HS setup?
Activity
richvdh commentedon Dec 3, 2019
synapse does not (normally) use twisted.names (see also #5053).
I think this must be specific to your setup, given the number of people that federate with matrix.org and have never reported this. What does your
/etc/resolv.confcontain?richvdh commentedon Dec 3, 2019
It seems it's more complicated and sometimes twisted.names is used.
That isn't what she said there.
Nevertheless I'd be surprised if this was https://twistedmatrix.com/trac/ticket/9691, and something specific to your environment seems more likely.
aryasenna commentedon Dec 8, 2019
My resolve.conf content was simply
I linked the matrix room chat, she said Synapse uses Python getbyaddress() or something, instead of twisted's implementation.
I found out that I am getting nslookup timeout error when I do repeated nslookup. (Maybe my provider rate-limits the DNS outbound query as some sort of DDoS mitigation).
In any case, Synapse certainly performs repeated lookup due to federation. So I installed systemd-resolved which caches the lookup result, and I have no more problems.
Probably it is specific to my setup, but Synapse's failure mode when dns lookup is rate-limited is not well defined here.
I'd think Synapse might want to rate limit the DNS query or cache it.
richvdh commentedon Dec 9, 2019
No, what she actually said was:
that is an aspiration, not a statement of fact. In practice twisted.names is used sometimes.
It can't ratelimit DNS without ratelimiting outgoing requests. That's basially #5373.
Caching is something that should be done by your system, not synapse. Installing
systemd-resolvedis a valid solution.