Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Synapse 1.3.0 fails to startup #5866

Closed
chrismoos opened this issue Aug 16, 2019 · 7 comments

Comments

@chrismoos
Copy link
Contributor

commented Aug 16, 2019

Description

Synapse 1.3.0 fails to startup with the following error:

2019-08-15 23:02:10,471 - twisted - 197 - ERROR - - Error during startup:
2019-08-15 23:02:10,472 - twisted - 197 - ERROR - - Traceback (most recent call last):
2019-08-15 23:02:10,472 - twisted - 197 - ERROR - - --- <exception caught here> ---
2019-08-15 23:02:10,473 - twisted - 197 - ERROR - -   File "/opt/venvs/matrix-synapse/lib/python3.6/site-packages/synapse/app/homeserver.py", line 429, in start
2019-08-15 23:02:10,473 - twisted - 197 - ERROR - -     yield acme.start_listening()
2019-08-15 23:02:10,473 - twisted - 197 - ERROR - -   File "/opt/venvs/matrix-synapse/lib/python3.6/site-packages/synapse/handlers/acme.py", line 74, in start_listening
2019-08-15 23:02:10,474 - twisted - 197 - ERROR - -     yield self._issuer._ensure_registered()
2019-08-15 23:02:10,474 - twisted - 197 - ERROR - - twisted.internet.error.DNSLookupError: DNS lookup failed: Couldn't find the hostname 'acme-v01.api.letsencrypt.org'.

It seems DNS related, but it's not. I looked into it and realized the issue is with Twisted's reactor (some race condition, not starting up at the right time or something). I've fixed it locally by reverting a change in 80cfad2, specifically 80cfad2#diff-01aac508ef38061957e6fb961582e5ceL450.

I'm not sure about the other files in the above change, but at least changing the one in homeserver.py seems to fix the issue. I'll push a PR with the change I made, but I'll let someone with more Twisted knowledge decide what the best long term fix is.

Steps to reproduce

Run Synapse 1.3.0 with ACME enabled.

Version information

  • Version: 1.3.0

  • Install method: apt

  • Platform: Ubuntu

chrismoos added a commit to chrismoos/synapse that referenced this issue Aug 16, 2019
chrismoos added a commit to chrismoos/synapse that referenced this issue Aug 16, 2019
Fix issue with Synapse not starting up. Fixes matrix-org#5866.
Signed-off-by: Chris Moos <chris@chrismoos.com>
@chresan

This comment has been minimized.

Copy link

commented Aug 16, 2019

So I got the same problem as.

My log files show:

2019-08-16 17:23:59,814 - twisted - 197 - ERROR - - Error during startup:
2019-08-16 17:23:59,815 - twisted - 197 - ERROR - - Traceback (most recent call last):
2019-08-16 17:23:59,815 - twisted - 197 - ERROR - - --- <exception caught here> ---
2019-08-16 17:23:59,815 - twisted - 197 - ERROR - -   File "/opt/venvs/matrix-synapse/lib/python3.5/site-packages/synapse/app/homeserver.py", line 429, in start
2019-08-16 17:23:59,816 - twisted - 197 - ERROR - -     yield acme.start_listening()
2019-08-16 17:23:59,816 - twisted - 197 - ERROR - -   File "/opt/venvs/matrix-synapse/lib/python3.5/site-packages/synapse/handlers/acme.py", line 74, in start_listening
2019-08-16 17:23:59,817 - twisted - 197 - ERROR - -     yield self._issuer._ensure_registered()
2019-08-16 17:23:59,817 - twisted - 197 - ERROR - - twisted.internet.error.DNSLookupError: DNS lookup failed: Couldn't find the hostname 'acme-v01.api.letsencrypt.org'.
2019-08-16 17:30:14,453 - twisted - 197 - CRITICAL - - Unhandled error in Deferred:
2019-08-16 17:30:14,454 - twisted - 197 - CRITICAL - -
Traceback (most recent call last):
  File "/opt/venvs/matrix-synapse/lib/python3.5/site-packages/synapse/app/homeserver.py", line 429, in start
    yield acme.start_listening()
twisted.internet.error.DNSLookupError: DNS lookup failed: Couldn't find the hostname 'acme-v01.api.letsencrypt.org'.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/venvs/matrix-synapse/lib/python3.5/site-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/opt/venvs/matrix-synapse/lib/python3.5/site-packages/twisted/python/failure.py", line 512, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/opt/venvs/matrix-synapse/lib/python3.5/site-packages/synapse/app/homeserver.py", line 448, in start
    sys.exit(1)
SystemExit: 1
@chresan

This comment has been minimized.

Copy link

commented Aug 16, 2019

I would like to add that the I can reach the mentioned host from the server.

ping acme-v01.api.letsencrypt.org
PING e14990.dscx.akamaiedge.net (104.111.246.175) 56(84) bytes of data.
64 bytes from a104-111-246-175.deploy.static.akamaitechnologies.com (104.111.246.175): icmp_seq=1 ttl=57 time=3.38 ms

@erikjohnston

This comment has been minimized.

Copy link
Member

commented Aug 16, 2019

A work around for now is to disable acme. We should release a fix for this soon

@chrismoos

This comment has been minimized.

Copy link
Contributor Author

commented Aug 16, 2019

@chresan You can run this to patch it:

curl -qL 'https://gist.github.com/chrismoos/504fdb1eebbdbffddea1fd3c4fc9d00f/raw/cd1e8db7e917c8eab849bc4b26d21d2a621ece92/gistfile1.txt' | sudo patch -p2 -d /opt/venvs/matrix-synapse/lib/python3.6/site-packages/synapse

@chresan

This comment has been minimized.

Copy link

commented Aug 16, 2019

I've just got the tip from eric (in synapse-dev room) to disable acme. After that I can run synapse again. I'll wait until there is a debian package with the fix instead of patching the code directly. Thanks.

@babolivier

This comment has been minimized.

Copy link
Member

commented Aug 17, 2019

FYI Synapse v1.3.1 has just been released with @chrismoos's fix.

@chresan

This comment has been minimized.

Copy link

commented Aug 23, 2019

I want to express my thanks to the devs for fixing this problem so quickly. I've updated my server and the issue is gone now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.