ACME support is broken with Python 3.5.2

[babolivier]

When upgrading my HS to 0.99.4, Synapse started crashing at startup, with the following error:

2019-05-17 09:04:12,811 - twisted - 242 - ERROR - - Error during startup:
2019-05-17 09:04:12,813 - twisted - 242 - ERROR - - Traceback (most recent call last):
2019-05-17 09:04:12,813 - twisted - 242 - ERROR - -   File "/opt/venvs/matrix-synapse/lib/python3.5/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
2019-05-17 09:04:12,815 - twisted - 242 - ERROR - -     result = g.send(result)
2019-05-17 09:04:12,816 - twisted - 242 - ERROR - -   File "/opt/venvs/matrix-synapse/lib/python3.5/site-packages/synapse/app/homeserver.py", line 432, in start
2019-05-17 09:04:12,818 - twisted - 242 - ERROR - -     yield acme.start_listening()
2019-05-17 09:04:12,818 - twisted - 242 - ERROR - -   File "/opt/venvs/matrix-synapse/lib/python3.5/site-packages/twisted/internet/defer.py", line 1613, in unwindGenerator
2019-05-17 09:04:12,819 - twisted - 242 - ERROR - -     return _cancellableInlineCallbacks(gen)
2019-05-17 09:04:12,820 - twisted - 242 - ERROR - -   File "/opt/venvs/matrix-synapse/lib/python3.5/site-packages/twisted/internet/defer.py", line 1529, in _cancellableInlineCallbacks
2019-05-17 09:04:12,821 - twisted - 242 - ERROR - -     _inlineCallbacks(None, g, status)
2019-05-17 09:04:12,822 - twisted - 242 - ERROR - - --- <exception caught here> ---
2019-05-17 09:04:12,822 - twisted - 242 - ERROR - -   File "/opt/venvs/matrix-synapse/lib/python3.5/site-packages/synapse/app/homeserver.py", line 432, in start
2019-05-17 09:04:12,823 - twisted - 242 - ERROR - -     yield acme.start_listening()
2019-05-17 09:04:12,824 - twisted - 242 - ERROR - -   File "/opt/venvs/matrix-synapse/lib/python3.5/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
2019-05-17 09:04:12,824 - twisted - 242 - ERROR - -     result = g.send(result)
2019-05-17 09:04:12,825 - twisted - 242 - ERROR - -   File "/opt/venvs/matrix-synapse/lib/python3.5/site-packages/synapse/handlers/acme.py", line 122, in start_listening
2019-05-17 09:04:12,826 - twisted - 242 - ERROR - -     yield self._issuer._ensure_registered()
2019-05-17 09:04:12,827 - twisted - 242 - ERROR - -   File "/opt/venvs/matrix-synapse/lib/python3.5/site-packages/txacme/service.py", line 243, in _ensure_registered
2019-05-17 09:04:12,828 - twisted - 242 - ERROR - -     return self._with_client(self._register)
2019-05-17 09:04:12,829 - twisted - 242 - ERROR - -   File "/opt/venvs/matrix-synapse/lib/python3.5/site-packages/txacme/service.py", line 184, in _with_client
2019-05-17 09:04:12,830 - twisted - 242 - ERROR - -     return self._client_creator().addCallback(f, *a, **kw)
2019-05-17 09:04:12,831 - twisted - 242 - ERROR - -   File "/opt/venvs/matrix-synapse/lib/python3.5/site-packages/synapse/handlers/acme.py", line 88, in <lambda>
2019-05-17 09:04:12,831 - twisted - 242 - ERROR - -     alg=RS256,
2019-05-17 09:04:12,832 - twisted - 242 - ERROR - -   File "/opt/venvs/matrix-synapse/lib/python3.5/site-packages/txacme/client.py", line 127, in from_url
2019-05-17 09:04:12,834 - twisted - 242 - ERROR - -     url=url, key_type=key.typ, alg=alg.name)
2019-05-17 09:04:12,834 - twisted - 242 - ERROR - -   File "/opt/venvs/matrix-synapse/lib/python3.5/site-packages/eliot/_validation.py", line 450, in __call__
2019-05-17 09:04:12,836 - twisted - 242 - ERROR - -     logger, self.action_type, self._serializers, **fields)
2019-05-17 09:04:12,836 - twisted - 242 - ERROR - -   File "/opt/venvs/matrix-synapse/lib/python3.5/site-packages/eliot/_action.py", line 772, in start_action
2019-05-17 09:04:12,838 - twisted - 242 - ERROR - -     parent = current_action()
2019-05-17 09:04:12,839 - twisted - 242 - ERROR - -   File "/opt/venvs/matrix-synapse/lib/python3.5/site-packages/eliot/_action.py", line 53, in current_action
2019-05-17 09:04:12,839 - twisted - 242 - ERROR - -     return _ACTION_CONTEXT.get(None)
2019-05-17 09:04:12,840 - twisted - 242 - ERROR - -   File "/opt/venvs/matrix-synapse/lib/python3.5/site-packages/contextvars/__init__.py", line 94, in get
2019-05-17 09:04:12,841 - twisted - 242 - ERROR - -     ctx = _get_context()
2019-05-17 09:04:12,842 - twisted - 242 - ERROR - -   File "/opt/venvs/matrix-synapse/lib/python3.5/site-packages/aiocontextvars.py", line 19, in _get_context
2019-05-17 09:04:12,843 - twisted - 242 - ERROR - -     state = _get_state()
2019-05-17 09:04:12,844 - twisted - 242 - ERROR - -   File "/opt/venvs/matrix-synapse/lib/python3.5/site-packages/aiocontextvars.py", line 33, in _get_state
2019-05-17 09:04:12,845 - twisted - 242 - ERROR - -     loop = asyncio._get_running_loop()
2019-05-17 09:04:12,845 - twisted - 242 - ERROR - - builtins.AttributeError: module 'asyncio' has no attribute '_get_running_loop'

I managed to get it back up by disabling ACME support.

If that's of any help, my HS is running on a Ubuntu Xenial server, and from the stacktrace it looks like it's using Python 3.5.

[richvdh]

Right, so:

• the acme support uses txacme
• which uses eliot
• which, since 1.8.0 (Switch to using contextvars, in order to support Trio itamarst/eliot#397) uses aiocontextvars
• which uses an undocumented method asyncio._get_running_loop, which was added to the standard library in python 3.5.3
• ubuntu xenial has 3.5.2.

So I guess a potential solution is to pin to eliot < 1.8.0 for xenial?

[richvdh]

I don't even know who I can blame for this. aiocontextvars for using private methods? xenial for using an old python 3.5 ?

[richvdh]

note that this applies to both the debian package, and those installing from pip.

[richvdh]

raised with eliot as itamarst/eliot#418