ResponseNeverReceived isn't always a timeout

[benbz]

Description

A Synapse installation configured with OIDC started failing after upgrading 1.21 -> 1.26/1.28 with RequestTimedOutError: 504: Timeout waiting for response from remote server. The installation is setup with a HTTP/HTTPS CONNECT proxy.

On investigation it turned out that Synapse was unable to verify the TLS certificate of the OIDC IdP with the default CA trust store.

I've changed the trust store by explicitly setting the SSL_CERT_FILE environment variable but this issue made it so the customer couldn't see the actual problem and took me editing the installed Python to discover it.

Steps to reproduce

• Synapse running in an environment that requires a HTTP/HTTPS CONNECT proxy
• Synapse configured with OIDC
• The default trust store (somewhat unsure which this is in this environment) can't verify the chain

....
Traceback (most recent call last):
  File "/opt/synapse/env/lib64/python3.6/site-packages/synapse/handlers/oidc_handler.py", line 97, in load_metadata
    await p.load_metadata()
  File "/opt/synapse/env/lib64/python3.6/site-packages/synapse/handlers/oidc_handler.py", line 372, in load_metadata
    metadata_response = await self._http_client.get_json(url)
  File "/opt/synapse/env/lib64/python3.6/site-packages/synapse/http/client.py", line 570, in get_json
    body = await self.get_raw(uri, args, headers=headers)
  File "/opt/synapse/env/lib64/python3.6/site-packages/synapse/http/client.py", line 655, in get_raw
    response = await self.request("GET", uri, headers=Headers(actual_headers))
  File "/opt/synapse/env/lib64/python3.6/site-packages/synapse/http/client.py", line 422, in request
    response = await make_deferred_yieldable(request_deferred)
  File "/opt/synapse/env/lib64/python3.6/site-packages/twisted/internet/defer.py", line 654, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/opt/synapse/env/lib64/python3.6/site-packages/synapse/http/client.py", line 750, in _timeout_to_request_timed_out_error
    raise RequestTimedOutError("Timeout waiting for response from remote server")
synapse.http.RequestTimedOutError: 504: Timeout waiting for response from remote server

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/synapse/env/lib64/python3.6/site-packages/synapse/app/_base.py", line 159, in wrapper
    await cb(*args, **kwargs)
  File "/opt/synapse/env/lib64/python3.6/site-packages/synapse/app/homeserver.py", line 431, in start
    await oidc.load_metadata()
  File "/opt/synapse/env/lib64/python3.6/site-packages/synapse/handlers/oidc_handler.py", line 102, in load_metadata
    ) from e
Exception: Error while initialising OIDC provider 'oidc'

Should have been:

...
Traceback (most recent call last):
  File "/opt/synapse/env/lib64/python3.6/site-packages/synapse/handlers/oidc_handler.py", line 97, in load_metadata
    await p.load_metadata()
  File "/opt/synapse/env/lib64/python3.6/site-packages/synapse/handlers/oidc_handler.py", line 372, in load_metadata
    metadata_response = await self._http_client.get_json(url)
  File "/opt/synapse/env/lib64/python3.6/site-packages/synapse/http/client.py", line 570, in get_json
    body = await self.get_raw(uri, args, headers=headers)
  File "/opt/synapse/env/lib64/python3.6/site-packages/synapse/http/client.py", line 655, in get_raw
    response = await self.request("GET", uri, headers=Headers(actual_headers))
  File "/opt/synapse/env/lib64/python3.6/site-packages/synapse/http/client.py", line 422, in request
    response = await make_deferred_yieldable(request_deferred)
twisted.web._newclient.ResponseNeverReceived: [<twisted.python.failure.Failure OpenSSL.SSL.Error: [('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')]>]

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/synapse/env/lib64/python3.6/site-packages/synapse/app/_base.py", line 159, in wrapper
    await cb(*args, **kwargs)
  File "/opt/synapse/env/lib64/python3.6/site-packages/synapse/app/homeserver.py", line 431, in start
    await oidc.load_metadata()
  File "/opt/synapse/env/lib64/python3.6/site-packages/synapse/handlers/oidc_handler.py", line 102, in load_metadata
    ) from e
Exception: Error while initialising OIDC provider 'oidc'

Version information

• Version: Experienced with 1.26 / 1.28. Was not happening on 1.21
• Install method:
  • Custom RPM built for S390x on RHEL7
  • The RPM is effectively a virtual environment with rh-python36