Handle random remote disconnects?

[randomir]

This is a traceback from a dwave-system integration test (Python 3.4 on OS X):

Traceback (most recent call last):
  File "/Users/distiller/repo/env/lib/python3.4/site-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/Users/distiller/repo/env/lib/python3.4/site-packages/urllib3/connectionpool.py", line 384, in _make_request
    six.raise_from(e, None)
  File "<string>", line 2, in raise_from
  File "/Users/distiller/repo/env/lib/python3.4/site-packages/urllib3/connectionpool.py", line 380, in _make_request
    httplib_response = conn.getresponse()
  File "/Users/distiller/.pyenv/versions/3.4.8/lib/python3.4/http/client.py", line 1227, in getresponse
    response.begin()
  File "/Users/distiller/.pyenv/versions/3.4.8/lib/python3.4/http/client.py", line 386, in begin
    version, status, reason = self._read_status()
  File "/Users/distiller/.pyenv/versions/3.4.8/lib/python3.4/http/client.py", line 356, in _read_status
    raise BadStatusLine(line)
http.client.BadStatusLine: ''

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/distiller/repo/env/lib/python3.4/site-packages/requests/adapters.py", line 445, in send
    timeout=timeout
  File "/Users/distiller/repo/env/lib/python3.4/site-packages/urllib3/connectionpool.py", line 638, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/Users/distiller/repo/env/lib/python3.4/site-packages/urllib3/util/retry.py", line 367, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/Users/distiller/repo/env/lib/python3.4/site-packages/urllib3/packages/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "/Users/distiller/repo/env/lib/python3.4/site-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/Users/distiller/repo/env/lib/python3.4/site-packages/urllib3/connectionpool.py", line 384, in _make_request
    six.raise_from(e, None)
  File "<string>", line 2, in raise_from
  File "/Users/distiller/repo/env/lib/python3.4/site-packages/urllib3/connectionpool.py", line 380, in _make_request
    httplib_response = conn.getresponse()
  File "/Users/distiller/.pyenv/versions/3.4.8/lib/python3.4/http/client.py", line 1227, in getresponse
    response.begin()
  File "/Users/distiller/.pyenv/versions/3.4.8/lib/python3.4/http/client.py", line 386, in begin
    version, status, reason = self._read_status()
  File "/Users/distiller/.pyenv/versions/3.4.8/lib/python3.4/http/client.py", line 356, in _read_status
    raise BadStatusLine(line)
urllib3.exceptions.ProtocolError: ('Connection aborted.', BadStatusLine("''",))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/distiller/repo/tests/integration/test_virtual_graph_composite.py", line 26, in test_construction
    sampler = VirtualGraphComposite(child_sampler, embedding)
  File "/Users/distiller/repo/dwave/system/composites/virtual_graph.py", line 133, in __init__
    max_age=flux_bias_max_age)
  File "/Users/distiller/repo/dwave/system/flux_bias_offsets.py", line 64, in get_flux_biases
    chain_strength=chain_strength)
  File "/Users/distiller/repo/env/lib/python3.4/site-packages/dwave/drivers/flux_bias_offsets.py", line 59, in oneshot_flux_bias
    ignore_single_qubit_chains=ignore_single_qubit_chains)
  File "/Users/distiller/repo/env/lib/python3.4/site-packages/dwave/drivers/flux_bias_offsets.py", line 195, in calculate_flux_biases
    averages[fbo] = chain_average(chain, response)
  File "/Users/distiller/repo/env/lib/python3.4/site-packages/dwave/drivers/flux_bias_offsets.py", line 245, in chain_average
    if not response:
  File "/Users/distiller/repo/env/lib/python3.4/site-packages/dimod/response.py", line 110, in __len__
    return self.record.__len__()
  File "/Users/distiller/repo/env/lib/python3.4/site-packages/dimod/response.py", line 151, in record
    self._resolve_future()
  File "/Users/distiller/repo/env/lib/python3.4/site-packages/dimod/response.py", line 383, in _resolve_future
    response = self._result_hook(self._future)
  File "/Users/distiller/repo/dwave/system/samplers/dwave_sampler.py", line 296, in _hook
    result = computation.result()
  File "/Users/distiller/repo/env/lib/python3.4/site-packages/dwave/cloud/computation.py", line 454, in result
    self._load_result()
  File "/Users/distiller/repo/env/lib/python3.4/site-packages/dwave/cloud/computation.py", line 619, in _load_result
    six.reraise(*self._exc_info)
  File "/Users/distiller/repo/env/lib/python3.4/site-packages/six.py", line 693, in reraise
    raise value
  File "/Users/distiller/repo/env/lib/python3.4/site-packages/dwave/cloud/client.py", line 1014, in _do_load_results
    response = self.session.get(posixpath.join(self.endpoint, query_string))
  File "/Users/distiller/repo/env/lib/python3.4/site-packages/requests/sessions.py", line 525, in get
    return self.request('GET', url, **kwargs)
  File "/Users/distiller/repo/env/lib/python3.4/site-packages/requests/sessions.py", line 512, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/distiller/repo/env/lib/python3.4/site-packages/requests/sessions.py", line 622, in send
    r = adapter.send(request, **kwargs)
  File "/Users/distiller/repo/env/lib/python3.4/site-packages/requests/adapters.py", line 495, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', BadStatusLine("''",))

The reason could be related to SAPI webservice Keep-Alive timeouts: server closes connection, and client tries to write while not realising the connection (http session) is closed.

In that case we could retry on BadStatusLine, or RemoteDisconnected (in py3.5+).

[randomir]

All signs point to web server configuration error, since the problem can not be reproduced after server config upgrade.

For other deployments, this might help: https://stackoverflow.com/a/47077330/404556.

The alternative of client-side-instructed preemptive connection closing (sending Connection: close) also proved to be effective.

[randomir]

Possible cause: psf/requests#4664.

Implementing #414 should serve as a workaround here.

[randomir]

Reopening until fix from #414 is verified.