Result file upload exception

[rpanah]

This seems to be caused by the newer versions of PyOpenSSL. The upload mechanism has to be changed to become compatible with the new library.

2015-10-20 23:13:26,246 backend.py(line 87) INFO: Uploading result file - /home/abbas/.centinel/results/results-2015-10-02T18:07:07.448419_1.tar.bz2
2015-10-20 23:13:26,435 backend.py(line 104) ERROR: Error trying to submit result: [('SSL routines', 'SSL3_WRITE_PENDING', 'bad write retry')]
Traceback (most recent call last):
  File "/home/abbas/iclab/centinel/centinel/backend.py", line 98, in submit_result
    verify=self.verify)
  File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 109, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 50, in request
    response = session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 468, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 576, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 370, in send
    timeout=timeout
  File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py", line 559, in urlopen
    body=body, headers=headers)
  File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py", line 353, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python2.7/httplib.py", line 1048, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python2.7/httplib.py", line 1088, in _send_request
    self.endheaders(body)
  File "/usr/lib/python2.7/httplib.py", line 1044, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python2.7/httplib.py", line 888, in _send_output
    self.send(msg)
  File "/usr/lib/python2.7/httplib.py", line 864, in send
    self.sock.sendall(data)
  File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/contrib/pyopenssl.py", line 220, in sendall
    sent = self._send_until_done(data[total_sent:total_sent+SSL_WRITE_BLOCKSIZE])
  File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/contrib/pyopenssl.py", line 206, in _send_until_done
    return self.connection.send(data)
  File "/usr/local/lib/python2.7/dist-packages/OpenSSL/SSL.py", line 1271, in send
    self._raise_ssl_error(self._ssl, result)
  File "/usr/local/lib/python2.7/dist-packages/OpenSSL/SSL.py", line 1187, in _raise_ssl_error
    _raise_current_error()
  File "/usr/local/lib/python2.7/dist-packages/OpenSSL/_util.py", line 48, in exception_from_error_queue
    raise exception_type(errors)
Error: [('SSL routines', 'SSL3_WRITE_PENDING', 'bad write retry')]
2015-10-20 23:13:26,437 backend.py(line 315) ERROR: Unable to send result file: [('SSL routines', 'SSL3_WRITE_PENDING', 'bad write retry')]
Traceback (most recent call last):
  File "./run.py", line 6, in <module>
    centinel.cli.run()
  File "/home/abbas/iclab/centinel/centinel/cli.py", line 108, in run
    centinel.backend.sync(configuration.params)
  File "/home/abbas/iclab/centinel/centinel/backend.py", line 316, in sync
    raise exp
OpenSSL.SSL.Error: [('SSL routines', 'SSL3_WRITE_PENDING', 'bad write retry')]

[rnithyanand]

These might help:

http://stackoverflow.com/questions/2997218/why-am-i-getting-error1409f07fssl-routinesssl3-write-pending-bad-write-retr

http://stackoverflow.com/questions/20715408/solved-pyopenssl-ssl3-write-pendingbad-write-retry-return-self-sslobj-write

[sneft]

We have identified this on an RPi in the field:

2015-10-19 14:51:55,650: INFO: Starting sync with https://server.iclab.org:8082
2015-10-19 14:51:55,654: INFO: Uploading result file - /root/.centinel/results/pcap_baseline-2015-10-15T06:17:21.332736.pcap.bz2
2015-10-19 14:52:01,471: ERROR: Error trying to submit result: [('SSL routines', 'SSL3_WRITE_PENDING', 'bad write retry')]
2015-10-19 14:52:01,513: ERROR: Unable to send result file: [('SSL routines', 'SSL3_WRITE_PENDING', 'bad write retry')]
Traceback (most recent call last):
  File "/usr/local/bin/centinel-dev", line 11, in <module>
    sys.exit(run())
  File "/usr/local/lib/python2.7/dist-packages/centinel/cli.py", line 100, in run
    centinel.backend.sync(configuration.params)
  File "/usr/local/lib/python2.7/dist-packages/centinel/backend.py", line 315, in sync
    raise exp
OpenSSL.SSL.Error: [('SSL routines', 'SSL3_WRITE_PENDING', 'bad write retry')]

[lianke123321]

I could not reproduce this issue, but it is highly possible that this is caused by urllib3 incompatibility with new pyopenssl library.

Reference: urllib3/urllib3#717

The urllib3 version on my machine is 1.9.1, and the newest version is 1.13.

@rpanah and @sneft could you confirm the urllib3 version on your machine or RPi and try to update urllib3?

[jakubd]

pip levels are at:

centinel-dev==0.1.5.3
cffi==1.1.2
chardet==2.3.0
colorama==0.3.2
configobj==5.0.6
cryptography==0.9.3
dnspython==1.12.0
enum34==1.0.4
html5lib==0.999
idna==2.0
ipaddress==1.0.14
M2Crypto==0.22.3
Mako==1.0.0
MarkupSafe==0.23
ndg-httpsclient==0.4.0
ply==3.4
pyasn1==0.1.8
pycparser==2.14
pyinotify==0.9.4
pyOpenSSL==0.15.1
requests==2.8.1
six==1.9.0
urllib3==1.12
wheel==0.24.0

problem persists after upgrade of urllib3 from 1.9.1 to 1.12

1.13 wasn't offered by pip after pip install -U

[lianke123321]

The bug reporter of urllib3/urllib3#717 is also using urllib3 1.12.

My guess is that they haven't released a new version that contains this fix?

[rpanah]

Yeah. I could set the dependency version to the last good version for now. I'll test this solution.

[rpanah]

urllib1.12 introduced this issue, which has been fixed, but hasn't been released yet. Given urllib's release cycle length, we expect to see a new version soon.
Until then, we can force pip to use the last known good version (1.9.1).

[sneft]

Still happening with:
centinel==0.1.5.4.1
urllib3==1.9.1

and after installing libcurl4-openssl-dev

[rpanah]

@sneft, you should uninstall PyOpenSSL.
sudo pip uninstall pyopenssl

Let me know if that fixes your problem.

[sneft]

Looks like this worked. Thanks!

[lianke123321]

I was able to identify the real cause of this issue.

The exception is caused by the urllib3 code inside requests package. As you can see, the path in stack trace is "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/contrib/pyopenssl.py".

And this is why upgrading urllib3 doesn't make a difference.

On my machine, This issue is resolved by upgrading requests package from 2.8.1 to 2.9.1.

Someone please verify this solution, so that we could add requests and correct version into repo. No need to uninstall pyopenssl.