Fix package upload using PyOpenSSL

[dsludwig]

Request body must only use bytes, rather than unicode objects.

Using six.b together with unicode_literals does not work correctly -
on Python 2.7, six.b is a pass-through (i.e. lambda x: x). If we are
using unicode_literals, then we should explicitly encode or use
b'strings' where appropriate.

Fixes #222, anaconda-graveyard/anaconda-build#108

[rafaelhbarros]

This won't work because of this traceback:

Using anaconda-server api site https://api.anaconda.org
Getting build product: /Users/rdebarros/work/anaconda-server
Submitting 1 sub builds
 0) linux-64    python=2.7       None           
Archiving build directory for upload ...
Created archive (805 files); Uploading to binstar
 uploaded 122 of 6108Kb: 2.00% ETA: 0.6 minutes[ERROR] 
Traceback (most recent call last):
  File "/Users/rdebarros/miniconda3/envs/anaconda.org/bin/anaconda", line 9, in <module>
    load_entry_point('anaconda-client==1.1.2+14.g241ef3c', 'console_scripts', 'anaconda')()
  File "/Users/rdebarros/work/anaconda_client/binstar_client/scripts/cli.py", line 82, in main
    description=__doc__, version=version)
  File "/Users/rdebarros/work/anaconda_client/binstar_client/scripts/cli.py", line 64, in binstar_main
    return args.main(args)
  File "/Users/rdebarros/work/anaconda-build/binstar_build_client/build_commands/submit.py", line 207, in main
    submit_build(binstar, args)
  File "/Users/rdebarros/work/anaconda-build/binstar_build_client/build_commands/submit.py", line 97, in submit_build
    test_only=args.test_only, callback=upload_print_callback(args))
  File "/Users/rdebarros/work/anaconda-build/binstar_build_client/mixins/build.py", line 57, in submit_for_build
    s3res = requests.post(s3url, data=data_stream, verify=True, timeout=10 * 60 * 60, headers=headers)
  File "/Users/rdebarros/miniconda3/envs/anaconda.org/lib/python2.7/site-packages/requests/api.py", line 109, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/Users/rdebarros/miniconda3/envs/anaconda.org/lib/python2.7/site-packages/requests/api.py", line 50, in request
    response = session.request(method=method, url=url, **kwargs)
  File "/Users/rdebarros/miniconda3/envs/anaconda.org/lib/python2.7/site-packages/requests/sessions.py", line 468, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/rdebarros/miniconda3/envs/anaconda.org/lib/python2.7/site-packages/requests/sessions.py", line 576, in send
    r = adapter.send(request, **kwargs)
  File "/Users/rdebarros/miniconda3/envs/anaconda.org/lib/python2.7/site-packages/requests/adapters.py", line 370, in send
    timeout=timeout
  File "/Users/rdebarros/miniconda3/envs/anaconda.org/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 559, in urlopen
    body=body, headers=headers)
  File "/Users/rdebarros/miniconda3/envs/anaconda.org/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 353, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/Users/rdebarros/miniconda3/envs/anaconda.org/lib/python2.7/httplib.py", line 1053, in request
    self._send_request(method, url, body, headers)
  File "/Users/rdebarros/miniconda3/envs/anaconda.org/lib/python2.7/httplib.py", line 1093, in _send_request
    self.endheaders(body)
  File "/Users/rdebarros/miniconda3/envs/anaconda.org/lib/python2.7/httplib.py", line 1049, in endheaders
    self._send_output(message_body)
  File "/Users/rdebarros/miniconda3/envs/anaconda.org/lib/python2.7/httplib.py", line 897, in _send_output
    self.send(message_body)
  File "/Users/rdebarros/miniconda3/envs/anaconda.org/lib/python2.7/httplib.py", line 866, in send
    self.sock.sendall(datablock)
  File "/Users/rdebarros/miniconda3/envs/anaconda.org/lib/python2.7/site-packages/requests/packages/urllib3/contrib/pyopenssl.py", line 220, in sendall
    sent = self._send_until_done(data[total_sent:total_sent+SSL_WRITE_BLOCKSIZE])
  File "/Users/rdebarros/miniconda3/envs/anaconda.org/lib/python2.7/site-packages/requests/packages/urllib3/contrib/pyopenssl.py", line 206, in _send_until_done
    return self.connection.send(data)
  File "/Users/rdebarros/miniconda3/envs/anaconda.org/lib/python2.7/site-packages/OpenSSL/SSL.py", line 950, in send
    self._raise_ssl_error(self._ssl, result)
  File "/Users/rdebarros/miniconda3/envs/anaconda.org/lib/python2.7/site-packages/OpenSSL/SSL.py", line 871, in _raise_ssl_error
    _raise_current_error()
  File "/Users/rdebarros/miniconda3/envs/anaconda.org/lib/python2.7/site-packages/OpenSSL/_util.py", line 22, in exception_from_error_queue
    raise exceptionType(errors)
Error: [('SSL routines', 'ssl3_write_pending', 'bad write retry')]

[dsludwig]

Ok.. I think this is blocked on urllib3/urllib3#717 then. As far as I can tell, this may be a while - a new version of urllib3 will need to be released, and then a new version of requests including the updated urllib3.

[rafaelhbarros]

try reproducing it, not sure if that was a fringe case or not.

[dsludwig]

I can reproduce this. Based on the linked bug, it would occur anytime the uploaded data is large enough.

Perhaps we should require requests <= 2.7?

[rafaelhbarros]

if that's the fix, I would suggest talking with more people.

[srossross]

Do we know what message was sent?

[rafaelhbarros]

I think I lost the context, @srossross , what do you mean?

[srossross]

how can I reproduce this issue? I can't seem to reproduce with this environment:

bash-4.1# conda list
# packages in environment at /opt/miniconda:
#
anaconda-client (/workspace/anaconda.client) 1.1.2+14.g241ef3c.dirty           <pip>
clyent                    1.0.0                    py27_0  
conda                     3.18.5                   py27_0  
conda-build               1.18.1                   py27_0  
conda-env                 2.4.4                    py27_0  
openssl                   1.0.2d                        0  
python                    2.7.10                        2  
requests                  2.8.1                    py27_0  
....
...

[dsludwig]

@srossross

to reproduce: conda install ndg-httpsclient pyasn1 PyOpenSSL and upload a sufficiently large package (not sure of the threshold, but Numpy would do).

[srossross]

👍