With SSL connection: TypeError: data must be a memoryview, buffer or byte string

[Arno0x]

Hi,

I'm having an exception with my program, probably because of an update in one of the python libs recently, but I don't know how to figure this out.

OS: Linux kali 4.6.0-kali1-amd64 #1 SMP Debian 4.6.4-1kali1 (2016-07-21) x86_64 GNU/Linux
Python: 2.7.13
Requests version:

# pip show requests
Name: requests
Version: 2.12.4
Summary: Python HTTP for Humans.
Home-page: http://python-requests.org
Author: Kenneth Reitz
Author-email: me@kennethreitz.com
License: Apache 2.0
Location: /usr/local/lib/python2.7/dist-packages
Requires: 

Traceback:

[...]
 File "/root/Tools/DBC2/lib/dropboxHandler.py", line 32, in sendRequest
    r = requests.post(url,headers=headers,data=data)
  File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 110, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 56, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 488, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 609, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 423, in send
    timeout=timeout
  File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py", line 594, in urlopen
    chunked=chunked)
  File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py", line 361, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python2.7/httplib.py", line 1042, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python2.7/httplib.py", line 1082, in _send_request
    self.endheaders(body)
  File "/usr/lib/python2.7/httplib.py", line 1038, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python2.7/httplib.py", line 886, in _send_output
    self.send(message_body)
  File "/usr/lib/python2.7/httplib.py", line 858, in send
    self.sock.sendall(data)
  File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/contrib/pyopenssl.py", line 292, in sendall
    sent = self._send_until_done(data[total_sent:total_sent + SSL_WRITE_BLOCKSIZE])
  File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/contrib/pyopenssl.py", line 281, in _send_until_done
    return self.connection.send(data)
  File "/usr/lib/python2.7/dist-packages/OpenSSL/SSL.py", line 1251, in send
    raise TypeError("data must be a memoryview, buffer or byte string")
TypeError: data must be a memoryview, buffer or byte string

Other info: The same exact version of my program (pulled from my git repo) works on MacOSX and on Alpine Linux (Docker image).

Any idea what the issue is ?

Thanks,
Arno
The exact

[Lukasa]

Yes, almost certainly you're sending a unicode string in your data keyword argument. You only see it on some platforms because most of the time it can be tolerated. What is your data keyword argument?

[Arno0x]

What the app is sending as a dataargument is always a byte array of AES256 encrypted data, so not really Unicode :-(

Unless the "data" in the context of SSL.py also includes the HTTP headers I'm providing to "requests". So I checked all my headers value and name with the following code:

for headerName, headerValue in headers.items():
				try:
					headerValue.decode('ascii')
					headerName.decode('ascii')
				except UnicodeDecodeError:
				    print "Header {} is NOT a ascii-encoded unicode string".format(headerValue)
				else:
				    print "Header {} may have been an ascii-encoded unicode string".format(headerValue)

And running this code seems to prove that everything is properly ASCII encoded.

The point is, the exact same app worked fine just a few days before, I made stricly no changes to my code but in the meantime I installed Impacket which probably comes with a bunch of python libs install/update which I'm suspecting causes the issue since this is the only thing I can see may have changed...

[Lukasa]

@Arno0x To be clear, the appearance of the problem was caused by you installing PyOpenSSL. This is because we preferentially use PyOpenSSL if it's available, but PyOpenSSL is very strict about the data that is handed to it.

The fact that you can safely call decode on the headers is not sufficient for a test. The headers must be native strings on Python: that means you should manually encode them if they are unicode strings.

[Arno0x]

@Lukasa thank you so much for helping me on this issue.
Ok, so I understand that I'm now using an installed PyOpenSSL lib on this system, rather than the PyOpenSSL that comes embedded with "requests", which explains the difference with my other systems.

If I get it right (provided I'm using Python2): all strings in my code are of type 'str' by default and I guess that's what you call native string (is it?). So it looks like all my headers (name or value) are already native python string. Still, just in case, I tried to change my code to add .encode('ascii')on all my header strings => no luck, still the same error.

So I've reduced my testing scenario down to the basic following test in python command line:

root@kali:~/Tools/DBC2# python
Python 2.7.12+ (default, Sep  1 2016, 20:27:38) 
[GCC 6.2.0 20160822] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> from lib.crypto import Crypto
>>>
>>> headers={"Authorization": "Bearer xxxxxMyDropBoxTokenxxxxx", "Content-Type": "application/octet-stream", "Dropbox-API-Arg":'{"path":"/test","mode":"overwrite"}'}
>>>
>>> requests.post("https://content.dropboxapi.com/2/files/upload", headers=headers, data='test')
<Response [200]>
>>>
>>> requests.post("https://content.dropboxapi.com/2/files/upload", headers=headers, data=Crypto.xor(bytearray('test'),'whateverkey'))
[...]
File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/contrib/pyopenssl.py", line 292, in sendall
    sent = self._send_until_done(data[total_sent:total_sent + SSL_WRITE_BLOCKSIZE])
  File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/contrib/pyopenssl.py", line 281, in _send_until_done
    return self.connection.send(data)
  File "/usr/lib/python2.7/dist-packages/OpenSSL/SSL.py", line 1251, in send
    raise TypeError("data must be a memoryview, buffer or byte string")
TypeError: data must be a memoryview, buffer or byte string

So it turns out not to be a header issue, but rather the data itself returned from my Crypto.xor function, which is very basic:

def xor(cls, data, key):
		l = len(key)
		keyAsInt = map(ord, key)
		return bytearray((
		    (data[i] ^ keyAsInt[i % l]) for i in range(0,len(data))
		))

I hope you'll get to see what's wrong in this piece of code that PyOpenSSL doesn't like... thanks again for your help.

[Lukasa]

So it looks like PyOpenSSL doesn't accept bytearrays: only buffers or bytes. You can work around this in the short term by calling bytes on the bytearray.

However, I'd call this a PyOpenSSL bug: there's no reason not to accept bytearrays here. So I recommend opening a bug report on the PyOpenSSL repository as well. =)

[Arno0x]

SPOTTED ! :-)
It works !

Thank you so much for your help. I'll certainly open an issue on the PyOpenSSL as well.

Cheers.

[Lukasa]

No problem. =)