Setting u'POST' in requests.models.PreparedRequest.prepare_method breaks attaching files.

[etscrivner]

Running Python 2.7 on Ubuntu 12.04 with requests 1.0.2

The following line of code which performs an HTTP POST request with an attached image file breaks (worked in previous versions of requests):

[1] requests.post(url, data=data, files={'file': ('test.jpg', open('/tmp/test123.jpg', 'rb'))}, params=params)
...
UnicodeDecodeError: 'ascii' codec can't decode byte 0xff in position 1066: ordinal not in range(128)

Running this through iPDB and stepping down through the requests API the problem seems directly related to issue #444 - If you're attaching a file then it will be encoded in the final HTTP body as a series of bytes.

The problem is prepare_method in models.py:223 - which, regardless of the version of Python used, tries to unicode encode the HTTP method name in all situations. This causes urllib3 trouble when trying to come up with a final encoding. Since the file serialization behavior is handled in requests and creates HTTP requests with inconsistent encodings in some situations this seems like a bug to me.

A potential solution would be to check for attached files and leave the encoding alone when preparing a request.

[kennethreitz]

Thanks for the detailed forensics! I'll try to get a fix out for this today.

Unless you want to send a pull requests of course :)

[myzhan]

As requests upgraded to 1.0.2. I came across a similar problem using requests.post().

    requests.post(url, data=req)

It worked fine when I used version 0.14.2.

Here is my traceback.

  File "/home/zhanqp/api_test/lib/python2.7/site-packages/requests/api.py", line 81, in post
    return request('post', url, data=data, **kwargs)
  File "/home/zhanqp/api_test/lib/python2.7/site-packages/requests/api.py", line 38, in request
    return session.request(method=method, url=url, **kwargs)
  File "/home/zhanqp/api_test/lib/python2.7/site-packages/requests/sessions.py", line 266, in request
    resp = self.send(prep, stream=stream, timeout=timeout, verify=verify, cert=cert, proxies=proxies)
  File "/home/zhanqp/api_test/lib/python2.7/site-packages/requests/sessions.py", line 357, in send
    r = adapter.send(request, **kwargs)
  File "/home/zhanqp/api_test/lib/python2.7/site-packages/requests/adapters.py", line 156, in send
    timeout=timeout,
  File "/home/zhanqp/api_test/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 422, in urlopen
    body=body, headers=headers)
  File "/home/zhanqp/api_test/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 274, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/user/local/python/lib/python2.7/httplib.py", line 955, in request
    self._send_request(method, url, body, headers)
  File "/user/local/python/lib/python2.7/httplib.py", line 989, in _send_request
    self.endheaders(body)
  File "/user/local/python/lib/python2.7/httplib.py", line 951, in endheaders
    self._send_output(message_body)
  File "/user/local/python/lib/python2.7/httplib.py", line 809, in _send_output
    msg += message_body
UnicodeDecodeError: 'ascii' codec can't decode byte 0xef in position 6: ordinal not in range(128)

[myzhan]

Tried to reproduce this problem.

coding:utf8

import requests
from utils import encode

url = "http://www.google.com"
req = encode("hi")

req = requests.post(url, data=req)
print req.text

encode is a C function wrapped by ctypes, which encodes a string into byte stream. Curiously, type(req) is str.

I think it is my own encode function that triggered this problem. But It worked fine when I used version 0.14.2.

[sigmavirus24]

@myzhan this should be fixed in 1.0.3, try upgrading to that. On a side note, if you wouldn't mind posting the encode function's code, I'd be interested in playing with it.

[myzhan]

@sigmavirus24 Cool!!! 1.0.3 works fine with my code, thx for your great help. But encode function's code is used for commercial encrypted file, I can't post it, sorry.

[sigmavirus24]

Ah no worries then. I didn't do much to help though. Cheers