Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Content-Length header is duplicated on Python3 #1335

Closed
valyagolev opened this issue Apr 30, 2013 · 6 comments
Closed

Content-Length header is duplicated on Python3 #1335

valyagolev opened this issue Apr 30, 2013 · 6 comments

Comments

@valyagolev
Copy link

Content-Length header is duplicated if specified on Python3, leading to 400 responses:

Python 3.3.0 (default, Apr 30 2013, 06:45:58)
[GCC 4.2.1 Compatible Apple Clang 4.0 ((tags/Apple/clang-421.0.60))] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> requests.post("http://httpbin.org/post", headers={"Content-Length": 0})
<Response [400]>

We need to go deeper:

>>> import six
>>> six.moves.http_client.HTTPConnection.debuglevel = 1
>>> requests.post("http://httpbin.org/post", headers={"Content-Length": 0})
send: b'POST /post HTTP/1.1\r\nHost: httpbin.org\r\nAccept-Encoding: identity\r\nContent-Length: 0\r\nUser-Agent: python-requests/1.2.0 CPython/3.3.0 Darwin/12.3.0\r\nContent-Length: 0\r\nAccept: */*\r\nAccept-Encoding: gzip, deflate, compress\r\n\r\n'
reply: 'HTTP/1.1 400 BAD_REQUEST\r\n'
header: Content-Length header: Connection <Response [400]>

That's what happens:

POST /post HTTP/1.1
Host: httpbin.org
Accept-Encoding: identity
Content-Length: 0
User-Agent: python-requests/1.2.0 CPython/3.3.0 Darwin/12.3.0
Content-Length: 0
Accept: */*
Accept-Encoding: gzip, deflate, compress

Two content-lengths, one 400 response :-(
On Python2 everything's OK.

@Lukasa
Copy link
Member

Lukasa commented Apr 30, 2013

Ok, so I've worked this out.

In PreparedRequest.prepare_headers(), we encode all the header keys as ASCII (the correctness of that is a discussion for another time). This includes the Content-Length you passed in on your original request. Later on, we call PreparedRequest.prepare_auth(), which calls PreparedRequest.prepare_content_length(). All fine.

However, prepare_content_length() sets its content length as a native string, not an encoded one. This is a simple bug, so I'm going to mark this contributor friendly. If no-one fixes it before this evening I'll open a quick PR.

@Lukasa
Copy link
Member

Lukasa commented Apr 30, 2013

Ugh, crap. Turns out fixing this breaks some other stuff. That's not ideal.

@GP89
Copy link

GP89 commented Apr 30, 2013

I noticed something today that's possibly related.

If I specify a Content-Length header, requests ignores it and instead uses one it takes from the file instead. (py2). Maybe this is intended but as I understood user specified headers are supposed to take precedence over implicit addition of headers? This has caused me a bit of a problem as I can't return anything over sys.maxint from __len__ of a file object as I get this traceback, so I can't upload files over 2GB using this method.

  File "/Users/paul/Source/Python/PythonApp/upload.py", line 911, in _gUpload
    response = requests.put(signed_url, file_iterator, headers=headers, timeout=settings.UPLOAD_TIMEOUT)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/requests/api.py", line 98, in put
    return request('put', url, data=data, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/requests/api.py", line 44, in request
    return session.request(method=method, url=url, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/requests/sessions.py", line 276, in request
    prep = req.prepare()
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/requests/models.py", line 224, in prepare
    p.prepare_body(self.data, self.files)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/requests/models.py", line 349, in prepare_body
    length = super_len(data)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/requests/utils.py", line 45, in super_len
    return len(o)
OverflowError: long int too large to convert to int

@Lukasa
Copy link
Member

Lukasa commented Jun 8, 2013

This bug should be resolved by #1338, so I'm going to close this issue to centralise discussion over there. Thanks so much for pointing it out!

@Lukasa Lukasa closed this as completed Jun 8, 2013
@valyagolev
Copy link
Author

@Lukasa thank you! Btw, you accidentally pushed a merge conflict :(

@Lukasa
Copy link
Member

Lukasa commented Jun 8, 2013

Oh man, so I did! I swear I fixed that! Awkward. =( Thanks for pointing it out. 🍰

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Sep 9, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

3 participants