content-length header value not available after get

[medington]

I'm unable to get content-length reliably with the requests library whereas urllib2 seems to retrieve the data consistently. Not sure if additional code is required or if the URL I'm hitting is problematic.

Run this script and compare the results. I consistently get None from requests. Using the other test_url I consistently get a result from both. What am I missing?

# Test which demonstrates issues with requests get of content-length header

# This URL seems problematic (it's a several minute download)
test_url = "http://standards-oui.ieee.org/oui.txt"
# test_url = 'https://github.com/kennethreitz/requests/tarball/master'

def requests_getlen(url):
    import requests
    r = requests.get(url, stream=True)
    return str(r.headers.get('content-length'))

def urllib_getlen(url):
    import urllib2
    req = urllib2.urlopen(url)
    return str(req.info().getheader('content-length'))

print 'requests content-length: ' + requests_getlen(test_url)
print 'urllib2  content-length: ' + urllib_getlen(test_url)

[sigmavirus24]

Questions belong on StackOverflow.

That said, have you checked to see if you're getting a chunked transfer-encoding?

[medington]

Sorry, my bad. I thought it might have been a legitimate issue since the results were inconsistent between runs even using the alternate test_url value pointing at the tarball file on GitHub (that one would sometimes have content-length included, but not always). I'll figure out the appropriate question(s) and post on StackOverflow.

In the meantime, after some poking around with Charles, I determined that the difference stems from the fact that the Requests library specifies gzip, deflate as acceptable encodings by default and urllib2 specifies identity by default.

Here is code which worked for this URL with requests to obtain content-length header:

r = requests.head(url, headers={'Accept-Encoding': 'identity'}, stream=True)
return str(r.headers.get('content-length'))

[sigmavirus24]

the results were inconsistent between runs even using the alternate test_url value pointing at the tarball file on GitHub (that one would sometimes have content-length included, but not always)

If you look at the raw data that we're sending we're sending the exact same time (some of the header order may be different but that's to be expected). This would seem to be a problem with GitHub, but as long as they're also sending Transfer-Encoding: chunked then they aren't required to send a content-length. Not everything on the internet has a content-length header associated with it.