New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bug report #1397

Closed
kennethreitz opened this Issue May 30, 2013 · 10 comments

Comments

Projects
None yet
5 participants
@Lukasa

This comment has been minimized.

Show comment
Hide comment
@Lukasa

Lukasa May 30, 2013

Member

We've seen this before, here. The short of it is that the remote server is claiming that the data is chunked, but it isn't.

Member

Lukasa commented May 30, 2013

We've seen this before, here. The short of it is that the remote server is claiming that the data is chunked, but it isn't.

@Anorov

This comment has been minimized.

Show comment
Hide comment
@Anorov

Anorov Jun 6, 2013

@Lukasa Perhaps requests could wrap the exception and display that? The server specified a chunked Transfer-Encoding response but did not send chunked data. or something along those lines.

Anorov commented Jun 6, 2013

@Lukasa Perhaps requests could wrap the exception and display that? The server specified a chunked Transfer-Encoding response but did not send chunked data. or something along those lines.

@Lukasa

This comment has been minimized.

Show comment
Hide comment
@Lukasa

Lukasa Jun 7, 2013

Member

It's possible. I'd need to reread the httplib code and confirm that we can't raise IncompleteRead for any other reason though.

Member

Lukasa commented Jun 7, 2013

It's possible. I'd need to reread the httplib code and confirm that we can't raise IncompleteRead for any other reason though.

@kennethreitz

This comment has been minimized.

Show comment
Hide comment
@kennethreitz

kennethreitz Jun 27, 2013

Member

We should be able to work around this, I think.

Member

kennethreitz commented Jun 27, 2013

We should be able to work around this, I think.

@sigmavirus24

This comment has been minimized.

Show comment
Hide comment
@sigmavirus24

sigmavirus24 Jun 30, 2013

Member

This is causing issues and is being masked by an SSL error. I'm 90% certain it is the cause of this: https://twitter.com/bkmontgomery/status/351090132004257792

Member

sigmavirus24 commented Jun 30, 2013

This is causing issues and is being masked by an SSL error. I'm 90% certain it is the cause of this: https://twitter.com/bkmontgomery/status/351090132004257792

@Lukasa

This comment has been minimized.

Show comment
Hide comment
@Lukasa

Lukasa Jun 30, 2013

Member

I don't think I have a good picture in my head of how we can get httplib to do the correct read here, when the server is sending incorrect headers.

Member

Lukasa commented Jun 30, 2013

I don't think I have a good picture in my head of how we can get httplib to do the correct read here, when the server is sending incorrect headers.

@kracekumar

This comment has been minimized.

Show comment
Hide comment
@kracekumar

kracekumar Jul 25, 2013

Contributor
#! /usr/bin/env python
import requests


sites = [
    "http://www.tusseymountaintitans.com",
    "http://www.abbottpanthers.com",
    "http://www.spanishmoms.com",
    "http://www.long-island-storage.com",
    "http://www.cupertinohelpwanted.com",
    "http://www.hoffmanestateshawks.com",
    "http://www.brothermartincrusaders.com",
    "http://www.1-800-printer.com",
    "http://www.impiretickets.com",
    "http://www.gdickinson.com",
    "http://www.forensicsline.com",
    "http://www.gardeningtime.com",
    "http://www.ecollegetennis.com",
    "http://www.milacasaints.com",
    "http://www.bartoninsuranceagency.com",
    "http://www.djnatural.com",
    "http://www.containers2000.com",
    "http://www.indiancreektimberwolves.com",
    "http://www.athenswarriors.com",
    "http://www.logansportcats.com",
    "http://www.osani.com",
    "http://www.xn--sammler-brse-djb.com",
    "http://www.800usahealth.com",
    "http://www.wealth-wise.com",
    "http://www.foothillmustangs.com",
    "http://www.manasquanbigblue.com",
    "http://www.bilhetos.com",
    "http://www.atlantahomesteam.com",
    "http://www.foxcitiessatellite.com",
    #"http://www.chargersmail.com", commenting because interpreter hangs in here
    "http://www.fighterplace.com"
]


if __name__ == "__main__":
    print("Total sites: %d " % (len(sites)))
    for index, site in enumerate(sites):
        r = requests.get(site, stream=True)
        print(index, site)
        if 'transfer-encoding' in r.headers:
            print(r.headers['transfer-encoding'], r.headers['Server'], r.headers['X-AspNet-Version'], r.headers["x-server"])

output

➜  snippets  python requests_bug_1397.py
Total sites: 30
(0, 'http://www.tusseymountaintitans.com')
('chunked', 'Microsoft-IIS/6.0', '2.0.50727', 'sjl06')
(1, 'http://www.abbottpanthers.com')
('chunked', 'Microsoft-IIS/6.0', '2.0.50727', 'sjl04')
(2, 'http://www.spanishmoms.com')
('chunked', 'Microsoft-IIS/6.0', '2.0.50727', 'sjl04')
(3, 'http://www.long-island-storage.com')
('chunked', 'Microsoft-IIS/6.0', '2.0.50727', 'sjl05')
(4, 'http://www.cupertinohelpwanted.com')
('chunked', 'Microsoft-IIS/6.0', '2.0.50727', 'sjl05')
(5, 'http://www.hoffmanestateshawks.com')
('chunked', 'Microsoft-IIS/6.0', '2.0.50727', 'ash02')
(6, 'http://www.brothermartincrusaders.com')
('chunked', 'Microsoft-IIS/6.0', '2.0.50727', 'sjl06')
(7, 'http://www.1-800-printer.com')
('chunked', 'Microsoft-IIS/6.0', '2.0.50727', 'ash01')
(8, 'http://www.impiretickets.com')
('chunked', 'Microsoft-IIS/6.0', '2.0.50727', 'ash02')
(9, 'http://www.gdickinson.com')
('chunked', 'Microsoft-IIS/6.0', '2.0.50727', 'sjl03')
(10, 'http://www.forensicsline.com')
('chunked', 'Microsoft-IIS/6.0', '2.0.50727', 'sjl05')
(11, 'http://www.gardeningtime.com')
('chunked', 'Microsoft-IIS/6.0', '2.0.50727', 'sjl01')
(12, 'http://www.ecollegetennis.com')
('chunked', 'Microsoft-IIS/6.0', '2.0.50727', 'sjl05')
(13, 'http://www.milacasaints.com')
('chunked', 'Microsoft-IIS/6.0', '2.0.50727', 'sjl02')
(14, 'http://www.bartoninsuranceagency.com')
('chunked', 'Microsoft-IIS/6.0', '2.0.50727', 'ash02')
(15, 'http://www.djnatural.com')
('chunked', 'Microsoft-IIS/6.0', '2.0.50727', 'ash05')
(16, 'http://www.containers2000.com')
('chunked', 'Microsoft-IIS/6.0', '2.0.50727', 'sjl08')
(17, 'http://www.indiancreektimberwolves.com')
('chunked', 'Microsoft-IIS/6.0', '2.0.50727', 'ash05')
(18, 'http://www.athenswarriors.com')
('chunked', 'Microsoft-IIS/6.0', '2.0.50727', 'sjl05')
(19, 'http://www.logansportcats.com')
('chunked', 'Microsoft-IIS/6.0', '2.0.50727', 'sjl05')
(20, 'http://www.osani.com')
('chunked', 'Microsoft-IIS/6.0', '2.0.50727', 'sjl04')
(21, 'http://www.xn--sammler-brse-djb.com')
('chunked', 'Microsoft-IIS/6.0', '2.0.50727', 'ash02')
(22, 'http://www.800usahealth.com')
('chunked', 'Microsoft-IIS/6.0', '2.0.50727', 'sjl05')
(23, 'http://www.wealth-wise.com')
('chunked', 'Microsoft-IIS/6.0', '2.0.50727', 'sjl02')
(24, 'http://www.foothillmustangs.com')
('chunked', 'Microsoft-IIS/6.0', '2.0.50727', 'sjl02')
(25, 'http://www.manasquanbigblue.com')
('chunked', 'Microsoft-IIS/6.0', '2.0.50727', 'ash05')
(26, 'http://www.bilhetos.com')
('chunked', 'Microsoft-IIS/6.0', '2.0.50727', 'sjl05')
(27, 'http://www.atlantahomesteam.com')
('chunked', 'Microsoft-IIS/6.0', '2.0.50727', 'sjl03')
(28, 'http://www.foxcitiessatellite.com')
('chunked', 'Microsoft-IIS/6.0', '2.0.50727', 'sjl03')
(29, 'http://www.fighterplace.com')

All the servers are running Microsoft-IIS/6.0 & ASP Version 2.0.5.0727. Latest version of ASP is 4.0 (Ref). Similar issue can be found here.

I suspect this problem is related to specific ASP.Net version. If we can find a site with same issue with different ASP version and IIS version worth investigating.

Contributor

kracekumar commented Jul 25, 2013

#! /usr/bin/env python
import requests


sites = [
    "http://www.tusseymountaintitans.com",
    "http://www.abbottpanthers.com",
    "http://www.spanishmoms.com",
    "http://www.long-island-storage.com",
    "http://www.cupertinohelpwanted.com",
    "http://www.hoffmanestateshawks.com",
    "http://www.brothermartincrusaders.com",
    "http://www.1-800-printer.com",
    "http://www.impiretickets.com",
    "http://www.gdickinson.com",
    "http://www.forensicsline.com",
    "http://www.gardeningtime.com",
    "http://www.ecollegetennis.com",
    "http://www.milacasaints.com",
    "http://www.bartoninsuranceagency.com",
    "http://www.djnatural.com",
    "http://www.containers2000.com",
    "http://www.indiancreektimberwolves.com",
    "http://www.athenswarriors.com",
    "http://www.logansportcats.com",
    "http://www.osani.com",
    "http://www.xn--sammler-brse-djb.com",
    "http://www.800usahealth.com",
    "http://www.wealth-wise.com",
    "http://www.foothillmustangs.com",
    "http://www.manasquanbigblue.com",
    "http://www.bilhetos.com",
    "http://www.atlantahomesteam.com",
    "http://www.foxcitiessatellite.com",
    #"http://www.chargersmail.com", commenting because interpreter hangs in here
    "http://www.fighterplace.com"
]


if __name__ == "__main__":
    print("Total sites: %d " % (len(sites)))
    for index, site in enumerate(sites):
        r = requests.get(site, stream=True)
        print(index, site)
        if 'transfer-encoding' in r.headers:
            print(r.headers['transfer-encoding'], r.headers['Server'], r.headers['X-AspNet-Version'], r.headers["x-server"])

output

➜  snippets  python requests_bug_1397.py
Total sites: 30
(0, 'http://www.tusseymountaintitans.com')
('chunked', 'Microsoft-IIS/6.0', '2.0.50727', 'sjl06')
(1, 'http://www.abbottpanthers.com')
('chunked', 'Microsoft-IIS/6.0', '2.0.50727', 'sjl04')
(2, 'http://www.spanishmoms.com')
('chunked', 'Microsoft-IIS/6.0', '2.0.50727', 'sjl04')
(3, 'http://www.long-island-storage.com')
('chunked', 'Microsoft-IIS/6.0', '2.0.50727', 'sjl05')
(4, 'http://www.cupertinohelpwanted.com')
('chunked', 'Microsoft-IIS/6.0', '2.0.50727', 'sjl05')
(5, 'http://www.hoffmanestateshawks.com')
('chunked', 'Microsoft-IIS/6.0', '2.0.50727', 'ash02')
(6, 'http://www.brothermartincrusaders.com')
('chunked', 'Microsoft-IIS/6.0', '2.0.50727', 'sjl06')
(7, 'http://www.1-800-printer.com')
('chunked', 'Microsoft-IIS/6.0', '2.0.50727', 'ash01')
(8, 'http://www.impiretickets.com')
('chunked', 'Microsoft-IIS/6.0', '2.0.50727', 'ash02')
(9, 'http://www.gdickinson.com')
('chunked', 'Microsoft-IIS/6.0', '2.0.50727', 'sjl03')
(10, 'http://www.forensicsline.com')
('chunked', 'Microsoft-IIS/6.0', '2.0.50727', 'sjl05')
(11, 'http://www.gardeningtime.com')
('chunked', 'Microsoft-IIS/6.0', '2.0.50727', 'sjl01')
(12, 'http://www.ecollegetennis.com')
('chunked', 'Microsoft-IIS/6.0', '2.0.50727', 'sjl05')
(13, 'http://www.milacasaints.com')
('chunked', 'Microsoft-IIS/6.0', '2.0.50727', 'sjl02')
(14, 'http://www.bartoninsuranceagency.com')
('chunked', 'Microsoft-IIS/6.0', '2.0.50727', 'ash02')
(15, 'http://www.djnatural.com')
('chunked', 'Microsoft-IIS/6.0', '2.0.50727', 'ash05')
(16, 'http://www.containers2000.com')
('chunked', 'Microsoft-IIS/6.0', '2.0.50727', 'sjl08')
(17, 'http://www.indiancreektimberwolves.com')
('chunked', 'Microsoft-IIS/6.0', '2.0.50727', 'ash05')
(18, 'http://www.athenswarriors.com')
('chunked', 'Microsoft-IIS/6.0', '2.0.50727', 'sjl05')
(19, 'http://www.logansportcats.com')
('chunked', 'Microsoft-IIS/6.0', '2.0.50727', 'sjl05')
(20, 'http://www.osani.com')
('chunked', 'Microsoft-IIS/6.0', '2.0.50727', 'sjl04')
(21, 'http://www.xn--sammler-brse-djb.com')
('chunked', 'Microsoft-IIS/6.0', '2.0.50727', 'ash02')
(22, 'http://www.800usahealth.com')
('chunked', 'Microsoft-IIS/6.0', '2.0.50727', 'sjl05')
(23, 'http://www.wealth-wise.com')
('chunked', 'Microsoft-IIS/6.0', '2.0.50727', 'sjl02')
(24, 'http://www.foothillmustangs.com')
('chunked', 'Microsoft-IIS/6.0', '2.0.50727', 'sjl02')
(25, 'http://www.manasquanbigblue.com')
('chunked', 'Microsoft-IIS/6.0', '2.0.50727', 'ash05')
(26, 'http://www.bilhetos.com')
('chunked', 'Microsoft-IIS/6.0', '2.0.50727', 'sjl05')
(27, 'http://www.atlantahomesteam.com')
('chunked', 'Microsoft-IIS/6.0', '2.0.50727', 'sjl03')
(28, 'http://www.foxcitiessatellite.com')
('chunked', 'Microsoft-IIS/6.0', '2.0.50727', 'sjl03')
(29, 'http://www.fighterplace.com')

All the servers are running Microsoft-IIS/6.0 & ASP Version 2.0.5.0727. Latest version of ASP is 4.0 (Ref). Similar issue can be found here.

I suspect this problem is related to specific ASP.Net version. If we can find a site with same issue with different ASP version and IIS version worth investigating.

@Lukasa

This comment has been minimized.

Show comment
Hide comment
@Lukasa

Lukasa Jul 25, 2013

Member

Interesting! This is some awesome detective work @kracekumar! Does look like a very specific bug here, but I'm still not convinced we can neatly work our way around it.

Member

Lukasa commented Jul 25, 2013

Interesting! This is some awesome detective work @kracekumar! Does look like a very specific bug here, but I'm still not convinced we can neatly work our way around it.

@Lukasa

This comment has been minimized.

Show comment
Hide comment
@Lukasa

Lukasa Jul 28, 2013

Member

So, we can't work around this, at least not on all platforms. Here's the relevant code from httplib in our supported versions of Python 2.X:

try:
    chunk_left = int(line, 16)
except ValueError:
    # close the connection as protocol synchronisation is
    # probably lost
    self.close()
    raise IncompleteRead(''.join(value))

At the point the exception is raised the connection has been closed. Additionally, the read data is thrown away, so we can't pull it out. Working around this would require patching the standard library or using a forked version of httplib in urllib3.

It looks like our best shot is actually just to wrap this exception: maybe a requests.ChunkedEncodingError?

Member

Lukasa commented Jul 28, 2013

So, we can't work around this, at least not on all platforms. Here's the relevant code from httplib in our supported versions of Python 2.X:

try:
    chunk_left = int(line, 16)
except ValueError:
    # close the connection as protocol synchronisation is
    # probably lost
    self.close()
    raise IncompleteRead(''.join(value))

At the point the exception is raised the connection has been closed. Additionally, the read data is thrown away, so we can't pull it out. Working around this would require patching the standard library or using a forked version of httplib in urllib3.

It looks like our best shot is actually just to wrap this exception: maybe a requests.ChunkedEncodingError?

@Lukasa

This comment has been minimized.

Show comment
Hide comment
@Lukasa

Lukasa Aug 17, 2013

Member

Mitigated by #1498.

Member

Lukasa commented Aug 17, 2013

Mitigated by #1498.

@Lukasa Lukasa closed this Aug 17, 2013

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment