New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Wrap IncompleteRead in ChunkedEncodingError. #1498

Merged
merged 1 commit into from Aug 1, 2013

Conversation

Projects
None yet
3 participants
@Lukasa
Member

Lukasa commented Jul 28, 2013

This should improve the user experience from #1397.

@sigmavirus24, @kennethreitz: I'm not really happy with the nested try...except blocks here. Suggestions for a better style? Nothing leaps out at me.

kennethreitz added a commit that referenced this pull request Aug 1, 2013

Merge pull request #1498 from Lukasa/incompleteread
Wrap IncompleteRead in ChunkedEncodingError.

@kennethreitz kennethreitz merged commit d5a8816 into requests:master Aug 1, 2013

1 check passed

default The Travis CI build passed
Details

@Lukasa Lukasa referenced this pull request Aug 17, 2013

Closed

bug report #1397

@keyz182 keyz182 referenced this pull request Nov 22, 2013

Open

ChunkedEncodingError #288

@johaven

This comment has been minimized.

Show comment
Hide comment
@johaven

johaven May 18, 2016

I can't read r after catching error (there is an explanation ?)

try:
   r = requests.post(url, params=self.req_params, data=self.upload_handler)
except requests.exceptions.ChunkedEncodingError, e:
   # got: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))
   # it's a possible behaviour in my case and i want to catch it and read status code or json response
   app.logger(str(e))
   print r.status_code # or print r.json()
   # here my code is blocked indefinitely

johaven commented May 18, 2016

I can't read r after catching error (there is an explanation ?)

try:
   r = requests.post(url, params=self.req_params, data=self.upload_handler)
except requests.exceptions.ChunkedEncodingError, e:
   # got: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))
   # it's a possible behaviour in my case and i want to catch it and read status code or json response
   app.logger(str(e))
   print r.status_code # or print r.json()
   # here my code is blocked indefinitely
@Lukasa

This comment has been minimized.

Show comment
Hide comment
@Lukasa

Lukasa May 18, 2016

Member

@yoyoprs Your code definitely blocks on status_code? You won't be able to read any json response because the response is incomplete, but the status_code will be fine unless the server hasn't responded at all.

Member

Lukasa commented May 18, 2016

@yoyoprs Your code definitely blocks on status_code? You won't be able to read any json response because the response is incomplete, but the status_code will be fine unless the server hasn't responded at all.

@johaven

This comment has been minimized.

Show comment
Hide comment
@johaven

johaven May 18, 2016

@Lukasa To sum up, i send file with some parameters, on server side according to the request parameters i start to read file or return a json with code 200. UWSGI doesn't like when the body is not consumed and raise error (IncompleteRead, i deal with it).
Request returned by server is ok (200) :

[pid: 55942|app: 0|req: 2/3] x.x.x.x () {42 vars in 940 bytes} [Wed May 18 16:04:05 2016] GET /api/v1/apps/sync/diff?uuid=9ab08c9e-3632-4e24-bcb3-ab7c9414d75f&firstsync=0&type=2&filters=&report=0&path=%2Ftest&id=38504 => generated 165 bytes in 154 msecs (HTTP/1.1 200) 2 headers in 80 bytes

But i can't read status_code or json ...

johaven commented May 18, 2016

@Lukasa To sum up, i send file with some parameters, on server side according to the request parameters i start to read file or return a json with code 200. UWSGI doesn't like when the body is not consumed and raise error (IncompleteRead, i deal with it).
Request returned by server is ok (200) :

[pid: 55942|app: 0|req: 2/3] x.x.x.x () {42 vars in 940 bytes} [Wed May 18 16:04:05 2016] GET /api/v1/apps/sync/diff?uuid=9ab08c9e-3632-4e24-bcb3-ab7c9414d75f&firstsync=0&type=2&filters=&report=0&path=%2Ftest&id=38504 => generated 165 bytes in 154 msecs (HTTP/1.1 200) 2 headers in 80 bytes

But i can't read status_code or json ...

@Lukasa

This comment has been minimized.

Show comment
Hide comment
@Lukasa

Lukasa May 18, 2016

Member

Hang on, why is r in scope? With the code called the way you show it above, r should not even be in scope, which means you should get a NameError. Why don't you?

Member

Lukasa commented May 18, 2016

Hang on, why is r in scope? With the code called the way you show it above, r should not even be in scope, which means you should get a NameError. Why don't you?

@johaven

This comment has been minimized.

Show comment
Hide comment
@johaven

johaven May 18, 2016

@Lukasa I just noticed that the variable r is no longer found in the namespace
No NameError !

tcpdump (no reset):

18:10:35.667569 IP client.50122 > server.443: Flags [S], seq 865485447, win 65535, options [mss 1460,nop,wscale 5,nop,nop,TS val 379018341 ecr 0,sackOK,eol], length 0
18:10:35.667623 IP server.443 > client.50122: Flags [S.], seq 4232375148, ack 865485448, win 28960, options [mss 1460,sackOK,TS val 2160547584 ecr 379018341,nop,wscale 7], length 0
18:10:35.681804 IP client.50122 > server.443: Flags [.], ack 1, win 4117, options [nop,nop,TS val 379018353 ecr 2160547584], length 0
18:10:35.684066 IP client.50122 > server.443: Flags [P.], seq 1:102, ack 1, win 4117, options [nop,nop,TS val 379018354 ecr 2160547584], length 101
18:10:35.684101 IP server.443 > client.50122: Flags [.], ack 102, win 227, options [nop,nop,TS val 2160547601 ecr 379018354], length 0
18:10:35.687532 IP server.443 > client.50122: Flags [.], seq 1:2897, ack 102, win 227, options [nop,nop,TS val 2160547604 ecr 379018354], length 2896
18:10:35.687642 IP server.443 > client.50122: Flags [P.], seq 2897:3095, ack 102, win 227, options [nop,nop,TS val 2160547604 ecr 379018354], length 198
18:10:35.702305 IP client.50122 > server.443: Flags [.], ack 2897, win 4050, options [nop,nop,TS val 379018373 ecr 2160547604], length 0
18:10:35.703812 IP client.50122 > server.443: Flags [.], ack 3095, win 4044, options [nop,nop,TS val 379018373 ecr 2160547604], length 0
18:10:35.714584 IP client.50122 > server.443: Flags [P.], seq 102:300, ack 3095, win 4096, options [nop,nop,TS val 379018383 ecr 2160547604], length 198
18:10:35.716330 IP server.443 > client.50122: Flags [P.], seq 3095:3361, ack 300, win 235, options [nop,nop,TS val 2160547633 ecr 379018383], length 266
18:10:35.729816 IP client.50122 > server.443: Flags [.], ack 3361, win 4087, options [nop,nop,TS val 379018398 ecr 2160547633], length 0
18:10:35.734595 IP client.50122 > server.443: Flags [P.], seq 300:726, ack 3361, win 4096, options [nop,nop,TS val 379018398 ecr 2160547633], length 426
18:10:35.773934 IP server.443 > client.50122: Flags [.], ack 726, win 243, options [nop,nop,TS val 2160547691 ecr 379018398], length 0
18:10:35.845389 IP server.443 > client.50122: Flags [P.], seq 3361:3766, ack 726, win 243, options [nop,nop,TS val 2160547762 ecr 379018398], length 405
18:10:35.859043 IP client.50122 > server.443: Flags [.], ack 3766, win 4083, options [nop,nop,TS val 379018524 ecr 2160547762], length 0
18:10:35.901352 IP client.50122 > server.443: Flags [P.], seq 726:1264, ack 3766, win 4096, options [nop,nop,TS val 379018561 ecr 2160547762], length 538
18:10:35.901394 IP server.443 > client.50122: Flags [.], ack 1264, win 252, options [nop,nop,TS val 2160547818 ecr 379018561], length 0
18:10:36.050250 IP server.443 > client.50122: Flags [P.], seq 3766:4235, ack 1264, win 252, options [nop,nop,TS val 2160547967 ecr 379018561], length 469
18:10:36.064314 IP client.50122 > server.443: Flags [.], ack 4235, win 4081, options [nop,nop,TS val 379018727 ecr 2160547967], length 0
18:10:36.935368 IP client.50122 > server.443: Flags [P.], seq 1264:1882, ack 4235, win 4096, options [nop,nop,TS val 379019590 ecr 2160547967], length 618
18:10:36.935408 IP server.443 > client.50122: Flags [.], ack 1882, win 262, options [nop,nop,TS val 2160548852 ecr 379019590], length 0
18:10:36.937074 IP client.50122 > server.443: Flags [P.], seq 1882:1956, ack 4235, win 4096, options [nop,nop,TS val 379019590 ecr 2160547967], length 74
18:10:36.937105 IP server.443 > client.50122: Flags [.], ack 1956, win 262, options [nop,nop,TS val 2160548854 ecr 379019590], length 0
18:10:37.029133 IP server.443 > client.50122: Flags [P.], seq 4235:4480, ack 1956, win 262, options [nop,nop,TS val 2160548946 ecr 379019590], length 245
18:10:37.029148 IP server.443 > client.50122: Flags [F.], seq 4480, ack 1956, win 262, options [nop,nop,TS val 2160548946 ecr 379019590], length 0
18:10:37.042577 IP client.50122 > server.443: Flags [.], ack 4480, win 4088, options [nop,nop,TS val 379019703 ecr 2160548946], length 0
18:10:37.044078 IP client.50122 > server.443: Flags [.], ack 4481, win 4096, options [nop,nop,TS val 379019703 ecr 2160548946], length 0
18:10:37.045338 IP client.50122 > server.443: Flags [F.], seq 1956, ack 4481, win 4096, options [nop,nop,TS val 379019704 ecr 2160548946], length 0
18:10:37.045369 IP server.443 > client.50122: Flags [.], ack 1957, win 262, options [nop,nop,TS val 2160548962 ecr 379019704], length 0

traceback :

Traceback (most recent call last):
  File "/my_app/handlers/propagate.py", line 445, in _upload
    up.start()
  File "/my_app/handlers/transfer.py", line 239, in start
    params=self.req_params, data=self.upload_handler)
  File "/my_app/handlers/http.py", line 26, in wrapped
    return f(self, *args, **kwargs)
  File "/my_app/handlers/http.py", line 108, in request_post
    timeout=(self.connect_timeout, self.read_timeout))
  File "/my_app/.virtualenvs/sync-in-app/lib/python2.7/site-packages/requests/sessions.py", line 511, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/my_app/.virtualenvs/sync-in-app/lib/python2.7/site-packages/requests/sessions.py", line 468, in request
    resp = self.send(prep, **send_kwargs)
  File "/my_app/.virtualenvs/sync-in-app/lib/python2.7/site-packages/requests/sessions.py", line 608, in send
    r.content
  File "/my_app/.virtualenvs/sync-in-app/lib/python2.7/site-packages/requests/models.py", line 737, in content
    self._content = bytes().join(self.iter_content(CONTENT_CHUNK_SIZE)) or bytes()
  File "/my_app/.virtualenvs/sync-in-app/lib/python2.7/site-packages/requests/models.py", line 663, in generate
    raise ChunkedEncodingError(e)
ChunkedEncodingError: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))

johaven commented May 18, 2016

@Lukasa I just noticed that the variable r is no longer found in the namespace
No NameError !

tcpdump (no reset):

18:10:35.667569 IP client.50122 > server.443: Flags [S], seq 865485447, win 65535, options [mss 1460,nop,wscale 5,nop,nop,TS val 379018341 ecr 0,sackOK,eol], length 0
18:10:35.667623 IP server.443 > client.50122: Flags [S.], seq 4232375148, ack 865485448, win 28960, options [mss 1460,sackOK,TS val 2160547584 ecr 379018341,nop,wscale 7], length 0
18:10:35.681804 IP client.50122 > server.443: Flags [.], ack 1, win 4117, options [nop,nop,TS val 379018353 ecr 2160547584], length 0
18:10:35.684066 IP client.50122 > server.443: Flags [P.], seq 1:102, ack 1, win 4117, options [nop,nop,TS val 379018354 ecr 2160547584], length 101
18:10:35.684101 IP server.443 > client.50122: Flags [.], ack 102, win 227, options [nop,nop,TS val 2160547601 ecr 379018354], length 0
18:10:35.687532 IP server.443 > client.50122: Flags [.], seq 1:2897, ack 102, win 227, options [nop,nop,TS val 2160547604 ecr 379018354], length 2896
18:10:35.687642 IP server.443 > client.50122: Flags [P.], seq 2897:3095, ack 102, win 227, options [nop,nop,TS val 2160547604 ecr 379018354], length 198
18:10:35.702305 IP client.50122 > server.443: Flags [.], ack 2897, win 4050, options [nop,nop,TS val 379018373 ecr 2160547604], length 0
18:10:35.703812 IP client.50122 > server.443: Flags [.], ack 3095, win 4044, options [nop,nop,TS val 379018373 ecr 2160547604], length 0
18:10:35.714584 IP client.50122 > server.443: Flags [P.], seq 102:300, ack 3095, win 4096, options [nop,nop,TS val 379018383 ecr 2160547604], length 198
18:10:35.716330 IP server.443 > client.50122: Flags [P.], seq 3095:3361, ack 300, win 235, options [nop,nop,TS val 2160547633 ecr 379018383], length 266
18:10:35.729816 IP client.50122 > server.443: Flags [.], ack 3361, win 4087, options [nop,nop,TS val 379018398 ecr 2160547633], length 0
18:10:35.734595 IP client.50122 > server.443: Flags [P.], seq 300:726, ack 3361, win 4096, options [nop,nop,TS val 379018398 ecr 2160547633], length 426
18:10:35.773934 IP server.443 > client.50122: Flags [.], ack 726, win 243, options [nop,nop,TS val 2160547691 ecr 379018398], length 0
18:10:35.845389 IP server.443 > client.50122: Flags [P.], seq 3361:3766, ack 726, win 243, options [nop,nop,TS val 2160547762 ecr 379018398], length 405
18:10:35.859043 IP client.50122 > server.443: Flags [.], ack 3766, win 4083, options [nop,nop,TS val 379018524 ecr 2160547762], length 0
18:10:35.901352 IP client.50122 > server.443: Flags [P.], seq 726:1264, ack 3766, win 4096, options [nop,nop,TS val 379018561 ecr 2160547762], length 538
18:10:35.901394 IP server.443 > client.50122: Flags [.], ack 1264, win 252, options [nop,nop,TS val 2160547818 ecr 379018561], length 0
18:10:36.050250 IP server.443 > client.50122: Flags [P.], seq 3766:4235, ack 1264, win 252, options [nop,nop,TS val 2160547967 ecr 379018561], length 469
18:10:36.064314 IP client.50122 > server.443: Flags [.], ack 4235, win 4081, options [nop,nop,TS val 379018727 ecr 2160547967], length 0
18:10:36.935368 IP client.50122 > server.443: Flags [P.], seq 1264:1882, ack 4235, win 4096, options [nop,nop,TS val 379019590 ecr 2160547967], length 618
18:10:36.935408 IP server.443 > client.50122: Flags [.], ack 1882, win 262, options [nop,nop,TS val 2160548852 ecr 379019590], length 0
18:10:36.937074 IP client.50122 > server.443: Flags [P.], seq 1882:1956, ack 4235, win 4096, options [nop,nop,TS val 379019590 ecr 2160547967], length 74
18:10:36.937105 IP server.443 > client.50122: Flags [.], ack 1956, win 262, options [nop,nop,TS val 2160548854 ecr 379019590], length 0
18:10:37.029133 IP server.443 > client.50122: Flags [P.], seq 4235:4480, ack 1956, win 262, options [nop,nop,TS val 2160548946 ecr 379019590], length 245
18:10:37.029148 IP server.443 > client.50122: Flags [F.], seq 4480, ack 1956, win 262, options [nop,nop,TS val 2160548946 ecr 379019590], length 0
18:10:37.042577 IP client.50122 > server.443: Flags [.], ack 4480, win 4088, options [nop,nop,TS val 379019703 ecr 2160548946], length 0
18:10:37.044078 IP client.50122 > server.443: Flags [.], ack 4481, win 4096, options [nop,nop,TS val 379019703 ecr 2160548946], length 0
18:10:37.045338 IP client.50122 > server.443: Flags [F.], seq 1956, ack 4481, win 4096, options [nop,nop,TS val 379019704 ecr 2160548946], length 0
18:10:37.045369 IP server.443 > client.50122: Flags [.], ack 1957, win 262, options [nop,nop,TS val 2160548962 ecr 379019704], length 0

traceback :

Traceback (most recent call last):
  File "/my_app/handlers/propagate.py", line 445, in _upload
    up.start()
  File "/my_app/handlers/transfer.py", line 239, in start
    params=self.req_params, data=self.upload_handler)
  File "/my_app/handlers/http.py", line 26, in wrapped
    return f(self, *args, **kwargs)
  File "/my_app/handlers/http.py", line 108, in request_post
    timeout=(self.connect_timeout, self.read_timeout))
  File "/my_app/.virtualenvs/sync-in-app/lib/python2.7/site-packages/requests/sessions.py", line 511, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/my_app/.virtualenvs/sync-in-app/lib/python2.7/site-packages/requests/sessions.py", line 468, in request
    resp = self.send(prep, **send_kwargs)
  File "/my_app/.virtualenvs/sync-in-app/lib/python2.7/site-packages/requests/sessions.py", line 608, in send
    r.content
  File "/my_app/.virtualenvs/sync-in-app/lib/python2.7/site-packages/requests/models.py", line 737, in content
    self._content = bytes().join(self.iter_content(CONTENT_CHUNK_SIZE)) or bytes()
  File "/my_app/.virtualenvs/sync-in-app/lib/python2.7/site-packages/requests/models.py", line 663, in generate
    raise ChunkedEncodingError(e)
ChunkedEncodingError: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))
@Lukasa

This comment has been minimized.

Show comment
Hide comment
@Lukasa

Lukasa May 18, 2016

Member

If you change the call to use stream=True you should get a response object back. It'll change where the exception comes in your code, but it'll let you see more about what's going on.

Member

Lukasa commented May 18, 2016

If you change the call to use stream=True you should get a response object back. It'll change where the exception comes in your code, but it'll let you see more about what's going on.

@johaven

This comment has been minimized.

Show comment
Hide comment
@johaven

johaven May 18, 2016

@Lukasa You are right, now i get status_code and ChunkedEncodingError exception is raised when i try to read r.json(). This behaviour suits me. Thank you :)

johaven commented May 18, 2016

@Lukasa You are right, now i get status_code and ChunkedEncodingError exception is raised when i try to read r.json(). This behaviour suits me. Thank you :)

@Lukasa

This comment has been minimized.

Show comment
Hide comment
@Lukasa

Lukasa May 18, 2016

Member

No problem, I'm glad we were able to find something that works for you!

Member

Lukasa commented May 18, 2016

No problem, I'm glad we were able to find something that works for you!

@johaven

This comment has been minimized.

Show comment
Hide comment
@johaven

johaven May 18, 2016

@Lukasa One last question: I think there is a bug when i try to read a custom header (the problem seems to be string encoding)

Example OK (original custom content: TODO.md)

print r.headers
# {'Transfer-Encoding': 'chunked', ..., 'custom': 'TODO.md', 'Content-Type': 'text/html; charset=utf-8'}
print r.headers['Transfer-Encoding']
# chunked
print r.headers['custom'] 
#TODO.md

Example KO (original custom content: hackiñg.txt)

print r.headers
# {'Transfer-Encoding': 'chunked', '..., 'custom': 'hacki\xf1g.txt', 'Content-Type': 'text/html; charset=utf-8'}
print r.headers['Transfer-Encoding']
# chunked
print r.headers['custom'] 
# here code is blocked

Some times (in second case) code block on printing Transfer-Encoding header
I use python 2.7

johaven commented May 18, 2016

@Lukasa One last question: I think there is a bug when i try to read a custom header (the problem seems to be string encoding)

Example OK (original custom content: TODO.md)

print r.headers
# {'Transfer-Encoding': 'chunked', ..., 'custom': 'TODO.md', 'Content-Type': 'text/html; charset=utf-8'}
print r.headers['Transfer-Encoding']
# chunked
print r.headers['custom'] 
#TODO.md

Example KO (original custom content: hackiñg.txt)

print r.headers
# {'Transfer-Encoding': 'chunked', '..., 'custom': 'hacki\xf1g.txt', 'Content-Type': 'text/html; charset=utf-8'}
print r.headers['Transfer-Encoding']
# chunked
print r.headers['custom'] 
# here code is blocked

Some times (in second case) code block on printing Transfer-Encoding header
I use python 2.7

@Lukasa

This comment has been minimized.

Show comment
Hide comment
@Lukasa

Lukasa May 18, 2016

Member

What Python version are you using, please?

Member

Lukasa commented May 18, 2016

What Python version are you using, please?

@johaven

This comment has been minimized.

Show comment
Hide comment
@johaven

johaven May 18, 2016

Python 2.7.10 (default, Oct 23 2015, 19:19:21)

johaven commented May 18, 2016

Python 2.7.10 (default, Oct 23 2015, 19:19:21)

@Lukasa

This comment has been minimized.

Show comment
Hide comment
@Lukasa

Lukasa May 18, 2016

Member

Are you sure the code blocks on the header print, or does it throw an exception you're swallowing?

Member

Lukasa commented May 18, 2016

Are you sure the code blocks on the header print, or does it throw an exception you're swallowing?

@johaven

This comment has been minimized.

Show comment
Hide comment
@johaven

johaven May 18, 2016

ok you can kill me ... i use ThreadPool which hide some exceptions (UnicodeDecodeError for example ... ARGHH) ... sorry for your time :(

johaven commented May 18, 2016

ok you can kill me ... i use ThreadPool which hide some exceptions (UnicodeDecodeError for example ... ARGHH) ... sorry for your time :(

@Lukasa

This comment has been minimized.

Show comment
Hide comment
@Lukasa

Lukasa May 18, 2016

Member

=P No worries. Glad you worked it out!

Member

Lukasa commented May 18, 2016

=P No worries. Glad you worked it out!

@johaven

This comment has been minimized.

Show comment
Hide comment
@johaven

johaven May 18, 2016

Thank's for your patience 👍

johaven commented May 18, 2016

Thank's for your patience 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment