Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Misleading IncompleteRead exception, if a HTTP response contains unexpected content #3261

Closed
crazyscientist opened this issue Jan 2, 2024 · 2 comments · Fixed by #3273
Closed

Misleading IncompleteRead exception, if a HTTP response contains unexpected content #3261

crazyscientist opened this issue Jan 2, 2024 · 2 comments · Fixed by #3273
Labels
💰 Bounty $100 If you complete this issue we'll pay you $100 on OpenCollective! Contributor Friendly ♥

Comments

@crazyscientist
Copy link
Contributor

Subject

When interacting with poorly implemented servers or when paying little attention during mocking of HTTP requests, one can be faced e.g. with a 204 response, which contains an unexpected body.

So far so bad. urllib3 only raises an IncompleteRead exception, which provides no insight into the reason (e.g. a 204 response with body).

Environment

  • Python 3.10
  • urllib3 2.1.0

Steps to Reproduce

pip install requests responses
# scratch.py
import unittest
import responses
import requests


class MyTest(unittest.TestCase):
    url = "http://example.com"

    def test_requestsmock(self):
        rmock = responses.RequestsMock()
        rmock.start()
        rmock.post(url=self.url,
                   status=204,
                   json={"status": "ok"},
                   auto_calculate_content_length=True)

        response = requests.post(url=self.url, data="Foo Bar!")

    @responses.activate
    def test_decorator(self):
        responses.add(method=responses.POST,
                      url=self.url,
                      status=204,
                      json={"status": "ok"},
                      auto_calculate_content_length=True)
        response = requests.post(url=self.url, data="Foo Bar!")


unittest.main()

Expected Behavior

If an empty response body is expected, the raised exception should complain about the presence of an unexpected body.

Also, the reason for expecting an empty body would be helpful and speed up debugging.

Actual Behavior

Instead, IncompleteRead(16 bytes read, -16 more expected) is raised:

python3.10 scratch.py
EE
======================================================================
ERROR: test_decorator (__main__.MyTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/andi/virtualenvs/checkers/lib/python3.10/site-packages/urllib3/response.py", line 712, in _error_catcher
    yield
  File "/home/andi/virtualenvs/checkers/lib/python3.10/site-packages/urllib3/response.py", line 833, in _raw_read
    raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
urllib3.exceptions.IncompleteRead: IncompleteRead(16 bytes read, -16 more expected)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/andi/virtualenvs/checkers/lib/python3.10/site-packages/requests/models.py", line 816, in generate
    yield from self.raw.stream(chunk_size, decode_content=True)
  File "/home/andi/virtualenvs/checkers/lib/python3.10/site-packages/urllib3/response.py", line 934, in stream
    data = self.read(amt=amt, decode_content=decode_content)
  File "/home/andi/virtualenvs/checkers/lib/python3.10/site-packages/urllib3/response.py", line 905, in read
    data = self._raw_read(amt)
  File "/home/andi/virtualenvs/checkers/lib/python3.10/site-packages/urllib3/response.py", line 811, in _raw_read
    with self._error_catcher():
  File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/home/andi/virtualenvs/checkers/lib/python3.10/site-packages/urllib3/response.py", line 729, in _error_catcher
    raise ProtocolError(f"Connection broken: {e!r}", e) from e
urllib3.exceptions.ProtocolError: ('Connection broken: IncompleteRead(16 bytes read, -16 more expected)', IncompleteRead(16 bytes read, -16 more expected))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/andi/virtualenvs/checkers/lib/python3.10/site-packages/responses/__init__.py", line 232, in wrapper
    return func(*args, **kwargs)
  File "/home/andi/.config/JetBrains/PyCharm2023.2/scratches/scratch.py", line 28, in test_decorator
    response = requests.post(url=self.url, data="Foo Bar!")
  File "/home/andi/virtualenvs/checkers/lib/python3.10/site-packages/requests/api.py", line 115, in post
    return request("post", url, data=data, json=json, **kwargs)
  File "/home/andi/virtualenvs/checkers/lib/python3.10/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "/home/andi/virtualenvs/checkers/lib/python3.10/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/andi/virtualenvs/checkers/lib/python3.10/site-packages/requests/sessions.py", line 747, in send
    r.content
  File "/home/andi/virtualenvs/checkers/lib/python3.10/site-packages/requests/models.py", line 899, in content
    self._content = b"".join(self.iter_content(CONTENT_CHUNK_SIZE)) or b""
  File "/home/andi/virtualenvs/checkers/lib/python3.10/site-packages/requests/models.py", line 818, in generate
    raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ('Connection broken: IncompleteRead(16 bytes read, -16 more expected)', IncompleteRead(16 bytes read, -16 more expected))

======================================================================
ERROR: test_requestsmock (__main__.MyTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/andi/virtualenvs/checkers/lib/python3.10/site-packages/urllib3/response.py", line 712, in _error_catcher
    yield
  File "/home/andi/virtualenvs/checkers/lib/python3.10/site-packages/urllib3/response.py", line 833, in _raw_read
    raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
urllib3.exceptions.IncompleteRead: IncompleteRead(16 bytes read, -16 more expected)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/andi/virtualenvs/checkers/lib/python3.10/site-packages/requests/models.py", line 816, in generate
    yield from self.raw.stream(chunk_size, decode_content=True)
  File "/home/andi/virtualenvs/checkers/lib/python3.10/site-packages/urllib3/response.py", line 934, in stream
    data = self.read(amt=amt, decode_content=decode_content)
  File "/home/andi/virtualenvs/checkers/lib/python3.10/site-packages/urllib3/response.py", line 905, in read
    data = self._raw_read(amt)
  File "/home/andi/virtualenvs/checkers/lib/python3.10/site-packages/urllib3/response.py", line 811, in _raw_read
    with self._error_catcher():
  File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/home/andi/virtualenvs/checkers/lib/python3.10/site-packages/urllib3/response.py", line 729, in _error_catcher
    raise ProtocolError(f"Connection broken: {e!r}", e) from e
urllib3.exceptions.ProtocolError: ('Connection broken: IncompleteRead(16 bytes read, -16 more expected)', IncompleteRead(16 bytes read, -16 more expected))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/andi/.config/JetBrains/PyCharm2023.2/scratches/scratch.py", line 18, in test_requestsmock
    response = requests.post(url=self.url, data="Foo Bar!")
  File "/home/andi/virtualenvs/checkers/lib/python3.10/site-packages/requests/api.py", line 115, in post
    return request("post", url, data=data, json=json, **kwargs)
  File "/home/andi/virtualenvs/checkers/lib/python3.10/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "/home/andi/virtualenvs/checkers/lib/python3.10/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/andi/virtualenvs/checkers/lib/python3.10/site-packages/requests/sessions.py", line 747, in send
    r.content
  File "/home/andi/virtualenvs/checkers/lib/python3.10/site-packages/requests/models.py", line 899, in content
    self._content = b"".join(self.iter_content(CONTENT_CHUNK_SIZE)) or b""
  File "/home/andi/virtualenvs/checkers/lib/python3.10/site-packages/requests/models.py", line 818, in generate
    raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ('Connection broken: IncompleteRead(16 bytes read, -16 more expected)', IncompleteRead(16 bytes read, -16 more expected))

----------------------------------------------------------------------
Ran 2 tests in 0.005s

FAILED (errors=2)
@sethmlarson
Copy link
Member

@crazyscientist Thanks for reporting this. I would accept a fix that detects this situation and reraises as a ProtocolError with a better error message.

@sethmlarson sethmlarson added Contributor Friendly ♥ 💰 Bounty $100 If you complete this issue we'll pay you $100 on OpenCollective! labels Jan 9, 2024
crazyscientist added a commit to crazyscientist/urllib3 that referenced this issue Jan 9, 2024
@crazyscientist
Distinguish between truncated and excess content in response
4bbb8ec 
`HTTPResponse._raw_read` raises `IncompleteRead` if the content length does not match the
expected content length. For malformed responses (e.g. 204 response with content) the re-raised
`ProtocolError` was a bit too unclear about the unexpected excess content being the reason for the
exception.

With this change, the exception points out, that the client is not dealing with a connection error
but a protocol violation.

Closes urllib3#3261
crazyscientist added a commit to crazyscientist/urllib3 that referenced this issue Jan 9, 2024
@crazyscientist
Distinguish between truncated and excess content in response
9f6bad7 
`HTTPResponse._raw_read` raises `IncompleteRead` if the content length does not match the
expected content length. For malformed responses (e.g. 204 response with content) the re-raised
`ProtocolError` was a bit too unclear about the unexpected excess content being the reason for the
exception.

With this change, the exception points out, that the client is not dealing with a connection error
but a protocol violation.

Closes urllib3#3261
@crazyscientist crazyscientist mentioned this issue Jan 9, 2024
Merged
crazyscientist added a commit to crazyscientist/urllib3 that referenced this issue Jan 9, 2024
@crazyscientist
Distinguish between truncated and excess content in response
8537ee4 
`HTTPResponse._raw_read` raises `IncompleteRead` if the content length does not match the
expected content length. For malformed responses (e.g. 204 response with content) the re-raised
`ProtocolError` was a bit too unclear about the unexpected excess content being the reason for the
exception.

With this change, the exception points out, that the client is not dealing with a connection error
but a protocol violation.

Closes urllib3#3261
crazyscientist added a commit to crazyscientist/urllib3 that referenced this issue Jan 9, 2024
@crazyscientist
Distinguish between truncated and excess content in response
64de020 
`HTTPResponse._raw_read` raises `IncompleteRead` if the content length does not match the
expected content length. For malformed responses (e.g. 204 response with content) the re-raised
`ProtocolError` was a bit too unclear about the unexpected excess content being the reason for the
exception.

With this change, the exception points out, that the client is not dealing with a connection error
but a protocol violation.

Closes urllib3#3261
crazyscientist added a commit to crazyscientist/urllib3 that referenced this issue Jan 10, 2024
@crazyscientist
Distinguish between truncated and excess content in response
27ed366 
`HTTPResponse._raw_read` raises `IncompleteRead` if the content length does
not match the expected content length. For malformed responses (e.g. 204
response with content) the re-raised `ProtocolError` was a bit too unclear
about the unexpected excess content being the reason for the exception.

With this change, the exception points out, that the client is not dealing
with a connection error but a protocol violation.

Closes urllib3#3261
crazyscientist added a commit to crazyscientist/urllib3 that referenced this issue Feb 5, 2024
@crazyscientist
Distinguish between truncated and excess content in response
da7e153 
`HTTPResponse._raw_read` raises `IncompleteRead` if the content length does
not match the expected content length. For malformed responses (e.g. 204
response with content) the re-raised `ProtocolError` was a bit too unclear
about the unexpected excess content being the reason for the exception.

With this change, the exception points out, that the client is not dealing
with a connection error but a protocol violation.

Closes urllib3#3261
@illia-v
Copy link
Member

illia-v commented Feb 12, 2024

@crazyscientist thanks for improving the error message, feel free to claim the $100 bounty from our OpenCollective

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
💰 Bounty $100 If you complete this issue we'll pay you $100 on OpenCollective! Contributor Friendly ♥
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Distinguish between truncated and excess content in response crazyscientist/urllib3
3 participants
@crazyscientist @illia-v @sethmlarson